Most popular crawler repositories and open source projects

pysaint

[deprecated] 유세인트 파이썬 클라이언트

3   31   31  

Spydan

A web spider for shodan.io without using the Developer API.

8   31   31  

spiderable-middleware

🤖 Prerendering for JavaScript powered websites. Great solution for PW...

4   31   31  

see

Search Engine in Erlang

3   31   31  

instagram_scraper

Extract instagram users informations from hashtags. This scraper can e...

14   31   31  

bet365API

The latest way to get bet365 data odds, with a delay of 0.2 seconds be...

8   31   31  

octopus

Recursive and multi-threaded broken link checker

11   31   31  

Mechanize.NET

Stateful programmatic web browsing, based on Python-Mechanize, which i...

9   30   30  

invana-bot

A Web Crawler that scrapes using YAML and python code.

9   30   30  

TTBot2.0

app版本今日头条 用户登录/个人主页/关注列表/粉丝列表/评论点赞收藏 关键...

16   30   30  

utsusemi

A tool to generate a static website by crawling the original site.

2   30   30  

SINA_Spider

新浪微博爬虫：登录、关键词微博查询、微博监控

20   30   30  

TaobaoAnalysis

练习NLP，分析淘宝评论的项目

7   30   30  

goGamer

巴哈姆特自訂API

2   30   30  

bot_bandejao_UFMG

🤖🍴 A Python script that scrapes UFMG's restaurants menus and publish...

0   30   30  

instagram-downloader

Node.js/Express app to retrive instagram video/image download urls

13   30   30  

igxe-c5-buff-csgo-skins-sale-data-catch

Automatically get the csgo skins sale data on igxe.cn and buff and c5g...

2   30   30  

squirm

This was the night of the crawling terror!

2   30   30  

reactor-crw

Simple content crawler for joyreactor.cc

2   29   29  

Bayesian-Stock-Market-Sentiment

A stock market text sentiment analysis website. A股舆情分析, web-crawl...

10   29   29  

iranian-calendar-events

Fetch Iranian calendar events (Jalali, Hijri and Gregorian) from time....

2   29   29  

integrada.minhabiblioteca.com.br

Download de livros para PDF/EPUB - Integrada.minhabiblioteca / vitalso...

10   29   29  

scrapit

Scraping scripts for various websites.

1   29   29  

DouyuBarrage

(2020年最新)斗鱼弹幕抓取及实时弹幕数据可视化，分为crawler(弹幕抓取)，s...

12   29   29  

tse-client

A client for fetching stock data from the Tehran Stock Exchange (TSETM...

10   29   29  

fmovies-crawler

A web scraper that allows you to fetch Movies and TV series informatio...

3   29   29  

crawler

爬取tumblr关注博主图片

7   28   28  

telegram_bbbot

Telegram Bug Bounty Bot

5   28   28  

python-crawl

Library to crawl and extract internal links from domain

8   28   28  

vietnam-ecommerce-crawler

Crawling the data from lazada, websosanh, compare.vn, cdiscount and cu...

9   28   28  

Moodle-Downloader

A Moodle Crawler that downloads course content from Moodle (eg. lectur...

5   28   28  

google-image-downloader

A script to download images from images.google.com

17   28   28  

scrapy-picture-spider

The project is a spider that uses scrapy and beautifulsoup4 for crawl...

7   28   28  

frisbee

Collect email addresses by crawling search engine results.

7   28   28  

fess-crawler

Web/FileSystem Crawler Library

18   28   28  

BitInsight

:earth_africa: Bittorrent Network Overview through Infohash Indexing,...

6   28   28  

AzureSearchCrawler

A simple web crawler, using Abot, that indexes page contents into Azur...

11   28   28  

advanced-php-crawler

新浪博客文章/wenku8轻小说文库爬虫，可抓取图片保存，一键制作电子书。kin...

11   28   28  

TwitterCrawler

抓取twitter数据，可根据时间、话题、用户名等条件抓取数据，twitter爬虫

9   28   28  

GooglePlayWebServiceAPI

Tiny script to crawl information of a specific application in the Goog...

10   28   28  

apkpure_download

a py module to download apk from apkpure.com

13   28   28  

cache-warmup

🔥 Composer package to warm up caches of URLs located in XML sitemaps

2   28   28  

YourLesson

深圳大学抢课系统

6   27   27  

spider

A web spider framework

7   27   27  

selenium_facebook_scraper

A simple python3 script used to download a users's friend list from fa...

15   27   27  

botcity-framework-web-python

BotCity Framework Web - Python

17   27   27  

pinscrape

A simple library to scrape Pinterest images written in Python

3   27   27  

iranian-news-agencies-crawler

a crawler to fetch last news from Iranian(Persian) news agencies.

2   27   27  

AyugeSpiderTools

scrapy 扩展库：其主要功能使 scrapy 开发不用在意 item，pipeline，middle...

3   27   27  

job-funnel-ts

Automated tool for scraping job postings into a .xlsx files inspired b...

0   27   27