Most popular crawler repositories and open source projects

utsusemi

A tool to generate a static website by crawling the original site.

2   30   30  

SINA_Spider

新浪微博爬虫：登录、关键词微博查询、微博监控

20   30   30  

TaobaoAnalysis

练习NLP，分析淘宝评论的项目

7   30   30  

goGamer

巴哈姆特自訂API

2   30   30  

bot_bandejao_UFMG

🤖🍴 A Python script that scrapes UFMG's restaurants menus and publishes...

0   30   30  

instagram-downloader

Node.js/Express app to retrive instagram video/image download urls

13   30   30  

igxe-c5-buff-csgo-skins-sale-data-catch

Automatically get the csgo skins sale data on igxe.cn and buff and c5g...

2   30   30  

squirm

This was the night of the crawling terror!

2   30   30  

scaling-to-distributed-crawling

Repository for the Mastering Web Scraping in Python: Scaling to Distri...

7   30   30  

iranian-calendar-events

Fetch Iranian calendar events (Jalali, Hijri and Gregorian) from time....

2   29   29  

integrada.minhabiblioteca.com.br

Download de livros para PDF/EPUB - Integrada.minhabiblioteca / vitalso...

10   29   29  

scrapit

Scraping scripts for various websites.

1   29   29  

DouyuBarrage

(2020年最新)斗鱼弹幕抓取及实时弹幕数据可视化，分为crawler(弹幕抓取)，s...

12   29   29  

tse-client

A client for fetching stock data from the Tehran Stock Exchange (TSETM...

10   29   29  

fmovies-crawler

A web scraper that allows you to fetch Movies and TV series informatio...

3   29   29  

reactor-crw

Simple content crawler for joyreactor.cc

2   29   29  

Bayesian-Stock-Market-Sentiment

A stock market text sentiment analysis website. A股舆情分析, web-crawl...

10   29   29  

crawler

爬取tumblr关注博主图片

7   28   28  

telegram_bbbot

Telegram Bug Bounty Bot

5   28   28  

python-crawl

Library to crawl and extract internal links from domain

8   28   28  

vietnam-ecommerce-crawler

Crawling the data from lazada, websosanh, compare.vn, cdiscount and cu...

9   28   28  

Moodle-Downloader

A Moodle Crawler that downloads course content from Moodle (eg. lectur...

5   28   28  

google-image-downloader

A script to download images from images.google.com

17   28   28  

scrapy-picture-spider

The project is a spider that uses scrapy and beautifulsoup4 for crawl...

7   28   28  

frisbee

Collect email addresses by crawling search engine results.

7   28   28  

fess-crawler

Web/FileSystem Crawler Library

18   28   28  

BitInsight

:earth_africa: Bittorrent Network Overview through Infohash Indexing,...

6   28   28  

serritor

Serritor is an open source web crawler framework built upon Selenium a...

15   28   28  

AzureSearchCrawler

A simple web crawler, using Abot, that indexes page contents into Azur...

11   28   28  

advanced-php-crawler

新浪博客文章/wenku8轻小说文库爬虫，可抓取图片保存，一键制作电子书。kin...

11   28   28  

TwitterCrawler

抓取twitter数据，可根据时间、话题、用户名等条件抓取数据，twitter爬虫

9   28   28  

GooglePlayWebServiceAPI

Tiny script to crawl information of a specific application in the Goog...

10   28   28  

apkpure_download

a py module to download apk from apkpure.com

13   28   28  

cache-warmup

🔥 Composer package to warm up caches of URLs located in XML sitemaps

2   28   28  

job-funnel-ts

Automated tool for scraping job postings into a .xlsx files inspired b...

0   27   27  

ytpriv

YT metadata exporter

7   27   27  

ds-video-helper

群晖Video Station助手，自动获取豆瓣电影信息，并填写Video Station视频信...

4   27   27  

scrapy_pro

关于5000+站点的scrapy爬虫开发，涉及一些技术架构搭建以及各种反爬方案，...

13   27   27  

php-google

Google search results crawler, get google search results that you need...

10   27   27  

PyperGrabber

Fetches PubMed article IDs (PMIDs) from email inbox, then crawls PubMe...

7   27   27  

Android-Apps-Downloader

📱 A tool to download android apps from Google Play Store and Xiaomi Ap...

22   27   27  

generic-seeder

Generic altcoin DNS seeder. Compatible with virtually any cryptocurren...

80   27   27  

YourLesson

深圳大学抢课系统

6   27   27  

spider

A web spider framework

7   27   27  

selenium_facebook_scraper

A simple python3 script used to download a users's friend list from fa...

15   27   27  

botcity-framework-web-python

BotCity Framework Web - Python

17   27   27  

pinscrape

A simple library to scrape Pinterest images written in Python

3   27   27  

iranian-news-agencies-crawler

a crawler to fetch last news from Iranian(Persian) news agencies.

2   27   27  

AyugeSpiderTools

scrapy 扩展库：其主要功能使 scrapy 开发不用在意 item，pipeline，middle...

3   27   27  

serverless-crawler-demo

Serverless Architecture Crawler demo

8   26   26