Most popular crawler repositories and open source projects

ZhengFang_System_Spider

:bug:一只登录正方教务管理系统，爬取数据的小爬虫

2   21   21  

html-article-extractor

A web page content extractor

1   21   21  

bthello-app

Python3 DHT 磁力种子爬虫 种子解析 种子搜索 演示地址

18   21   21  

covid-19-crawler

코로나 확진자 수/정보 크롤링

10   21   21  

xianzhi_articles

先知文章爬虫项目-[包含2021年7月之前所有文章]

5   21   21  

actor-youtube-scraper

Apify actor to scrape Youtube search results. You can set the maximum...

16   21   21  

proxycrawl-php

ProxyCrawl PHP library for scraping and crawling websites

5   21   21  

publiccode-crawler

publiccode.yml crawler for the Open Source software catalog of Develop...

49   21   21  

libp2p-dht-scrape-aas

🧹 A libp2p DHT scraper as a service allowing anyone to collect, consu...

8   21   21  

scrapy_poetry

本项目使用scrapy对 古诗文网 进行爬虫，获取不同分类（爱情、七夕等）的宋...

0   21   21  

emarketcrawlR

This R package provides a crawler to scrape the European Energy Market...

10   21   21  

estate-crawler

Scraping the real estate agencies for up-to-date house listings as soo...

5   21   21  

lopez

Crawling and scraping the Web for fun and profit

3   21   21  

Codeforces-AutoCommit

When you solve the problem of the Codeforces site, it automatically co...

5   21   21  

book-spider

🎉 开箱即用的高性能可自定义的笔趣阁小说爬虫 快速下载无广告小说

4   21   21  

ExHentaiReader

Best manga-viewer on windows for crawling/downloading/browsing exhenta...

1   21   21  

Crawler

整理本人在2021年10月-12月期间写的一些爬虫demo，比如用于渗透测试中SQL注...

5   21   21  

telegram-member-inviter

Crawling client's groups and channels to invite their members to a tar...

15   21   21  

retro-env-can-weather-chan

Retro Environment Canada Weather Channel for your browser

3   21   21  

Onlyfans-dl

This tool downloads all photos/videos from an OnlyFans profile, creati...

8   21   21  

scrapy-tor-proxy-rotation

An IP rotator via Tor for Scrapy.

1   21   21  

Fast-KTSpeechCrawler

Parallelized automatic corpus collection for ASR. Forked from https://...

2   20   20  

crawl_xuexi

学习强国APP上机器学习课程，学习慕课视频批量下载

18   20   20  

Damn-Small-URL-Crawler

A Minimal Yet Powerful Crawler for Extracting all The Internal/Externa...

11   20   20  

anime-tracker

:spider_web: All in one place to track your favorite animes

1   20   20  

ptt-crawler

ptt-crawler is a web crawler module designed to scarpe data from Ptt.

10   20   20  

k-webtoon-crawler

Korean webtoon crawler with Python 3. 한국 웹툰 크롤러.

2   20   20  

colymer-acquirers

各种爬虫（目前支持Instagram、Weibo、Twitter）Miscellaneous crawlers (c...

6   20   20  

Instagram_Crawler

인스타그램 크롤러 (Python, Selenium)

19   20   20  

Web-Crawler

A multithreaded web crawler using two mechanism - single lock and thre...

11   20   20  

zhihu_crawler

本程序支持关键词搜索、热榜、用户信息、回答、专栏文章、评论等信息的抓取

9   20   20  

vermouth

A torrent site written in the python language & douban scraper

11   20   20  

crawl

Lightweight library for scalable crawlers in Go.

4   20   20  

crawler

A simple and flexible web crawler framework for java.

5   20   20  

scrapy-azuresearch-crawler-samples

Scrapy as a Web Crawler for Azure Search Samples

7   20   20  

crawler

Crawl your own website with various clients for SEO and indexing purpo...

4   20   20  

movieRater.React

A useful website for finding movie's rating in Chinese and English. By...

2   20   20  

domfind

A Python DNS crawler to find identical domain names under different TL...

3   20   20  

hero

百万英雄答题助手 - 兼容全部答题 APP

7   20   20  

flutter_spider_fx

Flutter爬虫框架，帮助开发者快速在移动设备上构建爬虫，单线程版本

1   20   20  

peeling-onions

A repository to store Deep Web (onion domain) crawler, scraper, and NL...

7   20   20  

collector-filesystem

Norconex Filesystem Collector is a flexible crawler for collecting, pa...

11   20   20  

ppspider_example

ppspider爬虫例子，B站视频信息及评论爬取，qq音乐信息及评论爬取，推特主...

13   20   20  

googleart_scraper

Scrape images from googleart

3   20   20  

sse-option-crawler

SSE 50 index options crawler 上证50期权数据爬虫

8   20   20  

goApp

golang 的一些开源项目，垃圾清理小工具、华为官网抢购程序、房产爬虫、报...

3   20   20  

taiwanlottery

🇹🇼台灣樂透爬蟲🐛(台灣各類型樂透爬蟲)😄

4   20   20  

torrent-crawler

crawls and stores list of torrent links

7   20   20  

WebArchiver

Decentralized web archiving

3   20   20  

web-crawler

Python Web Crawler with Selenium and PhantomJS

14   19   19