Most popular crawler repositories and open source projects

Crawler-for-Github-Trending

🕷️ A node crawler for github trending.

19 180 180

xvideos

xvideos API library

61 180 180

packagist-mirror

📦✂️📋📦 Create a mirror of packagist.org metadata for use locally wit...

65 180 180

rotating-tor-http-proxy

A multi-arch image provides one HTTP proxy endpoint with many concurre...

40 178 178

datmusic-api

51 178 178

nCov2019_data_crawler

疫情数据爬虫，2019新型冠状病毒数据仓库，轨迹数据，同乘数据，报道

32 177 177

spoon

🥄 A package for building specific Proxy Pool for different Sites.

23 176 176

DotnetCrawler

DotnetCrawler is a straightforward, lightweight web crawling/scrapying...

66 176 176

authority-data

官方权威数据：统计年签，统计公报，互联网行业报告，工信部数据，ICT报告...

26 173 173

kuaishou-crawler

As you can see, a kuaishou crawler

65 172 172

search

An Open Source Search Engine

105 172 172

ChainWalker

Rapid Smart Contract Crawler

25 171 171

cocrawler

CoCrawler is a versatile web crawler built using modern tools and conc...

25 169 169

ScrapingOutsourcing

ScrapingOutsourcing专注分享爬虫代码尽量每周更新一个

44 168 168

python-dcdownloader

由Python编写的全异步实现的动漫之家(dmzj)漫画批量下载器（爬虫）

19 165 165

yispider

一款分布式爬虫平台，帮助你更好的管理和开发爬虫。内置一套爬虫定义规则...

31 163 163

mm131

MM131网站图片爬取 :rotating_light:

53 163 163

crypto-crawler-rs

A rock-solid cryptocurrency crawler library.

69 163 163

fun_crawler

Crawl some picture for fun

129 162 162

HttpCode.Core

简单、易用、高效一个有态度的开源.Net Http请求框架!可以用制作爬虫，api...

61 159 159

crawler-china-mainland-universities

中国大陆大学列表爬虫

49 159 159

TorCrawl.py

Crawl and extract (regular or onion) webpages through TOR network

43 159 159

soksaccounts

🔥 Shadowsocks 账号爬虫

50 157 157

Squidwarc

Squidwarc is a high fidelity, user scriptable, archival crawler that u...

28 156 156

DouyuBarrage-Pro

(2020年最新)斗鱼弹幕抓取及可视化管理平台第二版，提供弹幕抓取、弹幕实时...

25 156 156

WebScrapper

Powerful Telegram bot for web scraping and crawling. Fast, easy, and l...

94 156 156

evine

Interactive CLI Web Crawler

32 154 154

awesome-python-primer

自学入门 Python 优质中文资源索引，包含书籍 / 文档 / 视频，适用于爬虫...

25 154 154

ngMeta

Dynamic meta tags in your AngularJS single page application

43 153 153

tir

Have time.ir in shell!

8 153 153

onecomic

一本漫画

33 152 152

urlbuster

Powerful mutable web directory fuzzer to bruteforce existing and/or hi...

33 151 151

jlitespider

A lite distributed Java spider framework :-)

39 150 150

ir

Projeto de calculo de Imposto de Renda em operacoes na bovespa automat...

39 150 150

crawler

Go process used to crawl websites

20 149 149

pkulaw_spider

爬取北大法宝网http://www.pkulaw.cn/Case/

57 149 149

ghs

GitHub Search: Platform used to crawl, store and present projects from...

18 149 149

pachong

一些爬虫的代码

100 146 146

NewsCrawler

新闻爬虫，爬取新浪、搜狐、新华网即时财经新闻。

28 144 144

KTSpeechCrawler

Automatically constructing corpus for automatic speech recognition fro...

37 143 143

telegram-crawler

🕷 Automatically detect changes made to the official Telegram sites, cl...

16 143 143

crawley

The unix-way web crawler

8 143 143

pixiv_func_mobile

功能齐全的Pixiv第三方客户端免代理支持查看动图查看小说

9 142 142

fontObfuscator

字体混淆服务

14 141 141

aliexpress-product-scraper

Get Aliexpress product details as a json response including feedbacks,...

65 141 141

douban-movie

Golang爬虫爬取豆瓣电影Top250

67 140 140

taki

Take a snapshot of any website.

18 140 140

HotNewsAnalysis

利用文本挖掘技术进行新闻热点关注问题分析

46 139 139

bilibili_member_crawler

B站用户爬虫好耶~是爬虫

21 138 138

nebula

🌌 A libp2p DHT crawler, monitor, and measurement tool that exposes t...

17 137 137