Most popular crawler repositories and open source projects

ChainWalker

Rapid Smart Contract Crawler

25 171 171

cocrawler

CoCrawler is a versatile web crawler built using modern tools and conc...

25 169 169

ScrapingOutsourcing

ScrapingOutsourcing专注分享爬虫代码尽量每周更新一个

44 168 168

python-dcdownloader

由Python编写的全异步实现的动漫之家(dmzj)漫画批量下载器（爬虫）

19 165 165

yispider

一款分布式爬虫平台，帮助你更好的管理和开发爬虫。内置一套爬虫定义规则...

31 163 163

mm131

MM131网站图片爬取 :rotating_light:

53 163 163

crypto-crawler-rs

A rock-solid cryptocurrency crawler library.

69 163 163

fun_crawler

Crawl some picture for fun

129 162 162

HttpCode.Core

简单、易用、高效一个有态度的开源.Net Http请求框架!可以用制作爬虫，api...

61 159 159

crawler-china-mainland-universities

中国大陆大学列表爬虫

49 159 159

TorCrawl.py

Crawl and extract (regular or onion) webpages through TOR network

43 159 159

NGCBot

一个基于✨HOOK机制的微信机器人，支持🌱安全新闻定时推送【FreeBuf，先知，...

31 158 158

soksaccounts

🔥 Shadowsocks 账号爬虫

50 157 157

Squidwarc

Squidwarc is a high fidelity, user scriptable, archival crawler that u...

28 156 156

DouyuBarrage-Pro

(2020年最新)斗鱼弹幕抓取及可视化管理平台第二版，提供弹幕抓取、弹幕实时...

25 156 156

evine

Interactive CLI Web Crawler

32 154 154

ngMeta

Dynamic meta tags in your AngularJS single page application

43 153 153

DotnetCrawler

DotnetCrawler is a straightforward, lightweight web crawling/scrapying...

54 153 153

webpalm

WebPalm is a powerful command-line tool for website mapping and web sc...

18 153 153

onecomic

一本漫画

33 152 152

urlbuster

Powerful mutable web directory fuzzer to bruteforce existing and/or hi...

33 151 151

jlitespider

A lite distributed Java spider framework :-)

39 150 150

ir

Projeto de calculo de Imposto de Renda em operacoes na bovespa automat...

39 150 150

crawler

Go process used to crawl websites

20 149 149

pkulaw_spider

爬取北大法宝网http://www.pkulaw.cn/Case/

57 149 149

D4N155

OWASP D4N155 - Intelligent and dynamic wordlist using OSINT

44 148 148

pachong

一些爬虫的代码

100 146 146

NewsCrawler

新闻爬虫，爬取新浪、搜狐、新华网即时财经新闻。

28 144 144

KTSpeechCrawler

Automatically constructing corpus for automatic speech recognition fro...

37 143 143

telegram-crawler

🕷 Automatically detect changes made to the official Telegram sites, cl...

16 143 143

crawley

The unix-way web crawler

8 143 143

pixiv_func_mobile

功能齐全的Pixiv第三方客户端免代理支持查看动图查看小说

9 142 142

fontObfuscator

字体混淆服务

14 141 141

aliexpress-product-scraper

Get Aliexpress product details as a json response including feedbacks,...

65 141 141

douban-movie

Golang爬虫爬取豆瓣电影Top250

67 140 140

taki

Take a snapshot of any website.

18 140 140

HotNewsAnalysis

利用文本挖掘技术进行新闻热点关注问题分析

46 139 139

bilibili_member_crawler

B站用户爬虫好耶~是爬虫

21 138 138

nebula

🌌 A libp2p DHT crawler, monitor, and measurement tool that exposes ti...

17 137 137

spider

:star2::octocat: powered by python3( simple learning of spider) 百度文...

65 136 136

not-your-average-web-crawler

A web crawler (for bug hunting) that gathers more than you can imagine...

36 136 136

CrawlBox

Easy way to brute-force web directory.

42 136 136

Zhihu-Spider

一个获取知乎用户主页信息的多线程Python爬虫程序。

54 135 135

site-audit-seo

Web service and CLI tool for SEO site audit: crawl site, lighthouse al...

18 134 134

Skill-Share-Crawler---DL

Download Videos Skill Share per ID or per Class

35 133 133

leetcode-ranking-search

Leetcode Contest Ranking Searcher

22 133 133

dyer

Dyer is designed for reliable, flexible and fast web crawling, providi...

14 131 131

onegram

This repository is no longer maintained.

5 130 130

php-crawler

A php crawler that finds emails on the internets

65 130 130

picacomic_downloader

哔咔漫画收藏夹下载程序

17 129 129