Most popular crawler repositories and open source projects

RED_HAWK

All in one tool for Information Gathering, Vulnerability Scanning and...

823 2532 2532

gecco

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

890 2510 2510

instagram-scraper

scrapes medias, likes, followers, tags and all metadata. Inspired by i...

398 2495 2495

Python3-Spider

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团...

972 2491 2491

weibo-crawler

新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频

646 2479 2479

lianjia-beike-spider

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数...

645 2459 2459

work_crawler

Download comics novels 小说漫画下载工具小説漫画のダウンローダ小說漫...

289 2451 2451

grab

Web Scraping Framework

275 2405 2405

crawler

An easy to use, powerful crawler implemented in PHP. Can execute Java...

342 2362 2362

news-please

news-please - an integrated web crawler and information extractor for...

443 2310 2310

abot

Cross Platform C# web crawler framework built for speed and flexibilit...

560 2289 2289

gain

Web crawling framework based on asyncio.

212 2022 2022

skycaiji

蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运...

596 2016 2016

gocrawl

Polite, slim and concurrent web crawler.

196 2015 2015

DXY-COVID-19-Crawler

2019新型冠状病毒疫情实时爬虫及API | COVID-19/2019-nCoV Realtime Infect...

403 2012 2012

rendora

Dynamic server-side rendering using headless Chrome

106 1996 1996

vulnx

vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, an...

343 1920 1920

dirhunt

Find web directories without bruteforce

265 1901 1901

feapder

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder...

350 1899 1899

spider

Web crawler and scraper for Rust

153 1898 1898

lxSpider

爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、...

450 1834 1834

go_spider

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework...

470 1827 1827

FinalRecon

The Last Web Recon Tool You'll Need

381 1815 1815

Crawler-Detect

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via...

234 1785 1785

PSpider

简单易用的Python爬虫框架，QQ交流群：597510560

516 1780 1780

bilix

⚡️Lightning-fast async download tool for bilibili and more

172 1759 1759

xalpha

基金投资管理回测引擎

478 1756 1756

SCrawler

🏳️‍🌈 Media downloader from any sites, including Twitter, Reddit, Inst...

121 1741 1741

x-crawl

Flexible Node.js AI-assisted crawler library

108 1720 1720

ruia

Async Python 3.6+ web scraping micro-framework based on asyncio

180 1688 1688

AutoCrawler

Google, Naver multiprocess image web crawler (Selenium)

425 1670 1670

diskover-community

Diskover Community Edition - Open source file indexer, file search eng...

169 1609 1609

CatVodTVSpider

930 1587 1587

NewPipeExtractor

NewPipe's core library for extracting data from streaming sites

487 1573 1573

scrapoxy

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxie...

222 1570 1570

lightcrawler

Crawl a website and run it through Google lighthouse

165 1474 1474

goclone

Website Cloner - Utilizes powerful Go routines to clone websites to y...

301 1456 1456

fscrawler

Elasticsearch File System Crawler (FS Crawler)

304 1407 1407

SwiftLinkPreview

It makes a preview from an URL, grabbing all the information such as t...

200 1385 1385

mlscraper

🤖 Scrape data from HTML websites automatically by just providing exam...

91 1359 1359

wombat

Lightweight Ruby web crawler/scraper with an elegant DSL which extract...

131 1293 1293

OpenWPM

A web privacy measurement framework

316 1281 1281

jd-autobuy

Python爬虫，京东自动登录，在线抢购商品

607 1270 1270

go-dork

The fastest dork scanner written in Go.

134 1246 1246

fakebrowser

🤖 Fake fingerprints to bypass anti-bot systems. Simulate mouse and ke...

213 1224 1224

AppCrawler

基于appium的app自动遍历工具

474 1217 1217

Beanbun

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展...

252 1211 1211

bilili

:beers: bilibili video (including bangumi) and danmaku downloader | B...

91 1185 1185

tumblr-crawler

Easily download all the photos/videos from tumblr blogs. 下载指定的 Tu...

353 1144 1144

fess

Fess is very powerful and easily deployable Enterprise Search Server.

168 1056 1056