Most popular crawler repositories and open source projects

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python...

10965   57678   57678  

EasySpider

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器...

4968   40506   40506  

lux

👾 Fast and simple video download library and CLI tool written in Go

3153   29711   29711  

colly

Elegant Scraper and Crawler Framework for Golang

1815   24465   24465  

proxy_pool

Python ProxyPool for web spider

5297   22415   22415  

crawlee

Crawlee—A web scraping and browser automation library for Node.js to b...

889   18535   18535  

pyspider

A Powerful Spider(Web Crawler) System in Python.

3681   15945   15945  

newspaper

newspaper3k is a news, full-text, and article metadata extraction in P...

2130   14668   14668  

examples-of-web-crawlers

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、...

3842   14326   14326  

katana

A next-generation crawling and spidering framework.

745   13800   13800  

crawlab

Distributed web crawler admin platform for spiders management regardle...

1837   11756   11756  

webmagic

A scalable web crawler framework for Java.

4173   11533   11533  

Photon

Incredibly fast crawler designed for OSINT.

1409   9785   9785  

avbook

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆...

2026   9747   9747  

Python

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

4176   8463   8463  

spider-flow

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

1565   8149   8149  

awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing...

814   7082   7082  

node-crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

879   6766   6766  

autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

702   6716   6716  

awesome-crawler

A collection of awesome web crawler,spider in different languages

711   6646   6646  

pholcus

[Crawler for Golang] Pholcus is a distributed, high concurrency and po...

1502   6197   6197  

ferret

Declarative web scraping

307   5836   5836  

headless-chrome-crawler

Distributed crawler powered by Headless Chrome

409   5582   5582  

WechatSogou

基于搜狗微信搜索的微信公众号爬虫接口

1630   5541   5541  

scrapy-redis

Redis-based components for Scrapy.

1581   5307   5307  

haipproxy

:sparkling_heart: High available distributed ip proxy pool, powerd by...

945   5286   5286  

browser-fingerprinting

Analysis of Bot Protection systems with available countermeasures 🚿....

225   4247   4247  

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Cra...

288   4109   4109  

myGPTReader

A community-driven way to read and chat with AI bots - powered by chat...

421   4095   4095  

ECommerceCrawlers

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、...

1219   3969   3969  

dom-crawler

Eases DOM navigation for HTML and XML documents

123   3804   3804  

scylla

Intelligent proxy pool for Humans™

465   3746   3746  

DotnetSpider

DotnetSpider, a .NET standard web crawling library. It is lightweight,...

1002   3673   3673  

toapi

Every web site provides APIs.

235   3523   3523  

proxypool

自动抓取tg频道、订阅地址、公开互联网上的ss、ssr、vmess、trojan节点信息...

2558   3454   3454  

ProxyBroker

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:

951   3408   3408  

arachni

Web Application Security Scanner Framework

732   3405   3405  

TorBot

Dark Web OSINT Tool

585   3356   3356  

Douyin_TikTok_Download_API

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikTok...

612   3161   3161  

Crawler_Illegal_Cases_In_China

Collection of China illegal cases about web crawler 本项目用来整理所有...

250   3101   3101  

NGCBot

一个基于✨HOOK机制的微信机器人，支持🌱安全新闻定时推送【FreeBuf，先知...

392   2933   2933  

crawlergo

A powerful browser crawler for web vulnerability scanners

488   2932   2932  

owllook

owllook-小说搜索引擎

757   2766   2766  

geziyor

Geziyor, blazing fast web crawling & scraping framework for Go. Suppor...

152   2687   2687  

GoogleScraper

A Python module to scrape several search engines (like Google, Yandex,...

745   2683   2683  

DecryptLogin

DecryptLogin: APIs for loginning some websites by using requests.

737   2669   2669  

gospider

Gospider - Fast web spider written in Go

322   2655   2655  

QueryList

:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框...

435   2573   2573  

RED_HAWK

All in one tool for Information Gathering, Vulnerability Scanning and...

823   2532   2532  

gecco

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

890   2510   2510