Most popular crawler repositories and open source projects

SpiderBOX WuKongSecurity CSS

SpiderBox - 虫盒 - 爬虫逆向资源导航站

153 28 4

pinscrape iamatulsingh Python

A simple library to scrape Pinterest images.

152 31 1

IpProxyPool wuchunfu Go

Golang 实现的 IP 代理池, 涉及到的技术点: go gorm proxy proxypool ip crawler 爬虫 mysql viper cobra

152 37 3

ComputerStudent sfvsfv HTML

计算机专业系统性学习资料（python,c,c++,计算机组成，计算机网络，编译原理，电路，谷歌插件，爬虫）

151 55 151

ngMeta vinaygopinath JavaScript

Dynamic meta tags in your AngularJS single page application

150 43 7

WebReaper alex-on-ai C#

AI-native web scraper. Single binary with a bundled Claude Code skill. MIT-licensed alternative to Firecrawl.

150 33 3

go-crawler lizongying Go

A web crawling framework implemented in Golang, it is simple to write and delivers powerful performance. It comes with a wide range of practical middl...

150 20 3

crawler trandoshan-io Go

Go process used to crawl websites

150 21 1

pku3b sshwy Rust

🎓a Better BlackBoard for PKUers. 北京大学教学网命令行工具（🖥️Win/🐧Linux/🍏Mac）

149 19 1

Zhihu-Spider moranzcw Python

一个获取知乎用户主页信息的多线程Python爬虫程序。

149 50 11

not-your-average-web-crawler tijme Python

A web crawler (for bug hunting) that gathers more than you can imagine.

149 36 11

Darkweb-search-engine NexvisionLab Python

Dark web & deep web search engine, crawler, and indexer — open-source OSINT tooling for Tor-based darknet reconnaissance.

149 35 8

pylinkvalidator bartdag Python

pylinkvalidator is a standalone and pure python link validator and crawler that traverses a web site and reports errors (e.g., 500 and 404 errors) enc...

148 36 8

crawler_detect loadkpi Ruby

Ruby gem to detect bots and crawlers via the user agent

148 15 2

GoodreadsScraper havanagrawal Python

Scrape data from Goodreads using Scrapy and Selenium :books:

148 39 3

jlitespider luohaha Java

A lite distributed Java spider framework :-)

147 37 16

KamiYomu KamiYomu C#

A self-hosted, extensible manga reader and download tool with plug-in support.

146 9 1

sasori karthikuj JavaScript

Sasori is a dynamic web crawler powered by Puppeteer, designed for lightning-fast endpoint discovery.

146 17 4

spiderbuf hhuayuan Python

Spiderbuf 是一个专注于 Python 爬虫练习的网站。提供丰富的爬虫教程、爬虫案例解析和爬虫练习题。Python爬虫开发强化练习，在矛与盾的攻防中不断提高技术水平，...

145 14 1

moe-copy-ai yusixian TypeScript

✨ 萌萌哒的 AI 网页数据提取助手 ✨

145 11 2

fuckBookWalker VermiIIi0n Python

Download books from bookwalker.jp/bookwalker.com.tw

145 12 7

pachong jin10086 Jupyter Notebook

一些爬虫的代码

145 92 9

npm-search algolia TypeScript

🗿 npm ↔️ Algolia replication tool :skier: :snail: :artificial_satellite:

143 23 61

GoodBots AnTheMaker

Updated lists of IP addresses/whitelists of good bots and crawlers. Includes GoogleBot, BingBot, DuckDuckBot, etc.

143 28 6

bilibili_member_crawler cwjokaka Python

B站用户爬虫好耶~是爬虫

143 21 2

auto-lighthouse TGiles HTML

A utility package for automating lighthouse reporting

142 18 2

convertible-bond-crawler jackluson HTML

宁稳网(旧富投网)、集思录可转债数据&策略分析

142 59 2

agent-line-bot Lin-jun-xiang Python

🤖Free Agent Line Bot with Web Search, Google Image Search, Image Generator, Video Generator...

142 146 2

pixiv_func_mobile git-xiaocao Dart

功能齐全的Pixiv第三方客户端免代理支持查看动图查看小说

142 9 142

poopak teal33t Python

POOPAK - TOR Hidden Service Crawler

141 34 7

pricetrack duyet JavaScript

Price tracker monitors of products and alerts you when prices drop. Supported tiki.vn, shopee, lotte.vn, ... Built with firebase https://pricetrack.we...

140 49 1

PHPCreeper blogdaren PHP

A new generation of multi-process async event-driven spider engine based on workerman. Support headless browser. 🌿基于workerman实现的多进程异步事件...

140 15 6

Ceiba-Downloader jameshwc Python

This is a course-downloader to help NTU students download courses data from NTU Ceiba.

139 9 1

taki egoist TypeScript

Take a snapshot of any website.

139 18 1

blinkist-m4a-downloader luckylittle Go

Grabs all of the audio files from all of the Blinkist books

138 26 12

wget-lua ArchiveTeam C

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

138 19 19

sitemapper seantomburke TypeScript

Parse through any sitemap in Node.js

137 84 6

WeiboSpider CharesFang Python

微博爬虫，一个基于Scrapy框架的轻量微博爬虫，Sina Weibo Spider

137 28 4

PatentCrawler will4906 Python

scrapy专利爬虫（停止维护）

137 71 1

php-crawler hedii PHP

A php crawler that finds emails on the internets

136 62 1

docs zhangslob

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题；HTTP协议介绍； Requests使用；解析器Xpath介绍； MongoDB与MySQL；多线程...

136 30 2

Terpene-Profile-Parser-for-Cannabis-Strains MaxValue Python

Parser and database to index the terpene profile of different strains of Cannabis from online databases

136 18 17

web-scout-mcp pinkpixel-dev JavaScript

An MCP server providing web search and content extraction capabilities. Integrates DuckDuckGo search functionality and URL content extraction into you...

134 17 2