Topic

crawler

Repositories (1501)

kimuraframework
kimuraframework vifreefly Ruby

Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes w...

1.1k
FictionDown
FictionDown ma6254 Go

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

1.1k
awesome-ai-reverse
awesome-ai-reverse darbra

ai reverse 逆向一把梭

1k
google-play-scraper
google-play-scraper JoMingyu Python

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

1k
siteone-crawler
siteone-crawler janreges Rust

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers,...

1k
stormcrawler
stormcrawler apache Java

A scalable, mature and versatile web crawler based on Apache Storm

1k
Pxer
Pxer pea3nut JavaScript

A tool for pixiv.net. 人人可用的P站爬虫

1k
SpiderSuite
SpiderSuite spidersuite

SpiderSuite (web security crawler) releases, wiki and roadmap

990
magnet-dht
magnet-dht chenjiandongx Python

✌️ Python3 BitTorrent DHT crawler

967
crawler
crawler fredwu Elixir

A high performance web crawler / scraper in Elixir.

962
spider_reverse
spider_reverse 0xAllenChen Python

爬虫逆向案例,已完成:TLS指纹|瑞数|震坤行 | 网易易盾 | 微信小程序反编译逆向(百达星系) | 同花顺 | rpc解密 | 加速乐 | 极验滑块验证码 | 巨量算数 | Boss...

946
x-kit
x-kit xiaoxiunique Python

Pure-Python protocol tools and research notes for X.com, with VibeLoft Twitter account datasets

946
TikHub-API-Python-SDK
TikHub-API-Python-SDK TikHub Python

High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验...

943
SecCrawler
SecCrawler Le0nsec Go

一个方便安全研究人员获取每日安全日报的爬虫和推送程序,目前爬取范围包括先知社区、安全客、Seebug Paper、跳跳糖、奇安信攻防社区、棱角社区以及绿盟、腾讯玄...

937
icrawler
icrawler hellock Python

A multi-thread crawler framework with many builtin image crawlers provided.

936
zhihu-crawler
zhihu-crawler wycm Java

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

918
chatWeb
chatWeb SkywalkerDarren Python

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key...

916
BaiduImageSpider
BaiduImageSpider kong36088 Python

一个超级轻量的百度图片爬虫

913
TumblThree
TumblThree johanneszab C#

A Tumblr Blog Backup Application

906
scrapyrt
scrapyrt scrapinghub Python

HTTP API for Scrapy spiders

884
Scavenger
Scavenger rndinfosecguy Python

Crawler (Bot) searching for leaks on paste sites.

883
antibrow
antibrow antibrow TypeScript

Kernel-level antidetect browser with the Playwright API you already write. Python + Node SDKs, MCP-ready, unlimited local profiles. Linux x64 + arm64,...

882
skrape.it
skrape.it skrapeit Kotlin

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places...

874
seonaut
seonaut StJudeWasHere Go

Open source SEO audit tool.

873
bookcorpus
bookcorpus soskek Python

Crawl BookCorpus

866
ArrowDL
ArrowDL setvisible C++

ArrowDL (Arrow Downloader) is a download manager for Windows, MacOS and Linux

855
Krawl
Krawl BlessedRebuS Python

Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabi...

844
Weibo-Analyst
Weibo-Analyst KimMeen Python

Social media (Weibo) comments analyzing toolbox in Chinese 微博评论分析工具, 实现功能: 1.微博评论数据爬取; 2.分词与关键词提取; 3.词云与词频统计; 4.情...

838
spidr
spidr postmodern Ruby

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to...

835
seo-audits-toolkit
seo-audits-toolkit StanGirard Python

SEO & Security Audit for Websites. Lighthouse & Security Headers crawler, Sitemap/Keywords/Images Extractor, Summarizer, etc ...

829
easy-scraping-tutorial
easy-scraping-tutorial MorvanZhou Jupyter Notebook

Simple but useful Python web scraping tutorial code.

825
fortress
fortress tiliondev Python

Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

817
till
till DataHenHQ Go

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code...

815
xeHentai
xeHentai fffonion Python

Doujinshi downloader 绅士漫画下载

808
jvppeteer
jvppeteer fanyong920 Java

Java API For Chrome and Firefox

804
course-crawler
course-crawler Foair Python

🎓 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载。

804
Lulu
Lulu iawia002 Python

[Unmaintained] A simple and clean video/music/image downloader 👾

802
pic-gather
pic-gather Licoy

🛑 image collector, which supports custom acquisition source configuration and is compatible with MacOS and Windows operating systems.

801
linkedin-profile-scraper-api
linkedin-profile-scraper-api josephlimtech TypeScript

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

794
fetchbot
fetchbot PuerkitoBio Go

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

790
TheBigBrother
TheBigBrother chadi0x Python

The Big Brother is a weaponized OSINT platform featuring username enumeration (473+ platforms), quad-vector visual intelligence, Sky Radar tracking, c...

786
js-cookie-monitor-debugger-hook
js-cookie-monitor-debugger-hook JSREI TypeScript

js cookie逆向利器:js cookie变动监控可视化工具 & js cookie hook打条件断点

779
creeper
creeper wspl Go

:paw_prints: Creeper - The Next Generation Crawler Framework (Go)

777
lxBook
lxBook lixi5338619 JavaScript

《爬虫逆向进阶实战》书籍代码库

770
BaiduSpider
BaiduSpider BaiduSpider Python

BaiduSpider,一个爬取百度搜索结果的爬虫,目前支持百度网页搜索,百度图片搜索,百度知道搜索,百度视频搜索,百度资讯搜索,百度文库搜索,百度经验搜索和百...

765
xxl-crawler
xxl-crawler xuxueli Java

A lightweight web crawler framework.(Java爬虫框架)

756
hacker-news-digest
hacker-news-digest polyrabbit Python

:newspaper: Let LLM Summarize Hacker News for You

754
TumblThree
TumblThree TumblThreeApp C#

A Tumblr and Twitter Blog Backup Application

744
PyPtt
PyPtt PyPtt Python

The best PTT library

733
Free_Proxy_Website
Free_Proxy_Website cyubuchen Python

获取免费socks/https/http代理的网站集合

724