crawler
Repositories (1232)
Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。
:beers: bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器
Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片，视频
Fess is very powerful and easily deployable Enterprise Search Server.
📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.
Crawly, a high-level web crawling & scraping framework for Elixir.
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
massive SQL injection vulnerability scanner
👧 美女写真套图爬虫（二）
跨平台的 B 站视频下载工具，支持 Windows、Linux、macOS 三平台，下载 B 站视频/番剧/电影/纪录片等资源。
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests a...
浏览器内存漫游解决方案（探索中...）
A tool for pixiv.net. 人人可用的P站爬虫
A high performance web crawler / scraper in Elixir.
SpiderSuite releases, wiki and roadmap
A scalable, mature and versatile web crawler based on Apache Storm
一个方便安全研究人员获取每日安全日报的爬虫和推送程序，目前爬取范围包括先知社区、安全客、Seebug Paper、跳跳糖、奇安信攻防社区、棱角社区以及绿盟、腾讯玄...
zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目
一个超级轻量的百度图片爬虫
A Tumblr Blog Backup Application
ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key...
✌️ Python3 BitTorrent DHT crawler
HTTP API for Scrapy spiders
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places...
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to...
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code...
🎓 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载。
[Unmaintained] A simple and clean video/music/image downloader 👾
Simple but useful Python web scraping tutorial code.
🛑 image collector, which supports custom acquisition source configuration and is compatible with MacOS and Windows operating systems.
浏览过的精彩逆向文章汇总，值得一看
:paw_prints: Creeper - The Next Generation Crawler Framework (Go)
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百...
A multi-thread crawler framework with many builtin image crawlers provided.
A lightweight web crawler framework.（Java爬虫框架）
SEO & Security Audit for Websites. Lighthouse & Security Headers crawler, Sitemap/Keywords/Images Extractor, Summarizer, etc ...
The best PTT library
A Tumblr and Twitter Blog Backup Application
Crawl BookCorpus
Doujinshi downloader 绅士漫画下载
🕵️♂️ LinkedIn profile scraper returning structured profile data in JSON.
ArrowDL (Arrow Downloader) is a download manager for Windows, MacOS and Linux
A search application to explore, discover and share online files
K 哥爬虫代码分享，JS 逆向，爬虫进阶。关注公众号：K哥爬虫
OSINT Swiss Army Knife
Crawler (Bot) searching for credential leaks on paste sites.
NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。
python爬虫，目前库存：网易云音乐歌曲爬取，B站视频爬取，知乎问答爬取，壁纸爬取，xvideos视频爬取，有声书爬取，微博爬虫，安居客信息爬取+数据可视化，哔哩...