crawler

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

A scalable, mature and versatile web crawler based on Apache Storm

SpiderSuite (web security crawler) releases, wiki and roadmap

✌️ Python3 BitTorrent DHT crawler

A high performance web crawler / scraper in Elixir.

Pure-Python protocol tools and research notes for X.com, with VibeLoft Twitter account datasets

一个方便安全研究人员获取每日安全日报的爬虫和推送程序，目前爬取范围包括先知社区、安全客、Seebug Paper、跳跳糖、奇安信攻防社区、棱角社区以及绿盟、腾讯玄...

A multi-thread crawler framework with many builtin image crawlers provided.

爬虫逆向案例，已完成：TLS指纹｜瑞数｜震坤行 | 网易易盾 | 微信小程序反编译逆向(百达星系) | 同花顺 | rpc解密 | 加速乐 | 极验滑块验证码 | 巨量算数 | Boss...

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key...

一个超级轻量的百度图片爬虫

A Tumblr Blog Backup Application

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers,...

High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验...

HTTP API for Scrapy spiders

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places...

Crawl BookCorpus

ArrowDL (Arrow Downloader) is a download manager for Windows, MacOS and Linux

Social media (Weibo) comments analyzing toolbox in Chinese 微博评论分析工具, 实现功能: 1.微博评论数据爬取; 2.分词与关键词提取; 3.词云与词频统计; 4.情...

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to...

Crawler (Bot) searching for credential leaks on paste sites.

Simple but useful Python web scraping tutorial code.

ai reverse 一把梭

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code...

SEO & Security Audit for Websites. Lighthouse & Security Headers crawler, Sitemap/Keywords/Images Extractor, Summarizer, etc ...

🎓 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载。

Java API For Chrome and Firefox

Doujinshi downloader 绅士漫画下载

🛑 image collector, which supports custom acquisition source configuration and is compatible with MacOS and Windows operating systems.

[Unmaintained] A simple and clean video/music/image downloader 👾

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

js cookie逆向利器：js cookie变动监控可视化工具 & js cookie hook打条件断点

Open source SEO audit tool.

:paw_prints: Creeper - The Next Generation Crawler Framework (Go)

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

《爬虫逆向进阶实战》书籍代码库

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百...

The Big Brother V6.0 is a weaponized OSINT platform featuring username enumeration (473+ platforms), quad-vector visual intelligence, Sky Radar tracki...

A lightweight web crawler framework.（Java爬虫框架）

:newspaper: Let ChatGPT Summarize Hacker News for You

Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabi...

A Tumblr and Twitter Blog Backup Application

The best PTT library

Wscan is a web security scanner that focuses on web security, dedicated to making web security accessible to everyone.

获取免费socks/https/http代理的网站集合

A Facebook crawler

OSINT Swiss Army Knife

Search google, bing, yahoo, and other search engines with python

A search application to explore, discover and share online files