Web Application Security Scanner Framework
Eases DOM navigation for HTML and XML documents
Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era
Headless Chrome .NET API
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单,功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、Batc...
多平台内容监控·采集·搬运 一个 Web 面板管起抖音 / 小红书 / 快手 / 视频号 / TikTok / 哔哩哔哩
Movie metadata scraper
Every web site provides APIs.
Python爬虫实战 - 模拟登陆各大网站 包含但不限于:滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝,如果喜欢请start ❤️
Videodl: A lightweight video downloader written in pure python. (轻量级视频下载器,优先高清无水印,支持抖音,快手,小红书,B站,TikTok,YouTube,FIFA+...
一个基于✨HOOK机制的微信机器人,支持🌱安全新闻定时推送【FreeBuf,先知,安全客,奇安信攻防社区】,👯Kfc文案,⚡漏洞查询,⚡手机号归属地查询,⚡知识库查...
All In One Web Recon
A powerful browser crawler for web vulnerability scanners
Gospider - Fast web spider written in Go
Node.js scraper to get data from Google Play
owllook-小说搜索引擎
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
Foundational low latency web data collecting in Rust
DecryptLogin: APIs for loginning some websites by using requests.
https://spatie.be/docs/crawler
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
基金投资管理回测引擎
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes
Easy to use lightweight web crawler(易用的轻量化网络爬虫)
scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot
news-please - an integrated web crawler and information extractor for news that just works
Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.
Web Scraping Framework
链家网和贝壳网房价爬虫,采集北京上海广州深圳等21个中国主要城市的房价数据(小区,二手房,出租房,新房),稳定可靠快速!支持csv,MySQL, MongoDB,Excel, js...
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
🏳️🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, BlueSky, TikTok, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub...
Transform Web Content into LLM-Ready Data
vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform a...
蓝天采集器是一款开源免费的爬虫系统,仅需点选编辑规则即可采集数据,可运行在本地、虚拟主机或云服务器中,几乎能采集所有类型的网页,无缝对接各类CMS建站程...
NewPipe's core library for extracting data from streaming sites
Polite, slim and concurrent web crawler.
Web crawling framework based on asyncio.
Find web directories without bruteforce
Dynamic server-side rendering using headless Chrome
爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷、爱奇艺、携程、12306、58、搜狐、各种指数、维普万方、...
2019新型冠状病毒疫情实时爬虫及API | COVID-19/2019-nCoV Realtime Infection Crawler and API
磁力網站U3C3介紹以及域名更新
浏览器内存漫游解决方案(探索中...)
To extract article from given URL