crawler
Repositories (1431)
Headless Chrome .NET API
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、Batc...
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
Movie metadata scraper
Every web site provides APIs.
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
一个基于✨HOOK机制的微信机器人，支持🌱安全新闻定时推送【FreeBuf，先知，安全客，奇安信攻防社区】，👯Kfc文案，⚡漏洞查询，⚡手机号归属地查询，⚡知识库查...
A powerful browser crawler for web vulnerability scanners
Gospider - Fast web spider written in Go
DecryptLogin: APIs for loginning some websites by using requests.
owllook-小说搜索引擎
Node.js scraper to get data from Google Play
https://spatie.be/docs/crawler
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
All In One Web Recon
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot
基金投资管理回测引擎
链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, js...
Web Scraping Framework
news-please - an integrated web crawler and information extractor for news that just works
Web crawler and scraper for Rust
Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform a...
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程...
Polite, slim and concurrent web crawler.
Web crawling framework based on asyncio.
🏳️🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, BlueSky, TikTok, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub...
Dynamic server-side rendering using headless Chrome
Find web directories without bruteforce
2019新型冠状病毒疫情实时爬虫及API | COVID-19/2019-nCoV Realtime Infection Crawler and API
爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷、爱奇艺、携程、12306、58、搜狐、各种指数、维普万方、...
浏览器内存漫游解决方案（探索中...）
To extract main article from given URL with Node.js
简单易用的Python爬虫框架，QQ交流群：597510560
磁力網站U3C3介紹以及域名更新
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized c...
Flexible Node.js AI-assisted crawler library
Transform Web Content into LLM-Ready Data
NewPipe's core library for extracting data from streaming sites
Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch
⚡️Lightning-fast async download tool for bilibili and more
Async Python 3.6+ web scraping micro-framework based on asyncio
Google, Naver multiprocess image web crawler (Selenium)