crawler
Repositories (1232)
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频
链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, js...
Download comics novels 小说漫画下载工具 小説漫画のダウンローダ 小說漫畫下載:腾讯漫画 大角虫漫画 有妖气 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫...
Web Scraping Framework
An easy to use, powerful crawler implemented in PHP. Can execute Javascript.
news-please - an integrated web crawler and information extractor for news that just works
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Web crawling framework based on asyncio.
蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程...
Polite, slim and concurrent web crawler.
2019新型冠状病毒疫情实时爬虫及API | COVID-19/2019-nCoV Realtime Infection Crawler and API
Dynamic server-side rendering using headless Chrome
vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform a...
Find web directories without bruteforce
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、Batc...
Web crawler and scraper for Rust
爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷、爱奇艺、携程、12306、58、搜狐、各种指数、维普万方、...
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized c...
The Last Web Recon Tool You'll Need
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
简单易用的Python爬虫框架，QQ交流群：597510560
⚡️Lightning-fast async download tool for bilibili and more
基金投资管理回测引擎
🏳️🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, BlueSky, TikTok, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub...
Flexible Node.js AI-assisted crawler library
Async Python 3.6+ web scraping micro-framework based on asyncio
Google, Naver multiprocess image web crawler (Selenium)
Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch
NewPipe's core library for extracting data from streaming sites
Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!
Crawl a website and run it through Google lighthouse
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
Elasticsearch File System Crawler (FS Crawler)
It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.
🤖 Scrape data from HTML websites automatically by just providing examples
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
A web privacy measurement framework
Python爬虫，京东自动登录，在线抢购商品
The fastest dork scanner written in Go.
🤖 Fake fingerprints to bypass anti-bot systems. Simulate mouse and keyboard operations to make behavior like a real person.
基于appium的app自动遍历工具
Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。
:beers: bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器
Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片，视频
Fess is very powerful and easily deployable Enterprise Search Server.