crawler

Scrapy, a fast high-level web crawling & scraping framework for Python.

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：Service...

👾 Fast and simple video download library and CLI tool written in Go

Elegant Scraper and Crawler Framework for Golang

Python ProxyPool for web spider

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs,...

A Powerful Spider(Web Crawler) System in Python.

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are...

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

A next-generation crawling and spidering framework.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

A scalable web crawler framework for Java.

Incredibly fast crawler designed for OSINT.

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Jap...

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

List of libraries, tools and APIs for web scraping and data processing.

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

A collection of awesome web crawler,spider in different languages

[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.

Declarative web scraping

Redis-based components for Scrapy.

Distributed crawler powered by Headless Chrome

基于搜狗微信搜索的微信公众号爬虫接口

:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵...

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

A community-driven way to read and chat with AI bots - powered by chatGPT.

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景...

Web Application Security Scanner Framework

Eases DOM navigation for HTML and XML documents

Intelligent proxy pool for Humans™

Every web site provides APIs.

Dark Web OSINT Tool

自动抓取tg频道、订阅地址、公开互联网上的ss、ssr、vmess、trojan节点信息，聚合去重测试可用性后提供节点列表

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:

一个基于✨HOOK机制的微信机器人，支持🌱安全新闻定时推送【FreeBuf，先知，安全客，奇安信攻防社区】，👯Kfc文案，⚡漏洞查询，⚡手机号归属地查询，⚡知识库查...

Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作...

A powerful browser crawler for web vulnerability scanners

DecryptLogin: APIs for loginning some websites by using requests.

owllook-小说搜索引擎

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Gospider - Fast web spider written in Go

:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Node.js scraper to get data from Google Play