crawler

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg...

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with on...

Scrapy, a fast high-level web crawling & scraping framework for Python.

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：Ser...

Playwright MCP server undetected by anti-bots and captchas: AI agent browses the web on anti-detect stealth Firefox, Python, undetected browser automa...

👾 Fast and simple video download library and CLI tool written in Go

Python scraper based on AI

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs,...

Elegant Scraper and Crawler Framework for Golang

Python ProxyPool for web spider

🚀 抖音、TikTok 数据采集与无水印视频下载 API，自托管，支持 MCP 调用与 Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video do...

Turn any website into a structured API. Extract, automate, search and monitor the web.

A next-generation crawling and spidering framework.

A Powerful Spider(Web Crawler) System in Python.

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are...

Incredibly fast crawler designed for OSINT.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

A scalable web crawler framework for Java.

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Jap...

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, P...

为OPC/中小微企业量身打造的自媒体获客智能体

List of libraries, tools and APIs for web scraping and data processing.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

开源、免费、跨平台的 B 站视频下载工具，支持多线程加速、音视频分离、弹幕元数据获取、自定义命名与分类等功能。Open Source, Free, Cross-Platform Bilibili...

Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀

A collection of awesome web crawler,spider in different languages

Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

基于搜狗微信搜索的微信公众号爬虫接口

[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.

Declarative data automation language and Go runtime for structured extraction workflows.

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景...

Redis-based components for Scrapy.

Distributed crawler powered by Headless Chrome

:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵...

Dark Web OSINT Tool

Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作...

新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频

A community-driven way to read and chat with AI bots - powered by chatGPT.

Download comics novels 小说漫画下载工具 小説漫画のダウンローダ 小說漫畫下載:腾讯漫画 大角虫漫画 有妖气 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫...

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Web Application Security Scanner Framework

Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes