crawling

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EM...

Scrapy, a fast high-level web crawling & scraping framework for Python.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs,...

Elegant Scraper and Crawler Framework for Golang

Turn any website into a structured API. Extract, automate, search and monitor the web.

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, P...

List of libraries, tools and APIs for web scraping and data processing.

A Chrome DevTools Protocol driver for web automation and scraping.

Declarative data automation language and Go runtime for structured extraction workflows.

Distributed crawler powered by Headless Chrome

Exa MCP for web search and web crawling!

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

A list of AI agents and robots to block.

Headless Chrome .NET API

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Apache Nutch is an extensible and scalable web crawler

A curated list of awesome puppeteer resources.

Web Scraping Framework

📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告

蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程...

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy...

The complete web scraping toolkit for PHP.

🤖 Scrape data from HTML websites automatically by just providing examples

Run a high-fidelity browser-based web archiving crawler in a single Docker container

<6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)>의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디...

Crawly, a high-level web crawling & scraping framework for Elixir.

Scalable Python web scraping scripts for +40 popular domains

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers,...

Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.

Scrapy middleware to handle javascript pages using selenium

today we will hack the admin panel of the site.

HTTP API for Scrapy spiders

Open source SEO audit tool.

Simple but useful Python web scraping tutorial code.

[Unmaintained] A simple and clean video/music/image downloader 👾

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

Extract structured data from web sites. Web sites scraping.

100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by you...

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

Python 3 script to dump/scrape/extract company employees from LinkedIn API

ISP Data Pollution to Protect Private Browsing History with Obfuscation

Scrapy Extension for monitoring spiders execution.

a reliable high-level web crawling & scraping framework for Node.js.

Crawljax

Second-order subdomain takeover scanner

WarcDB: Web crawl data as SQLite databases.

🕸️ Crawl in the web network

Library for Rapid (Web) Crawler and Scraper Development

Tracks changes to Telegram's sites, beta apps and servers