scraping

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EM...

Scrapy, a fast high-level web crawling & scraping framework for Python.

🕵️‍♂️ Collect a dossier on a person by username from 6K websites

Python scraper based on AI

Open-source, self-hosted alternative to OpenAI Dots, Grok Bot. Built to be undetectable by anti-bot systems.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs,...

Elegant Scraper and Crawler Framework for Golang

Pythonic HTML Parsing for Humans™

🦊 Anti-detect browser

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

A scalable web crawler framework for Java.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, P...

List of libraries, tools and APIs for web scraping and data processing.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Tabula is a tool for liberating data tables trapped inside PDF files

Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

🚀 Free HTTP, SOCKS4, & SOCKS5 proxy list * Updated every 5 minutes * and rotating proxy API (100+ countries)

Declarative data automation language and Go runtime for structured extraction workflows.

Convert cURL commands to Python, JavaScript, Java, C#, PHP, Go, Dart, R, Ruby, Rust, MATLAB, Elixir, CFML, Ansible, Strest or JSON

Distributed crawler powered by Headless Chrome

Self-hosted webscraper.

Swiss-army tool for scraping and extracting data from online assets, made for hackers

Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

Mechanize is a ruby library that makes automated web interaction easy.

Collection of useful data science topics along with articles, videos, and code

Snoop — инструмент разведки на основе открытых данных (OSINT world)

Up-to-date simple useragent faker with real world database

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Nat...

Open-source AI agent for B2B lead generation — describe your product, it finds the people who fit, explains why each one does, and emails them from yo...

Scrape Facebook public pages without an API key

Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons...

An open source fingerprint browser based on Ungoogled Chromium. 指纹浏览器 隐私浏览器

A browser testing and web crawling library for PHP and Symfony

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity...

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Foundational low latency web data collecting in Rust

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

Advanced Privacy Browser Core with Unified Fingerprint Defense: Cloudflare, Akamai, Kasada, Shape, DataDome, PerimeterX, hCaptcha, FunCaptcha, Imperva...

Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.

A CLI utility for taking screenshots of websites, recording video demos and scraping sites using JavaScript

A curated list of awesome puppeteer resources.

Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Py...

Web Scraping Framework

Getting started with Puppeteer and Chrome Headless for Web Scraping

OSINT cheat sheet, list OSINT tools, wiki, dataset, article, book , red team OSINT for hackers and OSINT tips and OSINT branch. This repository will g...