scraping

🔥 The Web Data API for AI - Power AI agents with clean web data

Scrapy, a fast high-level web crawling & scraping framework for Python.

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

AIHawk aims to easy job hunt process by automating the job application process. Utilizing artificial intelligence, it enables users to apply for multi...

Elegant Scraper and Crawler Framework for Golang

Python scraper based on AI

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs,...

🕵️‍♂️ Collect a dossier on a person by username from 3000+ sites

Pythonic HTML Parsing for Humans™

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

A scalable web crawler framework for Java.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, P...

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

List of libraries, tools and APIs for web scraping and data processing.

Tabula is a tool for liberating data tables trapped inside PDF files

Convert cURL commands to Python, JavaScript, Java, C#, PHP, Go, Dart, R, Ruby, Rust, MATLAB, Elixir, CFML, Ansible, Strest or JSON

Declarative web scraping

Distributed crawler powered by Headless Chrome

Swiss-army tool for scraping and extracting data from online assets, made for hackers

Mechanize is a ruby library that makes automated web interaction easy.

Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

Collection of useful data science topics along with articles, videos, and code

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Up-to-date simple useragent faker with real world database

Snoop — инструмент разведки на основе открытых данных (OSINT world)

🦊 Anti-detect browser

Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons...

A browser testing and web crawling library for PHP and Symfony

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Scrape Facebook public pages without an API key

A curated list of awesome puppeteer resources.

Web Scraping Framework

Web crawler and scraper for Rust

Getting started with Puppeteer and Chrome Headless for Web Scraping

Advanced Privacy Browser Core with Unified Fingerprint Defense: Cloudflare, Akamai, Kasada, Shape, DataDome, PerimeterX, hCaptcha, FunCaptcha, Imperva...

Get info from any web service or page

A command-line utility for taking automated screenshots of websites

Internet-in-a-Box - Build your own LIBRARY OF ALEXANDRIA with a Raspberry Pi !

A Modern Search Engine API for Anime, Movies/TVShows, Books, Light Novels, Manga, etc.

Hide your scrapers IP behind the cloud. Provision proxy servers across different cloud providers to improve your scraping success.

Example end to end data engineering project.

Linkedin Automation Tool: Describe your product. Define your target market. The AI finds the leads for you.

🤖 Scrape data from HTML websites automatically by just providing examples

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy...

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements an...

Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.

Scrape tweets, profiles, followers and following from Twitter/X, no API key needed. Python library with smart multi-account pooling, proxy support and...

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.