crawling

Web scraper with a simple REST API living in Docker and using a Headless browser and Readability.js for parsing.

Stop stalking and start StopStalking :wink:

Lightweight web scraping toolkit for documents and structured data.

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

一个灵活、友好的爬虫框架

An Instagram bot developed using the Selenium Framework

Create you virus in termux!

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Laravel adapter for Roach, the complete web scraping toolkit for PHP.

SpideyX a multipurpose Web Penetration Testing tool with asynchronous concurrent performance with multiple mode and configurations.

Experience for effectively fetching Facebook data by Querying Graph API with Account-based Token and Operating undetectable scraping Bots to extract C...

Automated web scraping spider generation using Browser Use and LLMs. Streamline the creation of Playwright-based spiders with minimal manual coding. I...

네이버 뉴스 수집을 위한 도구

Crawler for linguistic corpora

Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them...

estela, an elastic web scraping cluster 🕸

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library des...

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

Download a large list of files concurrently

High-performance web crawling engine with bindings for 11 languages

A fast, modern and intelligent proxy rotator perfect for crawling and scraping public data.

Go process used to crawl websites

Sasori is a dynamic web crawler powered by Puppeteer, designed for lightning-fast endpoint discovery.

cdp4j - Chrome DevTools Protocol for Java

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless...

Parse through any sitemap in Node.js

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

Scraply a simple dom scraper to fetch information from any html based website

SimFin's open source PDF crawler

A simple Python script to crawl complete list of LinkedIn skills

🌱 goClone - clone websites in seconds

🗄️ A simple CLI for converting WARC to Parquet.

An asyncio + aiolibs crawler imitate scrapy framework

使用 Scrapy 写成的 JK 爬虫，图片源自哔哩哔哩、Tumblr、Instagram，以及微博、Twitter

qcrawl - fast async web crawling & scraping framework for Python.

Burp Suite's extension to scan and crawl Single Page Applications

Download DIG to run on your laptop or server.

Turn any developer documentation into a GPT

使 scrapy 开发不用在意 item，pipeline，middleware 等通用场景下模块的编写，解放开发者的双手。

Crawl sites for RSS, Atom, and JSON feeds.

Fast, highly configurable, cloud native dark web crawler.

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different website...

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

Powerful web scraping framework for Crystal

Build a small, 3 domain internet using Github pages and Wikipedia and construct a crawler to crawl, render, and index.

RAG Web Browser is an Apify Actor to feed your LLM applications and RAG pipelines with up-to-date text content scraped from the web.

Web crawling and document processing through a usable interface.