crawler

Fast, free, open-source technical SEO + GEO crawler: built for humans and agents.

Scrapy + Puppeteer

wxpath - declarative web crawling with XPath; a Web Query Language (WQL)

A complimentary proxy to help to use SPM with headless browsers

Simple Weibo Scraper

Spider ported to Python

爬取B站历史弹幕/全弹幕, 支持高级弹幕, Bas弹幕爬取. [2025年]可用; 内部爬取算法可以在 最优最少 请求次数下爬取弹幕, 并且 不会 丢失任何弹幕. 支持多任务管...

爬虫, http代理, 模拟登陆!

This package is a complete tool for creating a large dataset of images (specially designed -but not only- for machine learning enthusiasts). It can cr...

✨ BOSE IS SWISS ARMY KNIFE 🔪 FOR BOT DEVELOPMENT. THE ULTIMATE BOT DEVELOPMENT FRAMEWORK. 🤖

一个网络安全法律法规、安全政策、国家标准、行业标准知识库。A knowledge base of cybersecurity laws and regulations, security policies, national standard...

Get Aliexpress product details in JSON

gocrawler, go分布式爬虫框架

免费 IP 代理池。Scrapy 爬虫框架插件

A command-line tool to crawl websites using puppeteer.

Grabs current REWE discounts and saves them in a markdown file || Holt sich aktuelle REWE-Angebote und exportiert sie in eine Markdown-Liste

Parsed data from website https://jadwalsholat.org

Java 網路資料爬蟲包

Google Arts & Culture high quality image downloader

使用asyncio和aiohttp开发的轻量级异步协程web爬虫框架

A universal solution for web crawling lists. 抓取网页列表的通用解决方案

Continuously search imageboards threads for images/webms and download them

Turn any developer documentation into a GPT

Python: An all-in-one Web Crawler, Web Parser and Web Scrapping library!

使 scrapy 开发不用在意 item，pipeline，middleware 等通用场景下模块的编写，解放开发者的双手。

A package to get list of user agents based on filters such as operating system, software name etc..

练手项目：Comment of Interest 电商文本评论数据挖掘 （爬虫 + 观点抽取 + 句子级和观点级情感分析）

🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs

A multiprocessing crawler for weibo albums.

A spider on Dcard. Strong and speedy.

Discover hidden deepweb pages

Crawl sites for RSS, Atom, and JSON feeds.

Draco is a faster, cheaper and lighter firecrawl drop-in replacement that you own entirely, running on your machine, with a built in MCP, free websear...

👋 HOLA! ENJOY OUR GOOGLE MAPS SCRAPER 🚀 TO EFFORTLESSLY EXTRACT DATA SUCH AS NAMES, ADDRESSES, PHONE NUMBERS, WEBSITES, AND RATINGS FROM GOOGLE MAPS...

Download audio tracks from Netflix to sample your favorite shows

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

Enterprise-grade, multi-threaded SEO Crawler, Rule Engine, and Link Graph Analyzer. Built in TypeScript for speed, compliance, and deep site health au...

:spider: This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragen...

Fast, highly configurable, cloud native dark web crawler.

An infinite Pinterest crawler/scraper. Crawl image with inifnite-scroll!

Scrapy-based Crawlers for news of Taiwan

a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine

All In One, Fast, Easy Recon Tool

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

A collection of Python tools, scripts and utilities to make your life easier.

lianjia / beike estate crawler/analysis 2024

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blo...

医疗知识图谱构建实战，通过爬虫获取百度百科数据，使用Mongodb存储结构化三元组，并使用neo4j进行知识图谱的构建及可视化; Medical Knowledge Graph; Crawler;...

MediaCrawler is a powerful web scraper for self-media platforms. Easily collect and analyze content to enhance your digital strategy. 🌐🕷️