crawler

一个网络安全法律法规、安全政策、国家标准、行业标准知识库。A knowledge base of cybersecurity laws and regulations, security policies, national standard...

A multiprocessing crawler for weibo albums.

Discover hidden deepweb pages

使 scrapy 开发不用在意 item，pipeline，middleware 等通用场景下模块的编写，解放开发者的双手。

👋 HOLA! ENJOY OUR GOOGLE MAPS SCRAPER 🚀 TO EFFORTLESSLY EXTRACT DATA SUCH AS NAMES, ADDRESSES, PHONE NUMBERS, WEBSITES, AND RATINGS FROM GOOGLE MAPS...

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine

:spider: This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragen...

Collect, Download, Organize and Share your Favorite Anime Artworks.

🎓a Better BlackBoard for PKUers. 北京大学教学网命令行工具（🖥️Win/🐧Linux/🍏Mac）, 支持查看/提交作业、下载课程回放.

Fast, highly configurable, cloud native dark web crawler.

CrawlAI RAG is an AI-powered website intelligence platform that allows users to crawl entire websites, index their content, and ask natural-language q...

Scrapy-based Crawlers for news of Taiwan

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs

lianjia / beike estate crawler/analysis 2024

The LAW next generation crawler.

Crawl sites for RSS, Atom, and JSON feeds.

A collection of Python tools, scripts and utilities to make your life easier.

MediaCrawler is a powerful web scraper for self-media platforms. Easily collect and analyze content to enhance your digital strategy. 🌐🕷️

这是一个用Python写的小说爬虫软件

Price tracker of Amazon

Calendar of Public Holidays in China 中国大陆节假日日历订阅 自动节假日闹钟

爬取B站历史弹幕/全弹幕, 支持高级弹幕, Bas弹幕爬取. [2025年]可用; 内部爬取算法可以在 最优最少 请求次数下爬取弹幕, 并且 不会 丢失任何弹幕. 支持多任务管...

医疗知识图谱构建实战，通过爬虫获取百度百科数据，使用Mongodb存储结构化三元组，并使用neo4j进行知识图谱的构建及可视化; Medical Knowledge Graph; Crawler;...

A simple Elixir library for writing decently-performing crawlers with minimum effort.

An infinite Pinterest crawler/scraper. Crawl image with inifnite-scroll!

Selenium automation test framework

使用 SpringBoot2.0+ElasticSearch 实现的开源电影搜索引擎

A Python crawler tool that can automatically simulate browser operations to crawl all users' tweet content and save all static resources (videos, pict...

All In One, Fast, Easy Recon Tool

extract data from html table

Extract structured data from Shopify websites.

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blo...

ScrapeGPT is a RAG-based Telegram bot designed to scrape and analyze websites, then answer questions based on the scraped content. The bot utilizes Re...

Crawl and convert any website into clean markdown

Crawl & Visualize ICLR 2023 Data from OpenReview

Download audio tracks from Netflix to sample your favorite shows

爬取及整理Freebuf\安全客\先知\知道创宇等站点的”web安全“类优质文章

带图形界面的哔哩哔哩漫画下载工具

Dynamic configurable crawl (动态可配置化爬虫)

An intelligent web service to automatically detect web content and extract information from it.

web scraping extension

Crawler behind the Shopify App Marketplace dataset

Verify that a request is from Google crawlers using Google's DNS verification steps

Google Maps crawler using Selenium. All extracted data is forwarded to a SQS queue.

Fast website scraper and wordlist generator

PHP Metacritic API - Mirror from my GitLab

This repo is a part of blog series on several web scraping projects where we will explore scraping techniques to crawl data from simple websites to we...

轻量、异步、开箱即用的社交媒体聚合解析库