crawler

talospider - A simple,lightweight scraping micro-framework

[Updated] A simple python crawler for my tutorial blog at http://www.jianshu.com/p/8fb5bc33c78e

Crawl Instagram hashtags

Read an Amazon wishlist programmatically with Python

Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc.

使用RxJava2 和 Java 8的特性开发的图片爬虫

A web search engine built with Python which uses TF-IDF and PageRank to sort search results.

Node.js price monitoring library, leveraging the power of x-ray and nightmare.

Scrape public Facebook pages, posts, reviews and comments

gRPC web crawler turbo charged for performance

一句话监控网页内容变化，AI | 爬虫 | 网页监控 | 网页更新提醒 | 网页内容订阅

微信公众号爬虫，以API方式提供公众号文章获取，包括阅读量、点赞等

NewsCrap adalah alat scraping berita Google berbasis Command Line Interface (CLI) yang dirancang untuk riset, investigasi, dan pengumpulan data OSINT....

我的爬虫合集

Deep web crawler and search engine

Crawl a website to generate knowledge file for RAG

Kal El Network Stress Test and Penetration Testing Toolkit

A web browser :earth_americas: hosted as a service, to render your JavaScript web pages as HTML

Shopee coin getter is a script to collect daily shopee coins.

CLI to download all images/webms in a 4chan thread

RARBG command line interface for scraping the rarbg.to torrent search engine

uforall is a fast url crawler this tool crawl all URLs number of different sources, alienvault,WayBackMachine,urlscan,commoncrawl

Shared Go infrastructure for local-first crawler archives.

Python script to download messages from a Facebook page to a CSV file

Continuous scalable web crawler built on top of Flink and crawler-commons

网页解析器，用于网络爬虫解析页面, 不懂网页解析也能写爬虫

Scrapfly Python SDK for headless browsers and proxy rotation

⚡ A subdomain enumeration tool leveraging diverse techniques, designed for advanced pentesting operations

开盘啦 App 数据抓取与解析工具 - 批量抓取板块/个股数据，Token自动更新，mitmproxy流量拦截

Riichi Mahjong Kit: (1) Game log crawler (sqlite3, json, bs4); (2) Game log preprocessor; (3) Deterministic algorithms library

分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微服务架构，通过消息队列实现消息的异步发送，使用到的框架包括：redigo, gorm, goquer...

抓取twitter数据，可根据时间、话题、用户名等条件抓取数据，twitter爬虫

Armiarma is a Libp2p open-network crawler with a current focus on Ethereum's CL network

12306查票助手，一键查询沿途所有站点，先上车后补票，让你的出行更省心。

百度莱茨狗爬虫。

支付宝账单爬虫

Scrapy, a fast high-level web crawling & scraping framework for dart and Flutter

基于规则的跨平台一站式聚合搜索工具

search & get datas from youtube no google account needed

Crawl 100%-discount games on steam

facebook-messenger-bot-tutorial use Python Django

A web service that turns an arbitrary web page into structural JSON data and easy-to-use APIs with just a few clicks

A fluent and functional approach to querying HTML

NASTY Advanced Search Tweet Yielder

简单、实用的爬虫工具，仅需四步创建属于你的爬虫程序！

Extract instagram users informations from hashtags. This scraper can extract emails addresses from Bio section and business email.

Crawl novels from sfacg, ciweimao, esjzone, lightnovel and masiro; generate, append and extract epub

Next Crawler 是使用Playwright + Next.js + Prisma等主流技术搭建的网页数据采集器，通过可视化的UI进行配置，即可周期性的通过Playwright驱动浏览器爬取网页数...