crawler

一个自我托管的 JavBus API 服务

A non API python program to crawl public photos, posts or followers

🕸️ Crawl in the web network

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limi...

JS逆向Hook工具集，开源部分工具到这里

🌌 An agnostic network crawler exposing comprehensive peer information and network topology information.

今日头条机器人，支持用户登陆、关注、取消关注、获取关注粉丝、发文、发悟空问答、点赞、评论、采集各种类型新闻讯息等，使用今日头条网页版API实现

台灣上市上櫃股票爬蟲 Taiwan Stock Exchange Crawler

Lightweight scraper for Google News

Crawl telegra.ph searching for nudes!

News crawling with StormCrawler - stores content as WARC

使用python+copymanga API来下载copymanga(拷贝漫画)中的漫画(无速率限制)，支持批量+选话下载和获取您收藏的漫画并下载及半自动获取订阅下载！(全平台支持(pypi...

JavaScript + BeautifulSoup = JSSoup

Library for Rapid (Web) Crawler and Scraper Development

微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等...

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

中国场外基金数据爬取&汇总分析

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

An R web crawler and scraper

Independent search engine. Includes web crawling, search indexing, dictionary API, and more. https://vyntr.com

An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.

知乎模拟登录，支持提取验证码和保存 Cookies

🕷 Automatically detects changes to the official Telegram sites, beta clients, MTProto servers and mini apps

The simple, easy to use command line web crawler.

Real-time detection of anti-bot systems, CAPTCHAs & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Se...

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览

🍭 epub generator for (light)novels (轻)小说 epub 生成器，支持站点：轻之国度、轻小说文库

快速、简洁且强大的PHP爬虫框架

Creates an XML-Sitemap by crawling a given site.

The unix-way web crawler

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppetee...

Tiktok Downloader/Scraper using requests & bs4

Be nice on the web

SEO/GEO toolkit to analyse, crawl, parse and optimise websites & logs (Nginx & Apache)

一个可在线运行的微信视频号加密视频解密工具和 API 服务，基于逆向工程分析实现。本项目使用微信官方的 WebAssembly (WASM) 模块来生成 Isaac64 PRNG 密钥流，...

This is a wencai crawler.（i问财的策略回测接口的Pythonic工具包）

本仓库收集整理爬虫相关资源，开发语言以Java为主

Web scraper with a simple REST API living in Docker and using a Headless browser and Readability.js for parsing.

A Laravel wrapper for CrawlerDetect - the web crawler detection library

Crawl csgo skin info from `buff.163.com` and steam, then find the most suitable one to buy from the former and to sell to the latter.

Use LLMs to robustly extract web data

Python3 script to continuously download all images/videos of multiple 4chan threads simultaneously - without installation

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

蝦皮非同步爬蟲 + 競品賣家分析

Get Aliexpress product details as a json response including feedbacks, variants, shipping info, description, images, etc.,

In this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move...

JS逆向研究

GPlay Scraper is a powerful Python Google Play scraper library for extracting comprehensive app data from the Google Play Store. Scrape Google Play St...

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀