crawler

Selenium Open Source Search Engine & crawler

:musical_note: 缓存文件转换为 MP3 文件

Android 本地网络小说爬虫，基于jsoup及xpath

Google search results crawler, get google search results that you need

A search engine that doesn't track you.

一个自我托管的 JavBus API 服务

为超级个体和一个人公司打造一个人的大厂，Hydra九头龙构筑大规模AI调度、数据采集、情报系统、数据平台、分析决策、产品生产的'军事'工业基座。

Crawl GitHub APIs and store the discovered orgs, repos, commits, ...

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

A non API python program to crawl public photos, posts or followers

🕸️ Crawl in the web network

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limi...

一个可在线运行的微信视频号加密视频解密工具和 API 服务，基于逆向工程分析实现。本项目使用微信官方的 WebAssembly (WASM) 模块来生成 Isaac64 PRNG 密钥流，...

JS逆向Hook工具集，开源部分工具到这里

Lightweight scraper for Google News

🇰🇷 한국 정부 지원사업 전수조사 에이전트 스킬 [Claude Code·Codex·agy(Antigravity)·Cursor·Gemini CLI·Grok Build 지원] K-Startup·기업마당·NIPA·KOCCA·SMTE...

Crawl telegra.ph searching for nudes!

🌌 An agnostic network crawler exposing comprehensive peer information and network topology information.

Always up-to-date list of the top 100 most common browser user-agents for HTTP clients.

今日头条机器人，支持用户登陆、关注、取消关注、获取关注粉丝、发文、发悟空问答、点赞、评论、采集各种类型新闻讯息等，使用今日头条网页版API实现

News crawling with StormCrawler - stores content as WARC

台灣上市上櫃股票爬蟲 Taiwan Stock Exchange Crawler

JavaScript + BeautifulSoup = JSSoup

Library for Rapid (Web) Crawler and Scraper Development

使用python+copymanga API来下载copymanga(拷贝漫画)中的漫画(无速率限制)，支持批量+选话下载和获取您收藏的漫画并下载及半自动获取订阅下载！(全平台支持(pypi...

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等...

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

Independent search engine. Includes web crawling, search indexing, dictionary API, and more. https://vyntr.com

🕷 Automatically detects changes to the official Telegram sites, beta clients, MTProto servers and mini apps

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

中国场外基金数据爬取&汇总分析

An R web crawler and scraper

An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.

知乎模拟登录，支持提取验证码和保存 Cookies

The simple, easy to use command line web crawler.

🍭 epub generator for (light)novels (轻)小说 epub 生成器，支持站点：轻之国度、轻小说文库

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览

SEO/GEO toolkit to analyse, crawl, parse and optimise websites & logs (Nginx & Apache)

快速、简洁且强大的PHP爬虫框架

Creates an XML-Sitemap by crawling a given site.

The unix-way web crawler

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppetee...

Tiktok Downloader/Scraper using requests & bs4

Be nice on the web

Web scraper with a simple REST API living in Docker and using a Headless browser and Readability.js for parsing.

A Laravel wrapper for CrawlerDetect - the web crawler detection library

This is a wencai crawler.（i问财的策略回测接口的Pythonic工具包）

本仓库收集整理爬虫相关资源，开发语言以Java为主

蝦皮非同步爬蟲 + 競品賣家分析