crawler
Repositories (1431)
台灣上市上櫃股票爬蟲 Taiwan Stock Exchange Crawler
Lightweight scraper for Google News
JavaScript + BeautifulSoup = JSSoup
🌌 An agnostic network crawler exposing comprehensive peer information and network topology information.
Library for Rapid (Web) Crawler and Scraper Development
微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等...
爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
News crawling with StormCrawler - stores content as WARC
一个自我托管的 JavBus API 服务
An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.
An R web crawler and scraper
基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料
使用python+copymanga API来下载copymanga(拷贝漫画)中的漫画(无速率限制)，支持批量+选话下载和获取您收藏的漫画并下载及半自动获取订阅下载！(全平台支持(pypi...
Crawl telegra.ph searching for nudes!
中国场外基金数据爬取&汇总分析
知乎模拟登录，支持提取验证码和保存 Cookies
Independent search engine. Includes web crawling, search indexing, dictionary API, and more. https://vyntr.com
The simple, easy to use command line web crawler.
🕷 Automatically detect changes made to the official Telegram sites, clients and servers.
🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览
web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppetee...
快速、简洁且强大的PHP爬虫框架
Creates an XML-Sitemap by crawling a given site.
🍭 epub generator for (light)novels (轻)小说 epub 生成器，支持站点：轻之国度、轻小说文库
Tiktok Downloader/Scraper using requests & bs4
The unix-way web crawler
Be nice on the web
为超级个体和一个人公司打造一个人的中台，Hydra九头龙构筑大规模AI调度、数据采集、情报系统、数据平台、分析决策、产品生产的'军事'工业基座。
本仓库收集整理爬虫相关资源，开发语言以Java为主
A Laravel wrapper for CrawlerDetect - the web crawler detection library
This is a wencai crawler.（i问财的策略回测接口的Pythonic工具包）
Crawl csgo skin info from `buff.163.com` and steam, then find the most suitable one to buy from the former and to sell to the latter.
🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)
Web scraper with a simple REST API living in Docker and using a Headless browser and Readability.js for parsing.
Use LLMs to robustly extract web data
Python3 script to continuously download all images/videos of multiple 4chan threads simultaneously - without installation
In this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move...
微博超级话题爬虫，微博词频统计+情感分析+简单分类，新增肺炎超话爬取数据
直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀
Bitextor generates translation memories from multilingual websites
JS逆向研究
line-bot-tutorial use python flask
蝦皮非同步爬蟲 + 競品賣家分析
一个灵活、友好的爬虫框架
Python Lambda Chrome Automation (naming pending)
Web service and CLI tool for SEO site audit: crawl site, lighthouse all pages, view public reports in browser. Also output to console, json, csv, xlsx
An image crawler written in Python.
Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.
Gorecon is a All in one Reconnaissance Tool , a.k.a swiss knife for Reconnaissance , A tool that every pentester/bughunter might wanna consider into...