crawler
Repositories (1431)
LinkedIn Scraper (currently working 2020)
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Germany on platforms like ImmoScout24...
Moodle-DL downloads course content fast from Moodle (eg. lecture pdfs)
Free Web Scraping Tool with Java
Prying Deep - An OSINT tool to collect intelligence on the dark web.
Simple yet powerful automation stuffs.
a reliable high-level web crawling & scraping framework for Node.js.
👩 美女写真套图爬虫（一）
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
swiss army knife for hackers
Crawljax
Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
🌈Python3网络爬虫实战：QQ音乐歌曲、京东商品信息、房天下、破解有道翻译、构建代理池、豆瓣读书、百度图片、破解网易登录、B站模拟扫码登录、小鹅通、荔枝微课
Crawler for Nintendo Switch eShop
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabi...
Open-source Enterprise Grade Search Engine Software
a new crawler based on python with more function including Network fingerprint search
Crawl and extract (regular or onion) webpages through TOR network
A framework for creating semi-automatic web content extractors
Open-source, production-grade web scraping engine built for LLMs. Scrape and crawl the entire web, clean markdown, ready for your agents.
Html网页正文提取
⚡ 一款用于自动语音识别 (ASR)、翻译的高性能异步 API。不需要购买Whisper API，使用本地运行的Whisper模型进行推理，并支持多GPU并发，针对分布式部署进行设计...
Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.
台灣股票即時爬蟲。Taiwan Stock Exchange Real Time Crawler
Easily create XML sitemaps for your website.
A very simple news crawler with a funny name
澎湃新闻，新浪新闻，腾讯新闻，搜狐新闻，新闻联播，泰晤士报，纽约时报，BBCNews，旨在爬取所有新闻门户网站的新闻，禁止将所得数据商用！
This repository contains all the code I use in my YouTube tutorials.
🕵️ Python project to crawl for JavaScript files and search for secrets like API keys, authorization tokens, hardcoded credentials, etc.
Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok
各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)
Coomer| kemono .party or su downloader
dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators
Search emails from a domain through search engines
golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。
:musical_note: 缓存文件转换为 MP3 文件
Selenium Open Source Search Engine & crawler
Android 本地网络小说爬虫，基于jsoup及xpath
Google search results crawler, get google search results that you need
A search engine that doesn't track you.
Second-order subdomain takeover scanner
Crawl GitHub APIs and store the discovered orgs, repos, commits, ...
ScraperAI is an open-source, AI-powered tool designed to simplify web scraping for users of all skill levels.
Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.
A non API python program to crawl public photos, posts or followers
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limi...
JS逆向Hook工具集，开源部分工具到这里
🕸️ Crawl in the web network
今日头条机器人，支持用户登陆、关注、取消关注、获取关注粉丝、发文、发悟空问答、点赞、评论、采集各种类型新闻讯息等，使用今日头条网页版API实现
FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理，支持爬取数十个免费代理分享源，支持自定义规则代理筛选，爬虫与数据分析必备，...