Wscan is a web security scanner that focuses on web security, dedicated to making web security accessible to everyone.
A Facebook crawler
OSINT Swiss Army Knife
Search google, bing, yahoo, and other search engines with python
A search application to explore, discover and share online files
Moodle-DL downloads course content fast from Moodle (eg. lecture pdfs)
Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"
API of DouYin for Humans used to Crawl Popular Videos and Musics
Python的基础练习代码与各种爬虫代码
NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。
Locally saves webpages to your hard disk with images, css, js & links as is.
一个命令操作小红书、抖音、TikTok、B 站、快手、微博、闲鱼、淘宝、京东、X:统一的命令与 JSON 输出,给人和 AI Agent 用
爬取菜鸟教程网站并转PDF__python_crawer_by_chrome
Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Se...
Prying Deep - An OSINT tool to collect intelligence on the dark web.
What do people have in their dotfiles?
带你了解一下Golang的市场行情
LinkedIn Scraper (currently working 2020)
Jie stands out as a comprehensive security assessment and exploitation tool meticulously crafted for web applications. Its robust suite of features en...
FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理,支持爬取数十个免费代理分享源,支持自定义规则代理筛选,爬虫与数据分析必备,...
Free Web Scraping Tool with Java
👩 美女写真套图爬虫(一)
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
Simple yet powerful automation stuffs.
swiss army knife for hackers
a reliable high-level web crawling & scraping framework for Node.js.
Crawljax
Crawl and extract (regular or onion) webpages through TOR network
Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
✈️ 自托管的多渠道机票价格监控:航班明细采集 → 钉钉图文推送 → 达标四层强提醒。国内航线与国际航线(五洲目的地)同一套采集-判定-推送链路;数据留在本机,...
🌈Python3网络爬虫实战:QQ音乐歌曲、京东商品信息、房天下、破解有道翻译、构建代理池、豆瓣读书、百度图片、破解网易登录、B站模拟扫码登录、小鹅通、荔枝微课
Crawler for Nintendo Switch eShop
Open-source Enterprise Grade Search Engine Software
A framework for creating semi-automatic web content extractors
a new crawler based on python with more function including Network fingerprint search
ScraperAI is an open-source, AI-powered tool designed to simplify web scraping for users of all skill levels.
Html网页正文提取
⚡ 一款用于自动语音识别 (ASR)、翻译的高性能异步 API。不需要购买Whisper API,使用本地运行的Whisper模型进行推理,并支持多GPU并发,针对分布式部署进行设计...
A very simple news crawler with a funny name
澎湃新闻,新浪新闻,腾讯新闻,搜狐新闻,新闻联播,泰晤士报,纽约时报,BBCNews,旨在爬取所有新闻门户网站的新闻,禁止将所得数据商用!
台灣股票即時爬蟲。Taiwan Stock Exchange Real Time Crawler
Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.
Easily create XML sitemaps for your website.
Coomer| kemono .party or su downloader
🕵️ Python project to crawl for JavaScript files and search for secrets like API keys, authorization tokens, hardcoded credentials, etc.
各种App、小程序、网站的请求签名或加密算法。 现已有:自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)
Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok
Search emails from a domain through search engines
dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators
This repository contains all the code I use in my YouTube tutorials.