Topic

crawler

Repositories (1501)

wscan
wscan chushuai Go

Wscan is a web security scanner that focuses on web security, dedicated to making web security accessible to everyone.

717
fbcrawl
fbcrawl rugantio Python

A Facebook crawler

693
gOSINT
gOSINT Nhoya Go

OSINT Swiss Army Knife

679
Search-Engines-Scraper
Search-Engines-Scraper tasos-py Python

Search google, bing, yahoo, and other search engines with python

677
FileMasta
FileMasta ohhsodead C#

A search application to explore, discover and share online files

669
Moodle-DL
Moodle-DL C0D3D3V Python

Moodle-DL downloads course content fast from Moodle (eg. lecture pdfs)

668
Craw4LLM
Craw4LLM cxcscmu Python

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

667
DouYin
DouYin Python3WebSpider Python

API of DouYin for Humans used to Crawl Popular Videos and Musics

653
learnPython
learnPython rieuse Python

Python的基础练习代码与各种爬虫代码

648
NetDiscovery
NetDiscovery fengzhizi715 Java

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

643
pywebcopy
pywebcopy rajatomar788 Python

Locally saves webpages to your hard disk with images, css, js & links as is.

638
catbus
catbus cv-cat TypeScript

一个命令操作小红书、抖音、TikTok、B 站、快手、微博、闲鱼、淘宝、京东、X:统一的命令与 JSON 输出,给人和 AI Agent 用

637
runoob-PDF-
runoob-PDF- gagayuan Python

爬取菜鸟教程网站并转PDF__python_crawer_by_chrome

637
Antibot-Detector
Antibot-Detector scrapfly JavaScript

Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Se...

636
pryingdeep
pryingdeep iudicium Go

Prying Deep - An OSINT tool to collect intelligence on the dark web.

625
dotcommon
dotcommon Kharacternyk Python

What do people have in their dotfiles?

620
go_jobs
go_jobs go-crawler Go

带你了解一下Golang的市场行情

617
scrapedin
scrapedin linkedtales JavaScript

LinkedIn Scraper (currently working 2020)

612
Jie
Jie yhy0 Go

Jie stands out as a comprehensive security assessment and exploitation tool meticulously crafted for web applications. Its robust suite of features en...

610
freeproxy
freeproxy CharlesPikachu Python

FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理,支持爬取数十个免费代理分享源,支持自定义规则代理筛选,爬虫与数据分析必备,...

592
newcrawler
newcrawler speed JavaScript

Free Web Scraping Tool with Java

585
mmjpg
mmjpg chenjiandongx Python

👩 美女写真套图爬虫(一)

579
reader
reader vakra-dev TypeScript

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.

576
python-automation-scripts
python-automation-scripts avidLearnerInProgress Python

Simple yet powerful automation stuffs.

561
vault
vault abhisharma404 Python

swiss army knife for hackers

560
webster
webster zhuyingda JavaScript

a reliable high-level web crawling & scraping framework for Node.js.

560
crawljax
crawljax crawljax Java

Crawljax

559
TorCrawl.py
TorCrawl.py MikeMeliz Python

Crawl and extract (regular or onion) webpages through TOR network

548
freshonions-torscraper
freshonions-torscraper dirtyfilthy Python

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

543
ticket-monitoring
ticket-monitoring dengmeiluan Python

✈️ 自托管的多渠道机票价格监控:航班明细采集 → 钉钉图文推送 → 达标四层强提醒。国内航线与国际航线(五洲目的地)同一套采集-判定-推送链路;数据留在本机,...

529
Python3Webcrawler
Python3Webcrawler mochazi Python

🌈Python3网络爬虫实战:QQ音乐歌曲、京东商品信息、房天下、破解有道翻译、构建代理池、豆瓣读书、百度图片、破解网易登录、B站模拟扫码登录、小鹅通、荔枝微课

529
nintendo-switch-eshop
nintendo-switch-eshop lmmfranco TypeScript

Crawler for Nintendo Switch eShop

523
opensearchserver
opensearchserver jaeksoft Java

Open-source Enterprise Grade Search Engine Software

516
scrapple
scrapple AlexMathew Python

A framework for creating semi-automatic web content extractors

506
Scan-T
Scan-T nanshihui C

a new crawler based on python with more function including Network fingerprint search

500
scraperai
scraperai scraperai HTML

ScraperAI is an open-source, AI-powered tool designed to simplify web scraping for users of all skill levels.

497
Html2Article
Html2Article stanzhai C#

Html网页正文提取

496
Fast-Powerful-Whisper-AI-Services-API
Fast-Powerful-Whisper-AI-Services-API Evil0ctal Python

⚡ 一款用于自动语音识别 (ASR)、翻译的高性能异步 API。不需要购买Whisper API,使用本地运行的Whisper模型进行推理,并支持多GPU并发,针对分布式部署进行设计...

482
fundus
fundus flairNLP Python

A very simple news crawler with a funny name

481
AllNewsSpider
AllNewsSpider Python3Spiders Python

澎湃新闻,新浪新闻,腾讯新闻,搜狐新闻,新闻联播,泰晤士报,纽约时报,BBCNews,旨在爬取所有新闻门户网站的新闻,禁止将所得数据商用!

470
tsrtc
tsrtc Asoul JavaScript

台灣股票即時爬蟲。Taiwan Stock Exchange Real Time Crawler

462
ICLR2020-OpenReviewData
ICLR2020-OpenReviewData shaohua0116 Jupyter Notebook

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

459
sitemap-generator
sitemap-generator lgraubner JavaScript

Easily create XML sitemaps for your website.

453
coom-dl
coom-dl notFaad Dart

Coomer| kemono .party or su downloader

441
Pinkerton
Pinkerton dsssssssm Python

🕵️ Python project to crawl for JavaScript files and search for secrets like API keys, authorization tokens, hardcoded credentials, etc.

439
signature_algorithm
signature_algorithm gadfly0x Python

各种App、小程序、网站的请求签名或加密算法。 现已有:自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

433
media-scraper
media-scraper elvisyjlin Python

Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok

432
EmailFinder
EmailFinder Josue87 Python

Search emails from a domain through search engines

428
dude
dude roniemartinez Python

dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators

426
Youtube-Projects
Youtube-Projects ayushi7rawat Python

This repository contains all the code I use in my YouTube tutorials.

426