Topic

crawler

Repositories (1501)

x-crawl
x-crawl coder-hxl TypeScript

Flexible Node.js crawler library

1.9k
diskover-community
diskover-community diskoverdata PHP

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

1.8k
PSpider
PSpider xianhu Python

简单易用的Python爬虫框架,QQ交流群:597510560

1.8k
go_spider
go_spider hu17889 Go

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized c...

1.8k
bilix
bilix HFrost0 Python

⚡️Lightning-fast async download tool for bilibili and more

1.8k
boss-zhipin-scraper
boss-zhipin-scraper eatmoreduck Python

Boss直聘爬虫 / BOSS直聘职位数据抓取工具,基于 Chrome CDP 协议复用真实登录态,绕过字体反爬,输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BO...

1.8k
ruia
ruia howie6879 Python

Async Python 3.6+ web scraping micro-framework based on asyncio

1.7k
selectolax
selectolax rushter Python

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

1.7k
single-file-cli
single-file-cli gildas-lormeau JavaScript

CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

1.7k
AutoCrawler
AutoCrawler YoongiKim Python

Google, Naver multiprocess image web crawler (Selenium)

1.7k
spider_collection
spider_collection srx-2000 Python

python爬虫,目前库存:网易云音乐歌曲爬取,B站视频爬取,知乎问答爬取,壁纸爬取,xvideos视频爬取,有声书爬取,微博爬虫,安居客信息爬取+数据可视化,哔哩...

1.7k
fredy
fredy orangecoding JavaScript

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24,...

1.6k
douyin
douyin erma0

抖音爬虫——采集账号主页、喜欢、收藏、音乐原声、话题、搜索、合集、作品、关注、粉丝等公开数据。

1.6k
grab-site
grab-site ArchiveTeam Python

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

1.6k
CatVodTVSpider
CatVodTVSpider CatVodTVOfficial Java
1.6k
lightcrawler
lightcrawler github JavaScript

Crawl a website and run it through Google lighthouse

1.6k
scrapoxy
scrapoxy fabienvauchelles JavaScript

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

1.6k
wreq-python
wreq-python 0x676e67 Python

An ergonomic, privacy-aware Python HTTP Client

1.5k
rebrowser-patches
rebrowser-patches rebrowser JavaScript

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy...

1.5k
fscrawler
fscrawler dadoonet Java

Elasticsearch File System Crawler (FS Crawler)

1.5k
XHS-Spider
XHS-Spider xisuo67

小红书数据采集、网站图片、视频资源批量下载工具,颜值超高的数据采集工具(批量下载,视频提取,图片)Telegram:https://t.me/+ZtLSwuIKTo44MDY1

1.5k
OpenWPM
OpenWPM openwpm Python

A web privacy measurement framework

1.4k
sperm
sperm darbra

浏览过的精彩逆向文章汇总,值得一看

1.4k
crawler
crawler kgspider JavaScript

K 哥爬虫代码分享,JS 逆向,爬虫进阶。关注公众号:K哥爬虫

1.4k
mlscraper
mlscraper lorey Python

🤖 Scrape data from HTML websites automatically by just providing examples

1.4k
SwiftLinkPreview
SwiftLinkPreview leonardocardoso Swift

It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.

1.4k
wombat
wombat felipecsl Ruby

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

1.4k
instagram-profilecrawl
instagram-profilecrawl InstaPy Python

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

1.3k
go-dork
go-dork dwisiswant0 Go

The fastest dork scanner written in Go.

1.3k
XSRFProbe
XSRFProbe 0xInfection Python

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

1.3k
jd-autobuy
jd-autobuy adyzng Python

Python爬虫,京东自动登录,在线抢购商品

1.3k
crw
crw fastcrw Rust

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible...

1.3k
python-fxxk-spider
python-fxxk-spider ityard

收集各种免费的 Python 爬虫项目

1.3k
Beanbun
Beanbun kiddyuchina PHP

Beanbun 是用 PHP 编写的多进程网络爬虫框架,具有良好的开放性、高可扩展性,基于 Workerman。

1.3k
AppCrawler
AppCrawler seveniruby Scala

基于appium的app自动遍历工具

1.2k
mzitu
mzitu chenjiandongx Python

👧 美女写真套图爬虫(二)

1.2k
sqliv
sqliv the-robot Python

massive SQL injection vulnerability scanner

1.2k
BiliBili-Manga-Downloader
BiliBili-Manga-Downloader Zeal-L Python

一个好用的哔哩哔哩漫画下载器,拥有图形界面,支持关键词搜索漫画和二维码登入,黑科技下载未解锁章节,多线程下载,多种保存格式,本地漫画管理,一键检查更新...

1.2k
fakebrowser
fakebrowser kkoooqq JavaScript

🤖 Fake fingerprints to bypass anti-bot systems. Simulate mouse and keyboard operations to make behavior like a real person.

1.2k
browsertrix-crawler
browsertrix-crawler webrecorder TypeScript

Run a high-fidelity browser-based web archiving crawler in a single Docker container

1.2k
Browser4
Browser4 platonai Kotlin

Browser4 — an AI-native browser engine for autonomous agents, intelligent extraction, and large-scale web automation.

1.2k
bilili
bilili yutto-dev Python

:beers: bilibili video (including bangumi) and danmaku downloader | B站视频(含番剧)、弹幕下载器

1.2k
Pandora-Box
Pandora-Box snakem982 Vue

A Simple Mihomo GUI. 一个简易的 Mihomo 桌面客户端

1.2k
newspaper4k
newspaper4k AndyTheFactory Python

📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.

1.2k
tumblr-crawler
tumblr-crawler dixudx Python

Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片,视频

1.2k
fess
fess codelibs Java

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/...

1.2k
scrapfly-scrapers
scrapfly-scrapers scrapfly Python

Scalable Python web scraping scripts for +80 popular domains

1.1k
crawly
crawly elixir-crawly Elixir

Crawly, a high-level web crawling & scraping framework for Elixir.

1.1k
wreq
wreq 0x676e67 Rust

An ergonomic, privacy-aware Rust HTTP Client

1.1k
parse-video
parse-video wujunwei928 Go

Golang短视频去水印:抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频...

1.1k