Topic

crawler

Repositories (1501)

ipfs-crawler
ipfs-crawler trudi-group Go

A crawler for the IPFS network, code for our paper (https://arxiv.org/abs/2002.07747). Also holds scripts to evaluate the obtained data and make simil...

70
Tor_Spider
Tor_Spider absingh31 Python

Python project to crawl and scrap the lesser known deep web or one can say dark web. Just provide the onion link and get started.

70
tiktok-scraper-php
tiktok-scraper-php snuzi PHP

Tiktok (Musically) PHP scraper

70
crawlkit
crawlkit openclaw Go

Shared Go infrastructure for local-first crawler archives.

69
SecretScraper
SecretScraper PadishahIII Python

SecretScraper is a web scraper that crawl through target websites, scrape from http response and extract secret information via regular expression. Ru...

69
bilibili_comment_crawl
bilibili_comment_crawl lemonmindyes Python

爬取bilibili视频下的评论,最新出品!!!⚠本代码只适用于学习,做其他事情概不负责!!!

69
Crawling-CV-Conference-Papers
Crawling-CV-Conference-Papers seanywang0408 Jupyter Notebook

Crawling CV conference papers with Python.

69
robotstxt
robotstxt ropensci R

R 📦 for parsing and checking robots.txt files 🤖

69
BOJ-AutoCommit
BOJ-AutoCommit ISKU Python

When you solve the problem of Baekjoon Online Judge, it automatically commits and pushes to the remote repository.

69
spider
spider jhao104 Python

python crawler spider

69
Taiwan-Stocks
Taiwan-Stocks smalldan1022 Python

台灣上市櫃公司爬蟲,分析盤後股票趨勢以及繪製K線圖、均線圖、三大法人成交量

68
slime
slime nekolr Java

🍰 A visual crawler management platform

68
lezhin-comics-downloader
lezhin-comics-downloader ImSejin Java

📥 Downloader for lezhin comics

68
JewelCrawler
JewelCrawler DMinerJackie Java

豆瓣电影爬虫——a crawler which is able to crawl movie detail and short comments, save them to database mysql, also include Sentiment analysis based on...

68
bibliaAveMariaJSON
bibliaAveMariaJSON fidalgobr

Bíblia católica "Ave Maria" em formato JSON

67
chat-plugin-web-crawler
chat-plugin-web-crawler lobehub HTML

🧩 / 🕸 WebsiteCrawler - This plugin automatically crawls the main content of a specified URL webpage and uses it as context input.

67
nest-crawler
nest-crawler saltyshiomix TypeScript

An easiest crawling and scraping module for NestJS

67
bolsa
bolsa gicornachini Python

Biblioteca feita em Python com o objetivo de facilitar o acesso a dados de seus investimentos na bolsa de valores(B3/CEI) através do Portal CEI.

67
js_block
js_block webcoding HTML

研究学习各种拦截:反爬虫、拦截ad、防广告注入、斗黄牛等

67
hproxy
hproxy howie6879 Python

hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)

67
schinza-wechat-certificate
schinza-wechat-certificate Alexxxxxxxxxxxxy Python

最新微信公众号文章抓取工具,修复7月29号微信公众号接口下架后不能抓取的问题,支持windows、MacOS系统

66
github-trending
github-trending doforce Python

GitHub trending repositories and developers APIs for real time, powered by crawlers | 通过爬虫获取 GitHub 热门项目和开发者的实时 API

66
python-crawler
python-crawler ityouknow Python

Python Crawler

66
zhihu-crawler
zhihu-crawler NightMarcher Python

徒手实现定时爬取知乎,从中发掘有价值的信息,并可视化爬取的数据作网页展示。

66
dht-crawler
dht-crawler hijkzzz Go

A DHT Crawler based on Goroutine

65
free-llm-intel
free-llm-intel rockbenben Python

AI 厂商动态别挨家追:每天巡检 OpenAI、Claude、Gemini、DeepSeek、通义、Kimi 等的官方博客与更新页,生成可订阅的 RSS,英文原文逐段精译为中文,并附 70 家...

64
directed_graph
directed_graph simphotonics Dart

Dart implementation of a directed graphs and graph crawler. Provides algorithms for sorting vertices, retrieving a topological ordering and detecting...

64
novel-downloader
novel-downloader yjqiang Python

万能小说下载器

64
SoFIFA
SoFIFA DiogoDantas Jupyter Notebook

A SoFIFA webcrawler and Machine Learning prediction

64
medium-crawler
medium-crawler NISH1001 Python

A crawler for scraping posts from medium.com

64
crawdad
crawdad schollz Go

Cross-platform persistent and distributed web crawler :crab:

64
Chemrtron
Chemrtron cho45 TypeScript

Chemr is a document viewer; fuzzy match incremental search.

64
mcp
mcp supadata-ai TypeScript

Official Supadata MCP Server - Adds powerful video & web scraping to Cursor, Claude and any other LLM clients.

63
Tapestry
Tapestry NatsuFox Python

Tapestry - 基于 Agent Skill Bundle 的轻量级书签知识库

63
datacrawl
datacrawl DataCrawl-AI Python

A simple and easy to use web crawler for Python

63
Auto_Shadowsocks
Auto_Shadowsocks VonSdite Python

Shadowsocks. 科学上网, 仅供学习。是免费的服务器,可能存在科学上网不稳定。

63
bthello
bthello rehe0x Python

Python3 DHT 磁力种子爬虫 种子解析 种子搜索 演示地址

63
cloud-ip-ranges
cloud-ip-ranges disposable Python

An up-to-date export of cloud provider IP address ranges

62
Reverse-Engineering-Online-Toolkit
Reverse-Engineering-Online-Toolkit Evil0ctal JavaScript

Reverse Engineering Online Toolkit (REOT) 是一个基于纯前端实现的在线逆向工程与爬虫工具箱。本项目旨在为安全研究人员、逆向工程师、爬虫工程师和开发者提供...

62
thecrowler
thecrowler pzaino Go

an open, self-hosted framework for discovering, collecting, understanding, correlating, transforming, and acting on information from heterogeneous env...

62
webrtc-local-ip-leak
webrtc-local-ip-leak niespodd HTML

Oh no, stop this. You can see my local IP address 😲! Use `foundation` attribute against CRC32 lookup table to reveal local IP address of a Chrome/Chr...

62
a11y-sitechecker
a11y-sitechecker forsti0506 TypeScript

Automatic accessibility checker with website crawling + screenshots for easy use

62
crawler-project
crawler-project Albert-W Go

Google资深工程师深度讲解Go语言 爬虫项目。

62
m3u8Downloader
m3u8Downloader mrzhangfelix Python

meijuba.net,Python crawler,M3U8格式视频下载,桌面应用

62
Java-Carwler-Technology
Java-Carwler-Technology soberqian Java

网络数据采集技术—Java网络爬虫 (书稿完整代码,涉及网络爬虫的各种技术和知识点)

62
sciBASIC
sciBASIC xieguigang Visual Basic .NET

sciBASIC# is a kind of dialect language which is derive from the native VB.NET language, and written for the data scientist.

62
ZhihuVAPI
ZhihuVAPI cheezone Python

优雅地玩知乎

62
nepenthes-py
nepenthes-py NEPENTHESWEB Python

The first anti-AI tarpit. Rewritten in Python. Traps LLM crawlers in an infinite maze of fake pages and Markov babble.

61
google_news_scraper_and_sentiment_analyzer
google_news_scraper_and_sentiment_analyzer pratikpv Python

Downloads news articles from Google news and uses pre-trained NLP models to perform sentiment analysis

61
custom-crawler
custom-crawler rollrat C#

🌌 High productivity semi-automatic crawler generator 🛠️🧰

61