Topic

crawler

Repositories (1501)

crawler_JD_what_worthy_buying
crawler_JD_what_worthy_buying HarborZeng Python

爬取京东商品所有评论,利用情感分析,判断商品是否值得买

61
TumblTwo
TumblTwo johanneszab C#

TumblTwo, an Improved Fork of TumblOne, a Tumblr Downloader.

61
koshort
koshort koshort Python

(deprecated) :cat: koshort is a Python package for Korean internet spoken language crawling and processing... or maybe Korean domestic cat.

61
scrapy-distributed
scrapy-distributed Insutanto Python

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy...

61
WebCrawler
WebCrawler Misterhex C#

Just a simple web crawler which return crawled links as IObservable using reactive extension and async await.

61
DDoM
DDoM Endermanch Python

A simple, open-source, easy to use, and free download manager for malware samples.

60
snapcrawl
snapcrawl DannyBen Ruby

Crawl a website and take screenshots

60
rolling-news
rolling-news Jacen789 Python

获取滚动新闻

60
findopendata
findopendata findopendata Python

A search engine for Open Data

60
instagram-hashtag-crawler
instagram-hashtag-crawler simonseo Python

Crawl Instagram hashtags

60
WebSpider
WebSpider xdoer JavaScript

基于Nodejs,superagent,cheerio的在线web爬虫项目,支持生成API

60
spider-nodejs
spider-nodejs spider-rs Rust

Spider ported to Node.js

59
python
python joaopauloaramuni HTML

Repo Python

59
billboard-json
billboard-json KoreanThinker TypeScript

🎧 Get json type billboard hot 100 chart

59
Web-Iota
Web-Iota SatinWukerORIG Python

Iota is a web scraper that can find all of the images and links/suburls on a webpage

59
aio-vextractor
aio-vextractor panoslin Python

解析视频 网站/APP/H5 页面视频信息。支持抖音、腾讯视频、YouTube、Instagram 等40余个网站与APP

59
crawler-userscript
crawler-userscript zjh1943 JavaScript

一个基于 Tampermonkey 插件平台开发的爬虫。主要目的是最大限度模拟用户环境,避免被反爬虫系统识破。

59
gscholar-citations-crawler
gscholar-citations-crawler thu-pacman Python

Crawl all your citations from Google Scholar

59
lyrics-crawler
lyrics-crawler willamesoares Python

Get the lyrics for the song currently playing on Spotify

59
unfx-proxy-parser
unfx-proxy-parser openproxyspace JavaScript

Unfx Proxy Parser - Nextgen proxy parser with deep links crawler. Follow to internal links, third-party links. Sorting results by countries.

59
pomp
pomp estin Python

Screen scraping and web crawling framework

59
site-md
site-md yazinsai TypeScript

Serve clean Markdown from your Next.js site to AI agents, crawlers, and LLMs. Humans get HTML, agents get clean Markdown of the same pages. Two-file i...

58
Deepminer
Deepminer Conso1eCowb0y Python

Deep web crawler and search engine

58
simple_bank_korea
simple_bank_korea Beomi Python

simple crawler for Korean banks with Transactions

58
crawler
crawler tomasnorre PHP

Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc.

58
tool-gin
tool-gin bajins Go

基于go-gin框架建立减少冗余动作项目,如:下载一些工具

58
phpcrawl
phpcrawl mmerian PHP

Copy of http://phpcrawl.cuab.de/ for using with composer

58
Daily-code
Daily-code rui7157 Python

日常代码爬虫、gui小工具等

58
uforall
uforall rix4uni Go

uforall is a fast url crawler this tool crawl all URLs number of different sources, alienvault,WayBackMachine,urlscan,commoncrawl

57
wechat_biz
wechat_biz yusp998 Go

微信公众号爬虫,以API方式提供公众号文章获取,包括阅读量、点赞等

57
wishlist
wishlist Jaymon Python

Read an Amazon wishlist programmatically with Python

57
proxycrawl-python
proxycrawl-python crawlbase Python

ProxyCrawl Python library for scraping and crawling

57
All-IT-eBooks-Spider
All-IT-eBooks-Spider Kulbear Python

[Updated] A simple python crawler for my tutorial blog at http://www.jianshu.com/p/8fb5bc33c78e

57
SearchEngineScrapy
SearchEngineScrapy naqushab Python

Scrape data from Google.com, Bing.com, Baidu.com, Ask.com, Yahoo.com, Yandex.com

57
talospider
talospider howie6879 Python

talospider - A simple,lightweight scraping micro-framework

57
NewsCrap
NewsCrap odaysec Python

NewsCrap adalah alat scraping berita Google berbasis Command Line Interface (CLI) yang dirancang untuk riset, investigasi, dan pengumpulan data OSINT....

56
price-monitoring
price-monitoring roccomuso JavaScript

Node.js price monitoring library, leveraging the power of x-ray and nightmare.

56
actor-facebook-scraper
actor-facebook-scraper pocesar TypeScript

Scrape public Facebook pages, posts, reviews and comments

56
devsearch
devsearch nicholaskajoh Python

A web search engine built with Python which uses TF-IDF and PageRank to sort search results.

56
PicCrawler
PicCrawler fengzhizi715 Java

使用RxJava2 和 Java 8的特性开发的图片爬虫

56
bio999
bio999 brokeexee Rust

BIO999 is a free-running synthetic organism. It reads continuously and commits what it reads to a corpus that grows by roughly twenty gigabytes an hou...

55
subscan
subscan eredotpkfr Rust

⚡ A subdomain enumeration tool leveraging diverse techniques, designed for advanced pentesting operations

55
rag-crawler
rag-crawler sigoden TypeScript

Crawl a website to generate knowledge file for RAG

55
crawler
crawler a11ywatch Rust

gRPC web crawler turbo charged for performance

55
MyCrawler
MyCrawler netcan Python

我的爬虫合集

55
hentai-daily
hentai-daily bgzolab Python

[NSFW] A hentai daily paper combined with multi sources

54
rarbgcli
rarbgcli FarisHijazi Python

RARBG command line interface for scraping the rarbg.to torrent search engine

54
crawler_shopee
crawler_shopee charlie0227 Python

Shopee coin getter is a script to collect daily shopee coins.

54
browser-as-a-service
browser-as-a-service hfreire JavaScript

A web browser :earth_americas: hosted as a service, to render your JavaScript web pages as HTML

54
kalel
kalel noobscode Python

Kal El Network Stress Test and Penetration Testing Toolkit

54