Topic

crawler

Repositories (1501)

kabegame
kabegame kabegame TypeScript

Kabegame — An anime image crawler client with pluggable crawlers (from a GitHub plugin repo), wallpaper rotation, local folder sync album. Supports Wi...

79
xueqiu_spider_LQH_LZQ
xueqiu_spider_LQH_LZQ 1491270550 Python

雪球爬虫 高效爬取近期沪深A股股票评论并自动生成PDF版情感分析报告

79
Pasta
Pasta Kr0ff Python

A PasteBin scrapper that doesnt rely on the PasteBin scrape API

79
slideshare-downloader
slideshare-downloader yodiaditya Python

Slideshare to PDF downloader. Using Selenium and auto scroll-down to get the entire slides completely.

79
fund-crawler
fund-crawler nullpointer TypeScript

基于NodeJS的基金数据爬虫,爬取的数据存于github的@nullpointer/fund-data。

79
car-prices
car-prices go-crawler Go

Golang爬虫 爬取汽车之家 二手车产品库

79
scrapy-examples
scrapy-examples feiskyer Python

Some scrapy and web.py exmaples

79
puppeteer-walker
puppeteer-walker lrlna JavaScript

a puppeteer walker 🕷 🕸

79
arachnid
arachnid watzon Crystal

Powerful web scraping framework for Crystal

79
MusicTagger
MusicTagger Mai-icy Python

一个可以补全mp3,flac文件元数据的图形化界面。还可以下载歌词

78
librengine
librengine liameno C++

Privacy Web Search Engine (not meta, own crawler)

78
fetchman
fetchman da2vin Python

fetchman is a simple crawler system/简单好用的爬虫框架

78
ctrip_spider
ctrip_spider evanleungc Python

Scrape Learning (ctrip)

78
tumblr_crawler
tumblr_crawler 2024baibai Python

tumblr解析网站

78
ceiba-dl
ceiba-dl lantw44 Python

NTU CEIBA 資料下載工具

78
venera-jm
venera-jm BB-CHICKEN JavaScript

基于 Venera App 的禁漫天堂 (JMComic) 漫画源插件,支持浏览、搜索、签到、节点延迟测试、图片分流测速等功能。

77
steam-discount
steam-discount EXP-Tools Python

steam 特惠游戏榜单(自动刷新)

77
Google-Patents-Scraper
Google-Patents-Scraper wenyalintw Python

Automatically download all PDF files of searching results & their patent families found on Google Patents.

77
tg_crawler
tg_crawler vhdmsm Python

Just a crawler based on tg-cli for Telegram. Deprecated by now, please use telegram-export.

77
qr-pirate
qr-pirate mzollin Python

crawl QR-codes from search engines and look for bitcoin private keys

77
midnight_sea
midnight_sea RicYaben Python

Midnight Sea: navigating in the waters of dark web markets

76
Website-Crawler
Website-Crawler pc8544 Java

Extract data from websites in LLM ready JSON or CSV format. Crawl or Scrape entire website with Website Crawler

76
TikScraperPHP
TikScraperPHP pablouser1 JavaScript

Wrapper for TikTok API

76
cache-warmup
cache-warmup eliashaeussler PHP

🔥 PHP library to warm up caches of URLs located in XML sitemaps

76
tw-stock-telegram-bot
tw-stock-telegram-bot x3388638 JavaScript

台股機器人,提供即時個股及大盤報價、走勢、新聞、盤後資料等 Telegram bot to query real-time TW stock quotes, charts, news, and other related informatio...

76
eastmoney
eastmoney minicloudsky JavaScript

python requests + Django+ nodejs koa+ mysql to crawl eastmoney fund and stock data,for data analysis and visualiaztion .

76
venom
venom PreferredAI Java

Your preferred open source focused crawler for the deep web.

76
JobApplicationBot
JobApplicationBot drkostas HTML

A bot that automatically sends emails to new ads posted in any desired xe.gr search url.

75
python-testing-crawler
python-testing-crawler python-testing-crawler Python

A crawler for automated functional testing of a web application

75
Visual_MediaCrawler
Visual_MediaCrawler persist-1 Python

可视化爬虫(支持:哔哩哔哩 | 抖音 | 小红书 | 贴吧 | 微博 | 知乎 | 快手),异步、高效、直观地采集国内主流平台的媒体数据的前后端一体项目(Based on "Medi...

74
Instagram-downloader
Instagram-downloader fernandod1 Python

Instagram user's photos and videos downloader. Download all media files from any username. Working 2022!

74
newspaperjs
newspaperjs flickz HTML

News extraction and scraping. Article Parsing

74
light-crawler
light-crawler zhang2333 JavaScript

a simplified directed customizable website crawler

74
tumblr-crawler-cli
tumblr-crawler-cli tzw0745 Python

Tumblr Download Tool with High Speed and Customization. 高性能&高定制化的Tumblr下载工具。

74
crawler_examples
crawler_examples liuslnlp Python

Some classic web crawler projects.一些经典的爬虫

74
social-scraper
social-scraper behitek Python

Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)

74
meta-spy
meta-spy DEENUU1 Python

👾 CLI MetaSpy (Facebook, Instagram) scraper and crawler - instagram account, facebook accounts, pages and search

73
PSGameSpider
PSGameSpider RavelloH JavaScript

自动爬取所有PlayStationStore中的所有游戏信息,包括封面、描述、价格、评分等,生成网页并索引 # # # Automatically crawl all game infos in all playstation...

73
crawlerdetect
crawlerdetect x-way Go

Golang module to detect bots and crawlers via the user agent

73
ComicSpider
ComicSpider QuantumLiu Python

动漫之家漫画站电脑版原图爬虫

73
carbonbot
carbonbot crypto-crawler Rust

A command line tool based on the crypto-crawler library.

72
Wedge
Wedge LZ0211 JavaScript

可配置的小说下载及电子书生成工具

72
rubium
rubium vifreefly Ruby

Antidetect Headless Chrome Browser for Ruby Web Scraping and Automation

72
facebook-page-info-scraper
facebook-page-info-scraper wael-sudo2 Python

Free Facebook pages MetaData Scraping Library - Unlimited Calls

71
OpenCrawler
OpenCrawler merwin-asm Python

Open Crawler || Open Source Crawler

71
aio-scrapy
aio-scrapy ConlinH Python

Implement scrapy with asyncio

71
Pinterest-Crawler
Pinterest-Crawler SajjadAemmi Python

Download HD images from pinterest by your favorite keywords

71
paperCrawler
paperCrawler sucv Python

This is a Scrapy-based web-spider. It scrapes papers from TOP conferences and journals.

70
lrabbit_scrapy
lrabbit_scrapy litter-rabbit Python

a quick start python mutil thread crawl

70
site-mirror-py
site-mirror-py generals-space Python

[码云](https://gitee.com/generals-space/site-mirror-py) 通用爬虫, 仿站工具, 整站下载

70