Most popular crawler repositories and open source projects

web-crawljs

web crawler for Nodejs

4   21   21  

QQSpider

爬取QQ用户信息（qq号、昵称、生日、地址等基本信息）并做简要analysis。

4   21   21  

ZhengFang_System_Spider

:bug:一只登录正方教务管理系统，爬取数据的小爬虫

2   21   21  

bthello-app

Python3 DHT 磁力种子爬虫 种子解析 种子搜索 演示地址

18   21   21  

covid-19-crawler

코로나 확진자 수/정보 크롤링

10   21   21  

xianzhi_articles

先知文章爬虫项目-[包含2021年7月之前所有文章]

5   21   21  

actor-youtube-scraper

Apify actor to scrape Youtube search results. You can set the maximum...

16   21   21  

vermouth

A torrent site written in the python language & douban scraper

11   20   20  

crawl

Lightweight library for scalable crawlers in Go.

4   20   20  

crawler

A simple and flexible web crawler framework for java.

5   20   20  

scrapy-azuresearch-crawler-samples

Scrapy as a Web Crawler for Azure Search Samples

7   20   20  

movieRater.React

A useful website for finding movie's rating in Chinese and English. By...

2   20   20  

domfind

A Python DNS crawler to find identical domain names under different TL...

3   20   20  

hero

百万英雄答题助手 - 兼容全部答题 APP

7   20   20  

flutter_spider_fx

Flutter爬虫框架，帮助开发者快速在移动设备上构建爬虫，单线程版本

1   20   20  

peeling-onions

A repository to store Deep Web (onion domain) crawler, scraper, and NL...

7   20   20  

collector-filesystem

Norconex Filesystem Collector is a flexible crawler for collecting, pa...

11   20   20  

ppspider_example

ppspider爬虫例子，B站视频信息及评论爬取，qq音乐信息及评论爬取，推特主...

13   20   20  

googleart_scraper

Scrape images from googleart

3   20   20  

sse-option-crawler

SSE 50 index options crawler 上证50期权数据爬虫

8   20   20  

goApp

golang 的一些开源项目，垃圾清理小工具、华为官网抢购程序、房产爬虫、报...

3   20   20  

taiwanlottery

🇹🇼台灣樂透爬蟲🐛(台灣各類型樂透爬蟲)😄

4   20   20  

proxycrawl-node

ProxyCrawl Node library for scraping and crawling

5   20   20  

torrent-crawler

crawls and stores list of torrent links

7   20   20  

WebArchiver

Decentralized web archiving

3   20   20  

Fast-KTSpeechCrawler

Parallelized automatic corpus collection for ASR. Forked from https://...

2   20   20  

crawl_xuexi

学习强国APP上机器学习课程，学习慕课视频批量下载

18   20   20  

Damn-Small-URL-Crawler

A Minimal Yet Powerful Crawler for Extracting all The Internal/Externa...

11   20   20  

anime-tracker

:spider_web: All in one place to track your favorite animes

1   20   20  

k-webtoon-crawler

Korean webtoon crawler with Python 3. 한국 웹툰 크롤러.

2   20   20  

colymer-acquirers

各种爬虫（目前支持Instagram、Weibo、Twitter）Miscellaneous crawlers (c...

6   20   20  

book-spider

🎉 开箱即用的高性能可自定义的笔趣阁小说爬虫 快速下载无广告小说

4   20   20  

Instagram_Crawler

인스타그램 크롤러 (Python, Selenium)

19   20   20  

Web-Crawler

A multithreaded web crawler using two mechanism - single lock and thre...

11   20   20  

zhihu_crawler

本程序支持关键词搜索、热榜、用户信息、回答、专栏文章、评论等信息的抓取

9   20   20  

gdpr-scanner

A tool to check a list of domains for violations against the GDPR :mag...

2   19   19  

proxycrawl-php

ProxyCrawl PHP library for scraping and crawling websites

5   19   19  

dijnet-bot

Az összes számlád még egy helyen :)

0   19   19  

mvcrawler

动漫聚合小站

3   19   19  

AliCouponHunter

Aliexpress coupon search | Find cheapest item and show possible coupon...

7   19   19  

spiderman

your friendly neighborhood web crawler

4   19   19  

screamingFrogR

R integration with Screaming Frog CLI

3   19   19  

ptt-crawler

ptt-crawler is a web crawler module designed to scarpe data from Ptt.

8   19   19  

playwright-webcrawler

Parallel crawler powered by Playwright-Python

7   19   19  

plusfish

Plusfish is a classic web application vulnerability scanner/fuzzer and...

9   19   19  

Broken-Links-Crawler-Action

GitHub Action to check a website for broken links

2   19   19  

scrapy-diario-oficial-da-uniao

Script Python para buscar o conteúdo do Diário Oficial da União

5   19   19  

bilibili_comment_crawl

爬取bilibili视频下的评论，最新出品！！！⚠本代码只适用于学习，做其他事...

0   19   19  

crawler

Crawl your own website with various clients for SEO and indexing purpo...

4   19   19  

web-crawler

Python Web Crawler with Selenium and PhantomJS

14   19   19