Most popular crawler repositories and open source projects

Pinterest-infinite-crawler

An infinite Pinterest crawler/scraper. Crawl image with inifnite-scrol...

3 21 21

ExHentaiReader

Best manga-viewer on windows for crawling/downloading/browsing exhenta...

1 21 21

Crawler

整理本人在2021年10月-12月期间写的一些爬虫demo，比如用于渗透测试中SQL注...

5 21 21

telegram-member-inviter

Crawling client's groups and channels to invite their members to a tar...

15 21 21

retro-env-can-weather-chan

Retro Environment Canada Weather Channel for your browser

3 21 21

Onlyfans-dl

This tool downloads all photos/videos from an OnlyFans profile, creati...

8 21 21

scrapy-tor-proxy-rotation

An IP rotator via Tor for Scrapy.

1 21 21

vermouth

A torrent site written in the python language & douban scraper

11 20 20

crawl

Lightweight library for scalable crawlers in Go.

4 20 20

crawler

A simple and flexible web crawler framework for java.

5 20 20

scrapy-azuresearch-crawler-samples

Scrapy as a Web Crawler for Azure Search Samples

7 20 20

movieRater.React

A useful website for finding movie's rating in Chinese and English. By...

2 20 20

domfind

A Python DNS crawler to find identical domain names under different TL...

3 20 20

hero

百万英雄答题助手 - 兼容全部答题 APP

7 20 20

flutter_spider_fx

Flutter爬虫框架，帮助开发者快速在移动设备上构建爬虫，单线程版本

1 20 20

peeling-onions

A repository to store Deep Web (onion domain) crawler, scraper, and NL...

7 20 20

collector-filesystem

Norconex Filesystem Collector is a flexible crawler for collecting, pa...

11 20 20

ppspider_example

ppspider爬虫例子，B站视频信息及评论爬取，qq音乐信息及评论爬取，推特主...

13 20 20

googleart_scraper

Scrape images from googleart

3 20 20

sse-option-crawler

SSE 50 index options crawler 上证50期权数据爬虫

8 20 20

goApp

golang 的一些开源项目，垃圾清理小工具、华为官网抢购程序、房产爬虫、报...

3 20 20

taiwanlottery

🇹🇼台灣樂透爬蟲🐛(台灣各類型樂透爬蟲)😄

4 20 20

proxycrawl-node

ProxyCrawl Node library for scraping and crawling

5 20 20

torrent-crawler

crawls and stores list of torrent links

7 20 20

WebArchiver

Decentralized web archiving

3 20 20

Fast-KTSpeechCrawler

Parallelized automatic corpus collection for ASR. Forked from https://...

2 20 20

crawl_xuexi

学习强国APP上机器学习课程，学习慕课视频批量下载

18 20 20

Damn-Small-URL-Crawler

A Minimal Yet Powerful Crawler for Extracting all The Internal/Externa...

11 20 20

anime-tracker

:spider_web: All in one place to track your favorite animes

1 20 20

k-webtoon-crawler

Korean webtoon crawler with Python 3. 한국 웹툰 크롤러.

2 20 20

colymer-acquirers

各种爬虫（目前支持Instagram、Weibo、Twitter）Miscellaneous crawlers (c...

6 20 20

book-spider

🎉 开箱即用的高性能可自定义的笔趣阁小说爬虫快速下载无广告小说

4 20 20

Instagram_Crawler

인스타그램 크롤러 (Python, Selenium)

19 20 20

Web-Crawler

A multithreaded web crawler using two mechanism - single lock and thre...

11 20 20

zhihu_crawler

本程序支持关键词搜索、热榜、用户信息、回答、专栏文章、评论等信息的抓取

9 20 20

crawler

Crawl your own website with various clients for SEO and indexing purpo...

4 19 19

web-crawler

Python Web Crawler with Selenium and PhantomJS

14 19 19

2017_PyConTW_Talk

4 19 19

botanalyse

botsonar analyse open api

3 19 19

magento2-module-primer

Full Page Cache Priming tool for Magento 2

7 19 19

crawler

Nodejs crawler for cnbeta.com

10 19 19

baiduyun_spider

Python + MongoDB 开发的百度云资源爬虫

15 19 19

crawl

A concurrent crawler that minimizes memory use. Output suitable for us...

7 19 19

google-scholar-crawler

A crawler to crawl google scholar search page

13 19 19

sentinel-cendertron

Cendertron = Crawler + cendertron, Crawl AJAX-heavy client-side Single...

5 19 19

NEEA-TOEFL-Testseat-Crawler

托福考位爬虫 NEEA TOEFL Testseat Crawler

0 19 19

gdpr-scanner

A tool to check a list of domains for violations against the GDPR :mag...

2 19 19

proxycrawl-php

ProxyCrawl PHP library for scraping and crawling websites

5 19 19

dijnet-bot

Az összes számlád még egy helyen :)

0 19 19

mvcrawler

动漫聚合小站

3 19 19