Most popular crawler repositories and open source projects

DarkSpider

Anatomy and Visualization of the Network structure of the Dark web usi...

3   23   23  

crawler

gRPC web crawler turbo charged for performance

2   23   23  

Amipy

A micro asynchronous Python website crawler framework .（Python微型异...

11   23   23  

crawlerr

A simple and fully customizable web crawler/spider for Node.js with se...

7   23   23  

onionstack

A Pictorial Book of Tor Hidden Services.

3   23   23  

WebCrawler

one web crawler frame based on golang

15   23   23  

little-python

little python projects, 一些小的python项目.

12   23   23  

findmeaflat

Get notified of new listings on popular German real estate portals.

5   23   23  

googleplay_api

Google Play Unofficial Python 3 API Library

8   23   23  

doc_crawler.py

Explore a website recursively and download all the wanted documents (P...

7   22   22  

okcoin-socket-crawler

A okcoin crawler based on websocket, save data to mysql

9   22   22  

Gumo

A crawler that extracts data from a dynamic webpage. Written in node j...

0   22   22  

crawling-framework

Easily crawl news portals or blog sites using Storm Crawler.

3   22   22  

spider-video

Node 批量爬取头条视频

7   22   22  

RedBetter-WM2

Better.php crawler for Redacted that uses WhatManager

0   22   22  

exoskeleton

A Python framework to build polite, but tenacious crawlers / scrapers...

1   22   22  

linkedin-public-dir-companies

Crawler and scraper of the public directory of companies on LinkedIn.

5   22   22  

minicrawler

Multiplexing web client supporting HTTP/2 and WHATWG URL compliant par...

5   22   22  

html-article-extractor

A web page content extractor

1   22   22  

tistore

:camera: Tistory photo grabber

10   22   22  

Mini-Projects

A collection of short projects, you could try and implement these as s...

17   22   22  

gateway_to_DeepReinforcementLearning_DeepNN

:trophy: Welcome to the wonderland of "AI" = f(DL, RL, DRL, ML, NLP, K...

5   22   22  

fifa-stats-crawler

A web-crawler to scrape FIFA 20 and 21 players' latest information fro...

11   22   22  

site-mirror-go

来自[码云](https://gitee.com/generals-space/site-mirror-go) 通用爬虫,...

2   22   22  

indieweb-search

Source code for the IndieWeb search engine.

2   22   22  

httpsuite

A toolkit for web reconnaissance, it's fast and easy to use.

8   22   22  

armiarma

Armiarma is a Libp2p open-network crawler with a current focus on Ethe...

9   22   22  

Taiwan-Stocks

台灣上市櫃公司爬蟲，分析盤後股票趨勢以及繪製K線圖、均線圖、三大法人成...

7   22   22  

JobApplicationBot

A bot that automatically sends emails to new ads posted in any desired...

2   22   22  

publiccode-crawler

publiccode.yml crawler for the Open Source software catalog of Develop...

49   21   21  

libp2p-dht-scrape-aas

🧹 A libp2p DHT scraper as a service allowing anyone to collect, consu...

8   21   21  

scrapy_poetry

本项目使用scrapy对 古诗文网 进行爬虫，获取不同分类（爱情、七夕等）的宋...

0   21   21  

emarketcrawlR

This R package provides a crawler to scrape the European Energy Market...

10   21   21  

estate-crawler

Scraping the real estate agencies for up-to-date house listings as soo...

5   21   21  

lopez

Crawling and scraping the Web for fun and profit

3   21   21  

Codeforces-AutoCommit

When you solve the problem of the Codeforces site, it automatically co...

5   21   21  

Pinterest-infinite-crawler

An infinite Pinterest crawler/scraper. Crawl image with inifnite-scrol...

3   21   21  

ExHentaiReader

Best manga-viewer on windows for crawling/downloading/browsing exhenta...

1   21   21  

Crawler

整理本人在2021年10月-12月期间写的一些爬虫demo，比如用于渗透测试中SQL注...

5   21   21  

telegram-member-inviter

Crawling client's groups and channels to invite their members to a tar...

15   21   21  

retro-env-can-weather-chan

Retro Environment Canada Weather Channel for your browser

3   21   21  

Onlyfans-dl

This tool downloads all photos/videos from an OnlyFans profile, creati...

8   21   21  

scrapy-tor-proxy-rotation

An IP rotator via Tor for Scrapy.

1   21   21  

sinaCrawlerV

backup posts and comments of specify user in sina

1   21   21  

hupu_spider

虎扑步行街爬虫

8   21   21  

ParseMyCF-contest

A personal submission codeforces parser for CF, parsed by individual c...

8   21   21  

cea

高校高校统一身份认证 Node.js 优雅可扩展示例，已集成今日校园签到(支持多...

10   21   21  

akka-react-cloudant

A Soccer Dashboard created by scraping EPL website using Akka backend...

18   21   21  

rankr

🇰🇷 Realtime integrated information analysis service

2   21   21  

SlackWebhooksGithubCrawler

Search for Slack Webhooks token publicly exposed on Github

1   21   21