Most popular crawler repositories and open source projects

findmeaflat

Get notified of new listings on popular German real estate portals.

5   23   23  

googleplay_api

Google Play Unofficial Python 3 API Library

8   23   23  

Helios

A Python based Web Application security scanner

11   23   23  

Python

🍋 Python基础、Pygame游戏编程、Python算法与面试题、四种常用的Python Web...

4   23   23  

python-hacking-tools

Python tools for ethical hacking

11   23   23  

github-action-rss-crawler

Auto crawl RSS feeds using Github Action

9   23   23  

ghrr

A utility to collect data from github stargazers, subscribers and cont...

1   23   23  

DarkSpider

Anatomy and Visualization of the Network structure of the Dark web usi...

3   23   23  

crawler

gRPC web crawler turbo charged for performance

2   23   23  

doc_crawler.py

Explore a website recursively and download all the wanted documents (P...

7   22   22  

okcoin-socket-crawler

A okcoin crawler based on websocket, save data to mysql

9   22   22  

Gumo

A crawler that extracts data from a dynamic webpage. Written in node j...

0   22   22  

crawling-framework

Easily crawl news portals or blog sites using Storm Crawler.

3   22   22  

spider-video

Node 批量爬取头条视频

7   22   22  

RedBetter-WM2

Better.php crawler for Redacted that uses WhatManager

0   22   22  

exoskeleton

A Python framework to build polite, but tenacious crawlers / scrapers...

1   22   22  

linkedin-public-dir-companies

Crawler and scraper of the public directory of companies on LinkedIn.

5   22   22  

minicrawler

Multiplexing web client supporting HTTP/2 and WHATWG URL compliant par...

5   22   22  

html-article-extractor

A web page content extractor

1   22   22  

tistore

:camera: Tistory photo grabber

10   22   22  

Mini-Projects

A collection of short projects, you could try and implement these as s...

17   22   22  

gateway_to_DeepReinforcementLearning_DeepNN

:trophy: Welcome to the wonderland of "AI" = f(DL, RL, DRL, ML, NLP, K...

5   22   22  

fifa-stats-crawler

A web-crawler to scrape FIFA 20 and 21 players' latest information fro...

11   22   22  

site-mirror-go

来自[码云](https://gitee.com/generals-space/site-mirror-go) 通用爬虫,...

2   22   22  

indieweb-search

Source code for the IndieWeb search engine.

2   22   22  

httpsuite

A toolkit for web reconnaissance, it's fast and easy to use.

8   22   22  

armiarma

Armiarma is a Libp2p open-network crawler with a current focus on Ethe...

9   22   22  

Taiwan-Stocks

台灣上市櫃公司爬蟲，分析盤後股票趨勢以及繪製K線圖、均線圖、三大法人成...

7   22   22  

JobApplicationBot

A bot that automatically sends emails to new ads posted in any desired...

2   22   22  

sinaCrawlerV

backup posts and comments of specify user in sina

1   21   21  

hupu_spider

虎扑步行街爬虫

8   21   21  

ParseMyCF-contest

A personal submission codeforces parser for CF, parsed by individual c...

8   21   21  

cea

高校高校统一身份认证 Node.js 优雅可扩展示例，已集成今日校园签到(支持多...

10   21   21  

akka-react-cloudant

A Soccer Dashboard created by scraping EPL website using Akka backend...

18   21   21  

rankr

🇰🇷 Realtime integrated information analysis service

2   21   21  

SlackWebhooksGithubCrawler

Search for Slack Webhooks token publicly exposed on Github

1   21   21  

web-crawljs

web crawler for Nodejs

4   21   21  

QQSpider

爬取QQ用户信息（qq号、昵称、生日、地址等基本信息）并做简要analysis。

4   21   21  

ZhengFang_System_Spider

:bug:一只登录正方教务管理系统，爬取数据的小爬虫

2   21   21  

bthello-app

Python3 DHT 磁力种子爬虫 种子解析 种子搜索 演示地址

18   21   21  

covid-19-crawler

코로나 확진자 수/정보 크롤링

10   21   21  

xianzhi_articles

先知文章爬虫项目-[包含2021年7月之前所有文章]

5   21   21  

actor-youtube-scraper

Apify actor to scrape Youtube search results. You can set the maximum...

16   21   21  

publiccode-crawler

publiccode.yml crawler for the Open Source software catalog of Develop...

49   21   21  

libp2p-dht-scrape-aas

🧹 A libp2p DHT scraper as a service allowing anyone to collect, consum...

8   21   21  

scrapy_poetry

本项目使用scrapy对 古诗文网 进行爬虫，获取不同分类（爱情、七夕等）的宋...

0   21   21  

emarketcrawlR

This R package provides a crawler to scrape the European Energy Market...

10   21   21  

estate-crawler

Scraping the real estate agencies for up-to-date house listings as soo...

5   21   21  

lopez

Crawling and scraping the Web for fun and profit

3   21   21  

Codeforces-AutoCommit

When you solve the problem of the Codeforces site, it automatically co...

5   21   21