Most popular crawler repositories and open source projects

images-grabber

🖼️ Get all images from pixiv/twitter/deviantart

2   24   24  

Amipy

A micro asynchronous Python website crawler framework .（Python微型异...

11   23   23  

crawlerr

A simple and fully customizable web crawler/spider for Node.js with se...

7   23   23  

Mimo-Crawler

A web crawler that uses Firefox and js injection to interact with webp...

2   23   23  

onionstack

A Pictorial Book of Tor Hidden Services.

3   23   23  

WebCrawler

one web crawler frame based on golang

15   23   23  

little-python

little python projects, 一些小的python项目.

12   23   23  

proxycrawl-node

ProxyCrawl Node library for scraping and crawling

5   23   23  

findmeaflat

Get notified of new listings on popular German real estate portals.

5   23   23  

googleplay_api

Google Play Unofficial Python 3 API Library

8   23   23  

Helios

A Python based Web Application security scanner

11   23   23  

Python

🍋 Python基础、Pygame游戏编程、Python算法与面试题、四种常用的Python We...

4   23   23  

app-crawler

crawling App by uiautomator2 & mitmproxy

9   23   23  

python-hacking-tools

Python tools for ethical hacking

11   23   23  

github-action-rss-crawler

Auto crawl RSS feeds using Github Action

9   23   23  

ghrr

A utility to collect data from github stargazers, subscribers and cont...

1   23   23  

DarkSpider

Anatomy and Visualization of the Network structure of the Dark web usi...

3   23   23  

crawler

gRPC web crawler turbo charged for performance

2   23   23  

Mini-Projects

A collection of short projects, you could try and implement these as s...

17   22   22  

gateway_to_DeepReinforcementLearning_DeepNN

:trophy: Welcome to the wonderland of "AI" = f(DL, RL, DRL, ML, NLP, K...

5   22   22  

fifa-stats-crawler

A web-crawler to scrape FIFA 20 and 21 players' latest information fro...

11   22   22  

master-to-pythonista

A list of awesome beginners-friendly projects.

20   22   22  

udemyscraper

A Udemy Course Scraper built with bs4 and selenium, that fetches udemy...

11   22   22  

site-mirror-go

来自[码云](https://gitee.com/generals-space/site-mirror-go) 通用爬虫,...

2   22   22  

indieweb-search

Source code for the IndieWeb search engine.

2   22   22  

httpsuite

A toolkit for web reconnaissance, it's fast and easy to use.

8   22   22  

crawl-original-google-images

python scripts for crawling original image from Google Images

4   22   22  

armiarma

Armiarma is a Libp2p open-network crawler with a current focus on Ethe...

9   22   22  

Taiwan-Stocks

台灣上市櫃公司爬蟲，分析盤後股票趨勢以及繪製K線圖、均線圖、三大法人成...

7   22   22  

JobApplicationBot

A bot that automatically sends emails to new ads posted in any desired...

2   22   22  

doc_crawler.py

Explore a website recursively and download all the wanted documents (P...

7   22   22  

okcoin-socket-crawler

A okcoin crawler based on websocket, save data to mysql

9   22   22  

Gumo

A crawler that extracts data from a dynamic webpage. Written in node j...

0   22   22  

spider-video

Node 批量爬取头条视频

7   22   22  

RedBetter-WM2

Better.php crawler for Redacted that uses WhatManager

0   22   22  

linkedin-public-dir-companies

Crawler and scraper of the public directory of companies on LinkedIn.

5   22   22  

minicrawler

Multiplexing web client supporting HTTP/2 and WHATWG URL compliant par...

5   22   22  

tistore

:camera: Tistory photo grabber

10   22   22  

udemy-crawler

Crawling Udemy course info and save into JSON format.

7   22   22  

sinaCrawlerV

backup posts and comments of specify user in sina

1   21   21  

hupu_spider

虎扑步行街爬虫

8   21   21  

ParseMyCF-contest

A personal submission codeforces parser for CF, parsed by individual c...

8   21   21  

cea

高校高校统一身份认证 Node.js 优雅可扩展示例，已集成今日校园签到(支持多...

10   21   21  

crawling-framework

Easily crawl news portals or blog sites using Storm Crawler.

4   21   21  

akka-react-cloudant

A Soccer Dashboard created by scraping EPL website using Akka backend...

18   21   21  

rankr

🇰🇷 Realtime integrated information analysis service

2   21   21  

SlackWebhooksGithubCrawler

Search for Slack Webhooks token publicly exposed on Github

1   21   21  

exoskeleton

A Python framework to build polite, but tenacious crawlers / scrapers...

1   21   21  

web-crawljs

web crawler for Nodejs

4   21   21  

QQSpider

爬取QQ用户信息（qq号、昵称、生日、地址等基本信息）并做简要analysis。

4   21   21