Most popular crawler repositories and open source projects

grab_beautiful_girls_pictures

抓取MM131美女写真图片，并将其保存至本地指定的文件夹中。

17 36 36

crawler

十年磨一剑：Crawler4U, a general purpose focused crawler

6 36 36

node-html-crawler

Simple for use node html crawler (spider) of site web pages

12 36 36

Deepminer

Deep web crawler and search engine

7 36 36

robotstester

This Python script can enumerate all URLs present in robots.txt files,...

4 36 36

medup

Download all content from Medium and Dev.to to local folder

10 36 36

aio-scrapy

Implement scrapy with asyncio

8 36 36

usetube

search & get datas from youtube no google account needed

15 36 36

DDoM

A simple, open-source, easy to use, and free download manager for malw...

3 35 35

Bing-Wallpaper-Action

API with Redis / Vercel , DataBase with Json, Crawel with Github Actio...

6 35 35

ZUCC_ZhenFangHelper

正方教务管理系统学生版的自动登录、选课、信息获取

7 35 35

gargantua

The fast website crawler

3 35 35

soducrawler

12 35 35

schannel-qt5

A GUI client of schannel powered by therecipe/qt and golang

5 35 35

lostark-wait-notifier

🐤️ Lost Ark wait notifier

7 35 35

crawler

Crawler with Python 3.

19 35 35

ArticleSpider

Crawling zhihu, jobbole, lagou by Scrapy, and using Elasticsearch+Djan...

10 35 35

shadow_spider

14 35 35

crawlerdetect

🕷CrawlerDetect is a Python class for detecting bots/crawlers/spiders v...

6 35 35

NetEaseCloudMusicCrawler

HttpClient + Jsoup + Queue

14 34 34

imooc-crawler

[Obsolete] imooc web crawler in Node.js（使用 Node.js 编写的慕课网爬虫...

15 34 34

serverless-instagram-crawler

serverless, instagram hashtag crawler with lambda, dynamoDB

8 34 34

phpwebcrawler

A Web Crawler Created in PHP

32 34 34

ebedke

crawl pages to check what is for lunch today

5 34 34

BingGallery

A simple crawler to get all Bing gallery pictures.

14 34 34

toxcrawler

A Tox DHT network crawler

12 34 34

BilibiliCrawler

:cyclone: crawl bilibili user info and video info for data analysis |...

6 34 34

Youtube_Scraper

Scrape data about an entire Channel or just a Playlist, or get stats a...

6 34 34

go-crawler-distributed

分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微...

5 34 34

Crawling-Emails

Very simple bash script to crawl email addresses from a specific websi...

16 34 34

a11y-sitechecker

Automatic accessibility checker with website crawling + screenshots fo...

4 34 34

courlan

Clean, filter and sample URLs to optimize data collection – includes s...

4 33 33

toutiaocrawler

头条号爬虫案例

14 33 33

WebCrawler

A web crawler based on requests-html, mainly targets for url validatio...

12 33 33

ioweb

Web Scraping Framework

11 33 33

Youtube_Comment_Crawler

유튜브 댓글 크롤러 ( Python, BeautifulSoup, Selenium )

9 33 33

visual-spider

用JavaFX开发基于crawler4j的图形化的网络爬虫

9 32 32

proxi

Proxy pool. Finds and checks proxies with rest api for querying result...

4 32 32

INMET-API-temperature

Crawler dos dados metereológicos de estações convencionais do INMET (B...

7 32 32

2020-nCov-anhui

2020新型冠状病毒疫情数据爬取、可视化、网站开发部署

12 32 32

LOLPrediction

英雄联盟胜负预测

11 32 32

Facebooker

an unofficial facebook api

8 32 32

serritor

Serritor is an open source web crawler framework built upon Selenium a...

15 32 32

google_news_scraper_and_sentiment_analyzer

Downloads news articles from Google news and uses pre-trained NLP mode...

11 32 32

pyfutebol

Simples crawler para obter resultados dos jogos de futebol

15 32 32

scalpel

A fast and powerful web scraping library

2 32 32

learncpp-download

Scrape bot, to get you an offline copy of tutorials

15 31 31

local-api-client-csharp

This .NET Standard package provides convenient access to the Local API...

2 31 31

PTTmineR

Parallel Searching and Crawling Data from PTT 🚀

4 31 31

Search_Ads_Web_Service

Online search advertisement platform & Realtime Campaign Monitoring [M...

1 31 31