Most popular crawler repositories and open source projects

Bing-Wallpaper-Action

API with Redis / Vercel , DataBase with Json, Crawel with Github Actio...

6 35 35

ZUCC_ZhenFangHelper

正方教务管理系统学生版的自动登录、选课、信息获取

7 35 35

gargantua

The fast website crawler

3 35 35

soducrawler

12 35 35

schannel-qt5

A GUI client of schannel powered by therecipe/qt and golang

5 35 35

lostark-wait-notifier

🐤️ Lost Ark wait notifier

7 35 35

crawler

Crawler with Python 3.

19 35 35

ArticleSpider

Crawling zhihu, jobbole, lagou by Scrapy, and using Elasticsearch+Djan...

10 35 35

shadow_spider

14 35 35

crawlerdetect

🕷CrawlerDetect is a Python class for detecting bots/crawlers/spiders v...

6 35 35

NetEaseCloudMusicCrawler

HttpClient + Jsoup + Queue

14 34 34

imooc-crawler

[Obsolete] imooc web crawler in Node.js（使用 Node.js 编写的慕课网爬虫...

15 34 34

serverless-instagram-crawler

serverless, instagram hashtag crawler with lambda, dynamoDB

8 34 34

phpwebcrawler

A Web Crawler Created in PHP

32 34 34

ebedke

crawl pages to check what is for lunch today

5 34 34

BingGallery

A simple crawler to get all Bing gallery pictures.

14 34 34

proxi

Proxy pool. Finds and checks proxies with rest api for querying result...

4 34 34

toxcrawler

A Tox DHT network crawler

12 34 34

BilibiliCrawler

:cyclone: crawl bilibili user info and video info for data analysis |...

6 34 34

Youtube_Scraper

Scrape data about an entire Channel or just a Playlist, or get stats a...

6 34 34

go-crawler-distributed

分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微...

5 34 34

a11y-sitechecker

Automatic accessibility checker with website crawling + screenshots fo...

4 34 34

courlan

Clean, filter and sample URLs to optimize data collection – includes s...

4 33 33

toutiaocrawler

头条号爬虫案例

14 33 33

WebCrawler

A web crawler based on requests-html, mainly targets for url validatio...

12 33 33

ioweb

Web Scraping Framework

11 33 33

Youtube_Comment_Crawler

유튜브 댓글 크롤러 ( Python, BeautifulSoup, Selenium )

9 33 33

visual-spider

用JavaFX开发基于crawler4j的图形化的网络爬虫

9 32 32

INMET-API-temperature

Crawler dos dados metereológicos de estações convencionais do INMET (B...

7 32 32

2020-nCov-anhui

2020新型冠状病毒疫情数据爬取、可视化、网站开发部署

12 32 32

LOLPrediction

英雄联盟胜负预测

11 32 32

Facebooker

an unofficial facebook api

8 32 32

google_news_scraper_and_sentiment_analyzer

Downloads news articles from Google news and uses pre-trained NLP mode...

11 32 32

pyfutebol

Simples crawler para obter resultados dos jogos de futebol

15 32 32

scalpel

A fast and powerful web scraping library

2 32 32

CobWeb-lnx

CobWeb is a Python library for web scraping. The library consists of t...

0 32 32

learncpp-download

Scrape bot, to get you an offline copy of tutorials

15 31 31

InstaBot

Simple and friendly Bot for Instagram, using Selenium and Scrapy with...

12 31 31

PTTmineR

Parallel Searching and Crawling Data from PTT 🚀

4 31 31

Search_Ads_Web_Service

Online search advertisement platform & Realtime Campaign Monitoring [M...

1 31 31

pysaint

[deprecated] 유세인트 파이썬 클라이언트

3 31 31

Spydan

A web spider for shodan.io without using the Developer API.

8 31 31

spiderable-middleware

🤖 Prerendering for JavaScript powered websites. Great solution for PW...

4 31 31

see

Search Engine in Erlang

3 31 31

instagram_scraper

Extract instagram users informations from hashtags. This scraper can e...

14 31 31

bet365API

The latest way to get bet365 data odds, with a delay of 0.2 seconds be...

8 31 31

octopus

Recursive and multi-threaded broken link checker

11 31 31

Mechanize.NET

Stateful programmatic web browsing, based on Python-Mechanize, which i...

9 30 30

invana-bot

A Web Crawler that scrapes using YAML and python code.

9 30 30

TTBot2.0

app版本今日头条用户登录/个人主页/关注列表/粉丝列表/评论点赞收藏关键...

16 30 30