Most popular crawler repositories and open source projects

soducrawler

12 35 35

schannel-qt5

A GUI client of schannel powered by therecipe/qt and golang

5 35 35

lostark-wait-notifier

🐤️ Lost Ark wait notifier

7 35 35

crawler

Crawler with Python 3.

19 35 35

ArticleSpider

Crawling zhihu, jobbole, lagou by Scrapy, and using Elasticsearch+Djan...

10 35 35

shadow_spider

14 35 35

crawlerdetect

🕷CrawlerDetect is a Python class for detecting bots/crawlers/spiders v...

6 35 35

DDoM

A simple, open-source, easy to use, and free download manager for malw...

3 35 35

Bing-Wallpaper-Action

API with Redis / Vercel , DataBase with Json, Crawel with Github Actio...

6 35 35

NetEaseCloudMusicCrawler

HttpClient + Jsoup + Queue

14 34 34

imooc-crawler

[Obsolete] imooc web crawler in Node.js（使用 Node.js 编写的慕课网爬虫...

15 34 34

serverless-instagram-crawler

serverless, instagram hashtag crawler with lambda, dynamoDB

8 34 34

phpwebcrawler

A Web Crawler Created in PHP

32 34 34

ebedke

crawl pages to check what is for lunch today

5 34 34

BingGallery

A simple crawler to get all Bing gallery pictures.

14 34 34

proxi

Proxy pool. Finds and checks proxies with rest api for querying result...

4 34 34

toxcrawler

A Tox DHT network crawler

12 34 34

BilibiliCrawler

:cyclone: crawl bilibili user info and video info for data analysis |...

6 34 34

Youtube_Scraper

Scrape data about an entire Channel or just a Playlist, or get stats a...

6 34 34

go-crawler-distributed

分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微...

5 34 34

a11y-sitechecker

Automatic accessibility checker with website crawling + screenshots fo...

4 34 34

toutiaocrawler

头条号爬虫案例

14 33 33

WebCrawler

A web crawler based on requests-html, mainly targets for url validatio...

12 33 33

ioweb

Web Scraping Framework

11 33 33

Youtube_Comment_Crawler

유튜브 댓글 크롤러 ( Python, BeautifulSoup, Selenium )

9 33 33

courlan

Clean, filter and sample URLs to optimize data collection – includes s...

4 33 33

visual-spider

用JavaFX开发基于crawler4j的图形化的网络爬虫

9 32 32

INMET-API-temperature

Crawler dos dados metereológicos de estações convencionais do INMET (B...

7 32 32

node-html-crawler

Simple for use node html crawler (spider) of site web pages

12 32 32

2020-nCov-anhui

2020新型冠状病毒疫情数据爬取、可视化、网站开发部署

12 32 32

LOLPrediction

英雄联盟胜负预测

11 32 32

Facebooker

an unofficial facebook api

8 32 32

google_news_scraper_and_sentiment_analyzer

Downloads news articles from Google news and uses pre-trained NLP mode...

11 32 32

pyfutebol

Simples crawler para obter resultados dos jogos de futebol

15 32 32

scalpel

A fast and powerful web scraping library

2 32 32

CobWeb-lnx

CobWeb is a Python library for web scraping. The library consists of t...

0 32 32

InstaBot

Simple and friendly Bot for Instagram, using Selenium and Scrapy with...

12 31 31

PTTmineR

Parallel Searching and Crawling Data from PTT 🚀

4 31 31

Search_Ads_Web_Service

Online search advertisement platform & Realtime Campaign Monitoring [M...

1 31 31

pysaint

[deprecated] 유세인트 파이썬 클라이언트

3 31 31

Spydan

A web spider for shodan.io without using the Developer API.

8 31 31

spiderable-middleware

🤖 Prerendering for JavaScript powered websites. Great solution for PWA...

4 31 31

see

Search Engine in Erlang

3 31 31

instagram_scraper

Extract instagram users informations from hashtags. This scraper can e...

14 31 31

bet365API

The latest way to get bet365 data odds, with a delay of 0.2 seconds be...

8 31 31

octopus

Recursive and multi-threaded broken link checker

11 31 31

learncpp-download

Scrape bot, to get you an offline copy of tutorials

15 31 31

Mechanize.NET

Stateful programmatic web browsing, based on Python-Mechanize, which i...

9 30 30

invana-bot

A Web Crawler that scrapes using YAML and python code.

9 30 30

TTBot2.0

app版本今日头条用户登录/个人主页/关注列表/粉丝列表/评论点赞收藏关键...

16 30 30