Most popular crawler repositories and open source projects

AliCouponHunter

Aliexpress coupon search | Find cheapest item and show possible coupon...

7   19   19  

spiderman

your friendly neighborhood web crawler

4   19   19  

screamingFrogR

R integration with Screaming Frog CLI

3   19   19  

ptt-crawler

ptt-crawler is a web crawler module designed to scarpe data from Ptt.

8   19   19  

playwright-webcrawler

Parallel crawler powered by Playwright-Python

7   19   19  

plusfish

Plusfish is a classic web application vulnerability scanner/fuzzer and...

9   19   19  

Broken-Links-Crawler-Action

GitHub Action to check a website for broken links

2   19   19  

scrapy-diario-oficial-da-uniao

Script Python para buscar o conteúdo do Diário Oficial da União

5   19   19  

bilibili_comment_crawl

爬取bilibili视频下的评论，最新出品！！！⚠本代码只适用于学习，做其他事...

0   19   19  

scrapher

A web scraper for PHP to easily extract data from web pages

13   18   18  

mbfc_crawler

Crawls Media Bias/Fact Check and saves output to JSON.

6   18   18  

MercadoLivreProductsCrawler

PHP Console Crawler to Download Products from a Store on MercadoLivre....

6   18   18  

onion-crawler

Tor website crawler (specific for Alphabay at the time)

14   18   18  

node-dcard-scraper

it is an example of implementing cheerio scraper of extracting images...

5   18   18  

go-scrapy

Web crawling and scraping framework for Golang

4   18   18  

crawler

Web Crawler created with Node.js and Puppeteer

1   18   18  

json-web-crawler

Use JSON to list all elements (with css 3 and jquery selector) that yo...

2   18   18  

grapy

Grapy, a fast high-level web crawling framework for Python 3.3 or late...

8   18   18  

youtube-trends-spider

crawler youtube trends use selenium on python

11   18   18  

Email-Extractor

A spider to crawl webpages

3   18   18  

websight

🕷A simple but *really* fast crawler built with Node.js & TypeScript

14   18   18  

google-play-crawler

Crawler for google play to crawl all the app related data

17   18   18  

Academic-Paper-Title-Recommendation

Supervised text summarization (title generation/recommendation) based...

1   18   18  

magnet-crawler

一个磁力链接的爬虫。

13   18   18  

Sharingan

We will try to find your visible basic footprint from social media as...

6   18   18  

my-favourite-appliances

Laravel CRUD sample

5   18   18  

newspaper-crawler

Scrapy based crawler which crawls newspaper.

3   18   18  

master-to-pythonista

A list of awesome beginners-friendly projects.

19   18   18  

Google-Clone-Script

A search engine like Google made using PHP MySQL and JavaScript

17   18   18  

crowlet

Tiny sitemap crawler for cache warming, and website status monitoring

1   18   18  

froxy

Hide your IP with free proxies using Froxy 🔄

1   18   18  

udemyscraper

A Udemy Course Scraper built with bs4 and selenium, that fetches udemy...

10   18   18  

WMIRROR

wmirror allows you to download any website from the Internet to a loca...

2   18   18  

ActoCrawler

🕸️ Swift Concurrency-powered crawler engine on top of Actomaton.

1   18   18  

crawl-original-google-images

python scripts for crawling original image from Google Images

2   18   18  

billboard-json

🎧 Get json type billboard hot 100 chart

2   18   18  

wind-bell

风铃虫是一款轻量级的爬虫工具，似风铃一样灵敏，如蜘蛛一般敏捷，能感知任...

7   18   18  

doogle

Doogle is a search engine and web crawler which can search indexed web...

5   18   18  

arachnod

High performance crawler for Nodejs

2   17   17  

shub_cli

A CLI for dealing with the features of ScrapingHub

0   17   17  

webhunger

WebHunger is an extensible, full-scale crawler framework that supports...

4   17   17  

TripAdvisor-Crawling-Suite

Fetching hotel data from TripAdvisor.

7   17   17  

udemy-crawler

Crawling Udemy course info and save into JSON format.

5   17   17  

Hackerrank-Solution-Crawler

🐍 Crawls solutions of hackerrank and stores as local files.

8   17   17  

crawler-set

各种网站爬虫合集，持续更新中....

16   17   17  

photo-spider-scrapy

10 photo website spiders, 10 个国外图库的 scrapy 爬虫代码

10   17   17  

Douban-Crawler

抓取豆瓣小组相关信息（小组、用户、帖子）。

8   17   17  

images-downloader

A Node.js module for downloading a single image or multiple images to...

10   17   17  

XML-Parser

A Node.js XML DOM, Parser & Stringifier.

8   17   17  

news-sentiment-analysis

The spider crawls moneycontrol.com and economictimes.com to fetch news...

5   17   17