Most popular crawler repositories and open source projects

2017_PyConTW_Talk

4 19 19

botanalyse

botsonar analyse open api

3 19 19

scrapher

A web scraper for PHP to easily extract data from web pages

13 19 19

magento2-module-primer

Full Page Cache Priming tool for Magento 2

7 19 19

crawler

Nodejs crawler for cnbeta.com

10 19 19

baiduyun_spider

Python + MongoDB 开发的百度云资源爬虫

15 19 19

crawl

A concurrent crawler that minimizes memory use. Output suitable for us...

7 19 19

google-scholar-crawler

A crawler to crawl google scholar search page

13 19 19

sentinel-cendertron

Cendertron = Crawler + cendertron, Crawl AJAX-heavy client-side Single...

5 19 19

NEEA-TOEFL-Testseat-Crawler

托福考位爬虫 NEEA TOEFL Testseat Crawler

0 19 19

gdpr-scanner

A tool to check a list of domains for violations against the GDPR :mag...

2 19 19

dijnet-bot

Az összes számlád még egy helyen :)

0 19 19

mvcrawler

动漫聚合小站

3 19 19

AliCouponHunter

Aliexpress coupon search | Find cheapest item and show possible coupon...

7 19 19

spiderman

your friendly neighborhood web crawler

4 19 19

screamingFrogR

R integration with Screaming Frog CLI

3 19 19

playwright-webcrawler

Parallel crawler powered by Playwright-Python

7 19 19

plusfish

Plusfish is a classic web application vulnerability scanner/fuzzer and...

9 19 19

Broken-Links-Crawler-Action

GitHub Action to check a website for broken links

2 19 19

scrapy-diario-oficial-da-uniao

Script Python para buscar o conteúdo do Diário Oficial da União

5 19 19

bilibili_comment_crawl

爬取bilibili视频下的评论，最新出品！！！⚠本代码只适用于学习，做其他事...

0 19 19

websight

🕷A simple but *really* fast crawler built with Node.js & TypeScript

14 18 18

google-play-crawler

Crawler for google play to crawl all the app related data

17 18 18

Academic-Paper-Title-Recommendation

Supervised text summarization (title generation/recommendation) based...

1 18 18

magnet-crawler

一个磁力链接的爬虫。

13 18 18

Sharingan

We will try to find your visible basic footprint from social media as...

6 18 18

XML-Parser

A Node.js XML DOM, Parser & Stringifier.

8 18 18

my-favourite-appliances

Laravel CRUD sample

5 18 18

newspaper-crawler

Scrapy based crawler which crawls newspaper.

3 18 18

Google-Clone-Script

A search engine like Google made using PHP MySQL and JavaScript

17 18 18

crowlet

Tiny sitemap crawler for cache warming, and website status monitoring

1 18 18

WMIRROR

wmirror allows you to download any website from the Internet to a loca...

2 18 18

ActoCrawler

🕸️ Swift Concurrency-powered crawler engine on top of Actomaton.

1 18 18

wind-bell

风铃虫是一款轻量级的爬虫工具，似风铃一样灵敏，如蜘蛛一般敏捷，能感知任...

7 18 18

doogle

Doogle is a search engine and web crawler which can search indexed web...

5 18 18

mbfc_crawler

Crawls Media Bias/Fact Check and saves output to JSON.

6 18 18

MercadoLivreProductsCrawler

PHP Console Crawler to Download Products from a Store on MercadoLivre....

6 18 18

onion-crawler

Tor website crawler (specific for Alphabay at the time)

14 18 18

node-dcard-scraper

it is an example of implementing cheerio scraper of extracting images...

5 18 18

crawler

Web Crawler created with Node.js and Puppeteer

1 18 18

json-web-crawler

Use JSON to list all elements (with css 3 and jquery selector) that yo...

2 18 18

grapy

Grapy, a fast high-level web crawling framework for Python 3.3 or late...

8 18 18

youtube-trends-spider

crawler youtube trends use selenium on python

11 18 18

Email-Extractor

A spider to crawl webpages

3 18 18

arachnod

High performance crawler for Nodejs

2 17 17

shub_cli

A CLI for dealing with the features of ScrapingHub

0 17 17

webhunger

WebHunger is an extensible, full-scale crawler framework that supports...

4 17 17

TripAdvisor-Crawling-Suite

Fetching hotel data from TripAdvisor.

7 17 17

Hackerrank-Solution-Crawler

🐍 Crawls solutions of hackerrank and stores as local files.

8 17 17

crawler-set

各种网站爬虫合集，持续更新中....

16 17 17