Most popular crawler repositories and open source projects

fuli

A crawler which used chrome puppeteer.

1   11   11  

SongsPk-MusicScrapper

A Python Script which scrapes out all the Songs URL from SongsPK and s...

5   11   11  

ps5-bot

Bot for crawling popular polish shops checking for PS5 avalability

1   11   11  

fzutils

🍺 这是fz的python utils包, for Spider! enjoy!

7   11   11  

multitor

Make multiple Tor instances in Node.Js

1   11   11  

crawler_zhihu

知乎爬虫并做简单数据分析(大V关系链)

2   11   11  

linkbak

linkbak is a web page archiver : it reads a list of links and dumps th...

0   11   11  

aioscrapy

Python asynchronous library for web scrapping

3   11   11  

anp-price-collector

ANP Price Collector

4   11   11  

scrape-them-all

🚀 An easy-to-handle Node.js scraper that allow you to scrape them all...

0   11   11  

baidu-search-result-crawler

一个百度搜索结果内容获取爬虫。

5   11   11  

rxcrawler

a java crawler base on rx-java

5   11   11  

xiaohongshu-spider-visualizer

A distributed web crawler for xiaohongshu.com and visualization for th...

6   11   11  

insta-downloader

You Can Download Instagram Post With This Script

3   11   11  

newscorpus

Docker🐳 setup for automated news article crawling from German news web...

2   11   11  

PLeagueBot

P+ League Chatbot(unofficial)

1   11   11  

SECTOOL

sᴇᴀʀᴄʜ ᴇɴɢɪɴᴇ sᴄʀᴀᴘᴇʀ ᴛᴏᴏʟ (ʙᴀsʜ)

3   11   11  

xueqiu_spider

雪球爬虫，爬取长生生物10000+股友评论

3   11   11  

KnightReport

坎公骑冠剑会战统计工具

2   11   11  

httpscan

Scan a host for open HTTP ports and gain information about the service...

2   11   11  

installer

Installation script for SCNR

3   11   11  

crawler-news

Use python scrapy build crawler for real-time Taiwan NEWS website.

3   10   10  

Machine_Learning_Focused_Crawler

A focused web crawler that uses Machine Learning to fetch better relev...

3   10   10  

StoreReq

Nodejs/Crawler

0   10   10  

gollum

🤖 Robots.txt parser and fetcher for Elixir

6   10   10  

tradePy

Project for auto trading in Taiwan Stock Price Index Futures

2   10   10  

scrapy-blog-crawler

Crawl a blog url, and find all url from it, then save to mysql.

7   10   10  

goods-crawling

爬取amazon/bestbuy/costco/6pm 的商品详情

4   10   10  

all-it-ebooks-downloader

Python based crawler and downloader for books in allitebooks.com

4   10   10  

node-crawler-on-mongodb

🕷 NodeJS + Puppeteer crawler on MongoDB

2   10   10  

jiandan

Java / Python 图片爬虫

4   10   10  

RsparkleR

RsparkleR provides an R interface for launching virtual machines and d...

1   10   10  

python

python 脚本、python 爬虫、python 工具

2   10   10  

crawler-benchmark

A Reference Framework for the Automated Exploration of Web Application...

2   10   10  

LeetHub

An Android Client for LeetCode

2   10   10  

Siga-Bot

A simple discord bot that access the UFRJ SIGA and download the wanted...

0   10   10  

douban-movie

🎬a website based on python (flask) and react , for data display and da...

2   10   10  

Fetch-Crawler

📌 A Node.JS Web crawler using the API Fetch to scrap static websites

3   10   10  

metacritic-crawler

Scrapper of metacritic.com written in Python for educational purposes...

4   10   10  

NCrawler

基于nodejs编写的一套网页数据采集框架,开发者只需要简单编写网页解析器即...

1   10   10  

Covid19_Stats

코로나-19 에 대한 확진/완치/사망 에 대한 국내, 해외 정보를 수집합니다....

6   10   10  

rent-house

A simple 591 crawler

0   10   10  

smd

Simple Manga Downloader, a tool to search and download manga

0   9   9  

Web-Spider

Multi threaded Web crawler

2   9   9  

crawler

A toolchain for bringing web2 to web3

2   9   9  

Vison-WoC-Backend

Repository for Vison Backened for Winter of Code 2019.

6   9   9  

1024-web-crawler

🚗🚗1024社区单线程图片爬虫

9   9   9  

Spidey

A multi threaded web crawler library that is generic enough to allow d...

4   9   9  

cnki_crawler

中国知网文献爬虫

5   9   9  

movie_finder

Web crawler that gives a list of movie with download links from yts.ag...

0   9   9