scrapy-cluster

scraping distributed redis kafka

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

How to download and setup scrapy-cluster

Open terminal and run command

git clone https://github.com/istresearch/scrapy-cluster.git

git clone is used to create a copy or clone of scrapy-cluster repositories. You pass git clone a repository URL.
it supports a few different network protocols and corresponding URL formats.

Also you may download zip file with scrapy-cluster https://github.com/istresearch/scrapy-cluster/archive/master.zip

Or simply clone scrapy-cluster with SSH

[email protected]:istresearch/scrapy-cluster.git

If you have some problems with scrapy-cluster

You may open issue on scrapy-cluster support forum (system) here: https://github.com/istresearch/scrapy-cluster/issues

Similar to scrapy-cluster repositories

Here you may see scrapy-cluster alternatives and analogs

tensorflow scrapy CNTK diaspora requests-html Qix awesome-cheatsheets phpredis blog NodeBB medis technology-talk handson-ml Sasila dynomite ardb keyv redislite infinit diplomat elasticell olric acl qTox LightGBM h2o-3 catboost SncRedisBundle distributed tns