go-crawler-distributed

crawler elasticsearch

分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微服务架构，通过消息队列实现消息的异步发送，使用到的框架包括：redigo, gorm, goquery, easyjson, viper, amqp, zap, go-micro，并通过Docker实现容器化部署，中间爬虫节点支持水平拓展。

How to download and setup go-crawler-distributed

Open terminal and run command

git clone https://github.com/golang-collection/go-crawler-distributed.git

git clone is used to create a copy or clone of go-crawler-distributed repositories. You pass git clone a repository URL.
it supports a few different network protocols and corresponding URL formats.

Also you may download zip file with go-crawler-distributed https://github.com/golang-collection/go-crawler-distributed/archive/master.zip

Or simply clone go-crawler-distributed with SSH

[email protected]:golang-collection/go-crawler-distributed.git

If you have some problems with go-crawler-distributed

You may open issue on go-crawler-distributed support forum (system) here: https://github.com/golang-collection/go-crawler-distributed/issues

Similar to go-crawler-distributed repositories

Here you may see go-crawler-distributed alternatives and analogs

scrapy grafana elasticsearch Sasila Price-monitor FOSElasticaBundle webmagic colly headless-chrome-crawler Lulu newcrawler scrapple goose-parser arachnid gopa scrapy-zyte-smartproxy bookbrainz-site elastic4s elk-docker dev-setup Opserver elasticsearch-HQ node-crawler pipeline sentinl awesome-aws yii2-elasticsearch great-big-example-application arachni newspaper