crawlero
shavit
Distributed web crawlers. Fault tolerance, user-agent randomizer, RabbitMQ, Tor, PostgreSQL.
15
Stars
1
Forks
15
Watchers
Go
Language
Growth over time
5 data points · 2021-08-01 → 2022-09-01
Stars
Forks
Watchers
💬
How do you feel about this project?
Ask AI about crawlero
Question copied to clipboardWhat is the shavit/crawlero GitHub project? Description: "Distributed web crawlers. Fault tolerance, user-agent randomizer, RabbitMQ, Tor, PostgreSQL.". Written in Go. Explain what it does, its main use cases, key features, and who would benefit from using it.
Question is copied to clipboard — paste it after the AI opens.
How to clone crawlero
Clone via HTTPS
git clone https://github.com/shavit/crawlero.git
Clone via SSH
[email protected]:shavit/crawlero.git
Download ZIP
Download master.zipFound an issue?
Report bugs or request features on the crawlero issue tracker:
Open GitHub Issues