robots.txt

gradle crawler

:robot: robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

How to download and setup robots.txt

Open terminal and run command

git clone https://github.com/fooock/robots.txt.git

git clone is used to create a copy or clone of robots.txt repositories. You pass git clone a repository URL.
it supports a few different network protocols and corresponding URL formats.

Also you may download zip file with robots.txt https://github.com/fooock/robots.txt/archive/master.zip

Or simply clone robots.txt with SSH

[email protected]:fooock/robots.txt.git

If you have some problems with robots.txt

You may open issue on robots.txt support forum (system) here: https://github.com/fooock/robots.txt/issues

Similar to robots.txt repositories

Here you may see robots.txt alternatives and analogs

scrapy Sasila Price-monitor android-tips-tricks kotlin-examples webmagic colly headless-chrome-crawler Lulu newcrawler scrapple goose-parser arachnid gopa scrapy-zyte-smartproxy dryrun node-crawler MaterialStyledDialogs BottomDialogs arachni Charlatano gradle AndroidDevTools jib packer-ng-plugin walle hibernate-orm Robust T-MVP kotlin-dsl-samples