robots.txt

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

search-engine crawling

View on GitHub Website

86 Stars

38 Forks

86 Watchers

other License

Cost to Build

$9.8K

Market Value

$25.3K

How is this calculated?

Growth over time

11 data points · 2021-08-01 → 2025-07-01

Stars Forks Watchers

💬

How do you feel about this project?

Ask AI about robots.txt

Question copied to clipboard

What is the jonasjacek/robots.txt GitHub project? Description: "Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.". Explain what it does, its main use cases, key features, and who would benefit from using it.

Question is copied to clipboard — paste it after the AI opens.

How to clone robots.txt

Clone via HTTPS

git clone https://github.com/jonasjacek/robots.txt.git

Clone via SSH

[email protected]:jonasjacek/robots.txt.git

Download ZIP

Download master.zip

Found an issue?

Report bugs or request features on the robots.txt issue tracker:

Open GitHub Issues

Similar to robots.txt

elasticsearch learn-anything Mailpile riot Jets.js magnetissimo magnetico tntsearch RediSearch poseidon tantivy github-awesome-autocomplete opensse DuckieTV ambar search_cop fsearch FileMasta picky meta instantsearch-ios quark elasticsuite typesense algoliasearch-wordpress search-plugins fess opensearchserver resin algoliasearch-client-php