top-crawler-agents

Kikobeats

Curated list of common web crawler and bot User-Agent strings used across the internet.

scraping list

View on GitHub Website

29 Stars

2 Forks

1 Watchers

JavaScript Language

mit License

89.6 SrcLog Score

Cost to Build

$5.0K

Market Value

$9.3K

How is this calculated?

Growth over time

4 data points · 2026-08-10 → 2026-09-09

Stars Forks Watchers

💬

How do you feel about this project?

Ask AI about top-crawler-agents

Question copied to clipboard

What is the Kikobeats/top-crawler-agents GitHub project? Description: "Curated list of common web crawler and bot User-Agent strings used across the internet.". Written in JavaScript. Explain what it does, its main use cases, key features, and who would benefit from using it.

Question is copied to clipboard — paste it after the AI opens.

How to clone top-crawler-agents

Clone via HTTPS

git clone https://github.com/Kikobeats/top-crawler-agents.git

Clone via SSH

[email protected]:Kikobeats/top-crawler-agents.git

Download ZIP

Download master.zip

Found an issue?

Report bugs or request features on the top-crawler-agents issue tracker:

Open GitHub Issues

Similar to top-crawler-agents

free-programming-books github-cheat-sheet awesome-nodejs frontend-dev-bookmarks HEAD awesome-interview-questions awesome-macos-command-line open-source-ios-apps awesome-swift awesome-electron awesome-cpp awesome-shell css-protips awesome-remote-job awesome-docker react-virtualized htaccess awesome-graphql Machine-Learning-Tutorials pure-bash-bible awesome-falsehood interview stack-on-a-budget movies-for-hackers awesome-stock-resources terminals-are-sexy awesome-wpo awesome-vscode awesome-for-beginners redisson