WebCrawler
thomasgottvalles
This Python program is a bot designed to explore the pages of a website (crawling), extract the hyperlinks from each page and store them for later use. Each hyperlink on the website is tested with a request to obtain the HTTP response code (200, 404, 403, 500 etc) and to extract internal and external links from the page content (scraping).
Growth over time
1 data points · 2025-07-23 → 2025-07-23How do you feel about this project?
Ask AI about WebCrawler
Question copied to clipboardWhat is the thomasgottvalles/WebCrawler GitHub project? Description: "This Python program is a bot designed to explore the pages of a website (crawling), extract the hyperlinks from each page and store them for later use. Each hyperlink on the website is tested with a request to obtain the HTTP response code (200, 404, 403, 500 etc) and to extract internal and external links from the page content (scraping).". Written in Python. Explain what it does, its main use cases, key features, and who would benefit from using it.
Question is copied to clipboard — paste it after the AI opens.
How to clone WebCrawler
Clone via HTTPS
Clone via SSH
Download ZIP
Download master.zipFound an issue?
Report bugs or request features on the WebCrawler issue tracker:
Open GitHub Issues