3 repositories on SrcLog
A little like that j-thing, only in Go.
Polite, slim and concurrent web crawler.
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.