web-languages
commoncrawl
Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ for the code
46
Stars
56
Forks
46
Watchers
Growth over time
1 data points · 2025-07-23 → 2025-07-23
Stars
Forks
Watchers
💬
How do you feel about this project?
Ask AI about web-languages
Question copied to clipboardWhat is the commoncrawl/web-languages GitHub project? Description: "Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ for the code". Explain what it does, its main use cases, key features, and who would benefit from using it.
Question is copied to clipboard — paste it after the AI opens.
How to clone web-languages
Clone via HTTPS
git clone https://github.com/commoncrawl/web-languages.git
Clone via SSH
[email protected]:commoncrawl/web-languages.git
Download ZIP
Download master.zipFound an issue?
Report bugs or request features on the web-languages issue tracker:
Open GitHub IssuesSimilar to web-languages
scrapy
newspaper
colly
headless-chrome-crawler
Lulu
isp-data-pollution
Sasila
webster
cdp4j
crawler
spidy
stopstalk-deployment
N2H4
memorious
easy-scraping-tutorial
antch
pomp
Harvester
diffbot-php-client
talospider
corpuscrawler
Python-Crawling-Tutorial
learn.scrapinghub.com
crawling-projects
dig-etl-engine
crawlkit
scrapy-selenium
spidyquotes
zcrawl
podcastcrawler