adbar

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Curated list of open-access/open-source/off-the-shelf resources and tools developed with a particular focus on German

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

Fast and robust date extraction from web pages, with Python or on the command-line