crawling

My first web scraper! I used this program to get some whiskey info.

AWS Lambda service layer by Youtube data selenium crawling

Web crawler using Playwright. It extracts URLs from a given website and saves them in either JSON or TXT format. It includes options to skip crawling...

Simple data crawler for some radiograph diagnosis quizzes

Webcrawl is a Python web crawler that recursively follows links from a starting URL to extract and print unique HTTP links. Using 'requests and 'Beaut...

Complete pipeline for obtaining LLM training data at scale

Smart crawling request utility for Python.

News crawler

Retrieve an old market

🔔 Quick and easy way to get notified from all kind of new free games available from different platforms to claim.

powerful scripts allow you search all email address under you desired URL with different options, 功能强大的网站邮箱爬虫

🤖 Ready-to-run Python examples for Browse Anything API. Automate web scraping, price monitoring, multi-step workflows, QA testing, and lead generatio...

🏢 Manage employee data efficiently with this RESTful API featuring full CRUD operations using Node.js, Express.js, and Azure SQL Database.

A wrapper for the more cheaper alternative - scrappey - for scraping 40+ sites

A high-performance, concurrent web crawler in Go that extracts URLs, downloads content, and converts tens of thousands of web pages to Markdown in min...

Elegant Scraper and Crawler Framework for Golang

A .NET console application that searches multiple search engines for email addresses, validates them, and stores them in a SQL Server database. Built...

Web scraping and crawling with Firecrawl API - markdown conversion, screenshots, structured data extraction

Open-source SEO CLI — crawl, audit, SERP analysis, backlinks, and keyword research from the command line

A CLI-friendly crawler that can optionally authenticate, crawl a website, and mirror pages and static assets into a local directory so the result can...

A GitHub release searcher that searches for repositories with certain file types in the releases. Made in HTML and JS.

Scrape Kurdish Kurmanji pages

An open-source meta-search engine for AI.

An open source, rust-based tool for local load tests and checking all interactive elements, to make sure that a user won't encounter an issue

Un crawler web récursif, rapide et efficace

A bittorrent dht scraper

Crawlify is an efficient web scraping tool designed to help developers, researchers, and businesses extract, analyze, and automate data collection fro...

EMail Miner Pro is designed specifically for professionals scraping data from search engines like Google, ensuring that generic emails (e.g., Gmail, Y...

🔍 Extract job details from Germany’s employment portal and convert them into structured datasets for efficient analysis of the job market.

Multi-Level Approach to Managing AI Crawler Behaviour and Content Protection for the IAB Workshop on AI-CONTROL 2024/25

Async-first web scraping framework

A collection of ASP.NET web crawlers for extracting email addresses from online sources including job sites, APIs, public pages. Built in 2012–2016, t...

AI-powered web scraper. Give it a URL and tell it what data you want — it handles the rest.

A complete 2-hour training on Web Scraping with Python, featuring practical projects using Camoufox, Playwright, and Scrapy. Includes real-world crawl...

Simplify web scraping by extracting data from modern websites with an easy-to-use Python library designed for efficiency and clarity.

Original trilogy, prelogy and postlogy introductions of Star Wars in CSS3

Lightweight Python web scanner with BFS crawling, form analysis, multi-threaded requests, and automated tests for XSS, SQLi, and missing security head...

🕷️ Discover and use popular web crawlers across various programming languages to efficiently extract data from the web.

Simple yet flexible URL crawler.

Random Proxy Wrapper for Python Requests

textmining project in Github

under development

clojure-crawling

Very basic proxy

Crawling many images in google

Kotlin Web Crawler library

crawling system

Crawls job advertisements from a popular spanish site using bs4