Most popular crawling repositories and open source projects

scrapy-scraper ivan-sincek Python

Web crawler and scraper based on Scrapy and Playwright's headless browser.

17 7 1

scrapy-mcp-server scrapoxy

MCP server that enables self-healing automatic repair of Scrapy spiders. When websites change, your scrapers fix themselves.

17 1 0

Google-Search-URL-Crawler ElektroStudios Visual Basic .NET

Desktop app that crawls urls from Google's search engine results

17 2 1

WebSearch iTeam-S Python

Python module allowing you to do various searches for links on the Web.

17 8 0

pyReptile xyjw Python

web crawling & scraping framework for Python

16 7 16

node-vgmusic-downloader gogson JavaScript

Node.js tool for downloading all free MIDI files on VGMusic.com

16 5 1

browser-agent lightfeed TypeScript

Replayable Browser Agent

16 0 1

webscrape-tutorial octobug1 Python

A basic tutorial to web scraping using python for beginners

16 0 0

free-llmstxt-generator moinulmoin TypeScript

converts webpage content into Markdown format, optimized for LLM training and context

16 1 1

crawlly gaurav-gogia Go

A simple web crawller in go

15 10 0

kasthack.osp kasthack-labs C#

Генератор сырых дампов пользователей VK.

15 5 1

facebook-scraper-for-non-english-user dizwe Python

crawling facebok page

15 4 2

pumba sultaniman Elixir

Fetch, store and access user agent strings for different browsers

15 1 1

re-employment-kraken uschtwill JavaScript

re-employment-kraken scrapes (job) sites, remembers what it saw and notifies downstream systems of any new sightings.

15 1 2

velog-dashboard Check-Data-Out JavaScript

2023.11) velog statistics dashboard fullstack

15 1 0

twitter-account-data-crawler somniLegacy Python

Crawl and track followers count of Twitter account

15 2 0

firecrawler sammcj JavaScript

A lightweight frontend for self-hosted Firecrawl instances

15 12 1

darklight bunseokbot Python

Engine for collecting onion domains and crawling from webpage based on Tor network

15 4 1

img-cli selmi-karim JavaScript

An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL

14 3 0

awesome-scraping ScrapeRouter Python

The definitive list of the latest libraries, tools, APIs and providers for web scraping. The only daily-updated collection of web scraping resources.

14 2 0

nutch-solr-integration basraven

An ultra small PoC to show how to combine Apache Nutch and Apache Solr, crawling through web pages and storing the results in Solr for quering

14 9 0

Octocrawl b3rt1ng Python

Fast, parallel and easy to use web crawler for penetration testing and bug bounty

14 1 0

route-waypoints hkbus Python

Crawling route waypoints for HK bus routes

14 13 3

Arachnida guillim JavaScript

App to scrap the web, for people without coding skills. Fully integrates WebCrawlers (Headless Chrome) and the interface to deal with it.

14 12 3

omkar-temp-mail omkarcloud Python

🚀 OMKAR TEMP MAIL HELPS YOU USE TEMPORARY EMAILS. 🤖

14 4 1

FinalProject-Datascience HungTrinhIT Jupyter Notebook

Đồ án cuối kì môn khoa học dữ liệu ứng dụng. Thu thập data bằng cách parsing HTML và sử dụng các mô hình học máy để giải quyết câu hỏi được đặt ra ba...

14 6 1

clever_searcher Azzedde Python

Intelligent web discovery agent with LLM-powered planning, multi-source search, smart deduplication, and GRPO preference dataset collection. Autonomou...

13 0 1

React-YouTube-Comment-Section-Scraper MikeM711 JavaScript

A full stack application that scrapes & filters YouTube comments using Google's Puppeteer, instead of using the YouTube API

13 6 1

TGCrawl Puzzaks Dart

Telegram channel relations analyzer

13 3 1

wikipedia-externallinks-fast-extraction lovasoa Rust

Fast extraction of all external links from wikipedia

13 3 0

Web-Crawling-Stock-Data- Ariannahs Jupyter Notebook

东方财富网股票数据爬取

13 5 2

dropship-trend-crawler nabz0r JavaScript

A sophisticated data-driven system that revolutionizes product discovery for dropshipping businesses. Unlike traditional web crawlers, this platform l...

13 4 1

house-bob peterwade153 Python

A django application for scraping properties with scrapy.

13 7 1

smd adbenitez Python

Simple Manga Downloader, a tool to search and download manga

12 0 1

insta-downloader amirzenoozi Python

You Can Download Instagram Post With This Script

12 3 1

googlescholar-crawler debanjanmahata

This is a crawler for crawling papers from google scholar (http://scholar.google.com). Credits for this code goes to (https://github.com/ckreibich/sch...

12 0 0

crawlee-web-scraping-tutorial oxylabs JavaScript

This article covers everything you need to get started with Crawlee. Learn more about its benefits and see a working example of scraping a website wit...

12 5 1

mb-checker juvalen Python

Python scripts, first traverses chrome Bookmark file and second removes stale entries. Includes Jenkinsfile to generate docker images.

12 0 1

fundus-evaluation dobbersc Python

[ACL 2024] Evaluation of the Fundus News Scraper

12 1 2

Scraping-IMDB RaedAddala Python

This Python script extracts comprehensive movie data from IMDB, focusing on top-grossing movies from 1920 to 2025. The scraper collects detailed infor...

12 4 1