crawling
Repositories (1350)
MCP server that enables self-healing automatic repair of Scrapy spiders. When websites change, your scrapers fix themselves.
Search Engine projects
web crawling & scraping framework for Python
Node.js tool for downloading all free MIDI files on VGMusic.com
converts webpage content into Markdown format, optimized for LLM training and context
Replayable Browser Agent
Python module allowing you to do various searches for links on the Web.
A simple web crawller in go
Генератор сырых дампов пользователей VK.
crawling facebok page
Fetch, store and access user agent strings for different browsers
Crawl and track followers count of Twitter account
re-employment-kraken scrapes (job) sites, remembers what it saw and notifies downstream systems of any new sightings.
Engine for collecting onion domains and crawling from webpage based on Tor network
2023.11) velog statistics dashboard fullstack
An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL
An ultra small PoC to show how to combine Apache Nutch and Apache Solr, crawling through web pages and storing the results in Solr for quering
🚀 OMKAR TEMP MAIL HELPS YOU USE TEMPORARY EMAILS. 🤖
A full stack application that scrapes & filters YouTube comments using Google's Puppeteer, instead of using the YouTube API
🕷️ Adaptive web scraping skill for OpenClaw agents — bypasses anti-bot, survives site redesigns. Powered by MyClaw.ai
A lightweight frontend for self-hosted Firecrawl instances
Fast extraction of all external links from wikipedia
App to scrap the web, for people without coding skills. Fully integrates WebCrawlers (Headless Chrome) and the interface to deal with it.
Fast, parallel and easy to use web crawler for penetration testing and bug bounty
Đồ án cuối kì môn khoa học dữ liệu ứng dụng. Thu thập data bằng cách parsing HTML và sử dụng các mô hình học máy để giải quyết câu hỏi được đặt ra ba...
东方财富网股票数据爬取
Uses Sankey Diagrams to visualize politicians that have "crossed the floor" from election to election.
A django application for scraping properties with scrapy.
Simple Manga Downloader, a tool to search and download manga
You Can Download Instagram Post With This Script
This is a crawler for crawling papers from google scholar (http://scholar.google.com). Credits for this code goes to (https://github.com/ckreibich/sch...
Telegram channel relations analyzer
Demonstration for crawling Laptop products on Tiki ecomercial website
Crawling route waypoints for HK bus routes
Extraction, versioning and machine-readable provisioning of public data.
🕷️ Easily scrap the web for torrent and media files.
sᴇᴀʀᴄʜ ᴇɴɢɪɴᴇ sᴄʀᴀᴘᴇʀ ᴛᴏᴏʟ (ʙᴀsʜ)
Crawler written in TypeScript using ES6 generators.
Python scripts, first traverses chrome Bookmark file and second removes stale entries. Includes Jenkinsfile to generate docker images.
Intelligent web discovery agent with LLM-powered planning, multi-source search, smart deduplication, and GRPO preference dataset collection. Autonomou...
[ACL 2024] Evaluation of the Fundus News Scraper
Simple scripts for crawling shopee's shop and product information from shopee.vn
crawling china stock recommendation from Sina Weibo, create pyecharts for data
파이썬을 활용한 실전 웹크롤링 CAMP 강의 1-2기 소스코드
Docker🐳 setup for automated news article crawling from German news websites. Written in Python🐍, uses MongoDB
Crawl Anne Shirley's Quotes from Web | استخراج نقل قول های آن شرلی از وب
A sophisticated data-driven system that revolutionizes product discovery for dropshipping businesses. Unlike traditional web crawlers, this platform l...
Python script for crawling ResearchGate.net papers.✨⭐️📎
Python web crawler tool
This Python script extracts comprehensive movie data from IMDB, focusing on top-grossing movies from 1920 to 2025. The scraper collects detailed infor...