Most popular crawling repositories and open source projects

awesome-webscraping-blogs

Curated list of technical blogs and videos on web scraping·

1   9   9  

YouTubeChanelsScraper

Program that scrape emails from youtube chanels

2   9   9  

Desktop_App_for_Sitemap_Generator

Sitemap Generator Desktop App For Windows And Linux

0   8   8  

born2crawl

A highly performant and versatile crawling engine, designed with scala...

0   8   8  

bilibili_video_crawing

Python 对哔哩哔哩，B站视频爬取，B站封面原图爬取保存到本地

0   8   8  

crawlbase-ruby

Fast Crawlbase API crawling library

1   8   8  

sher-look

A high-performance search engine that crawls, indexes, and ranks web c...

1   8   8  

crawlbase-node

Fast dependency free library for Crawlbase API

1   8   8  

Library-Data-Assistant

Java-based client-server application for managing library book data wi...

0   8   8  

Recusive-web-crawler

"Recursive Web Crawler: A Python tool for deep website exploration, fi...

3   8   8  

dropship-trend-crawler

A sophisticated data-driven system that revolutionizes product discove...

1   8   8  

TGCrawl

Telegram channel relations analyzer

1   8   8  

AutoTor

Simple package to make requests throughout Tor with circuit renewal.

1   8   8  

web-crawlers

Web Crawl

0   8   8  

leechcrawler

Incremental crawling capabilities for Apache Tika. Crawl content out o...

5   8   8  

Data-Analytics

제품 정보 크롤링 및 리뷰 텍스트 마이닝

3   8   8  

where-is-my-customs

내 통관은 어디쯤? 카카오톡 봇

4   8   8  

Crawling-Book

🧾🔍 끝내주는 크롤링&메크로 스크립트를 작성하는 방법 (with Python)

1   8   8  

simplified-search-engine

Multithreaded Web Crawler, Scraper, Indexer

2   8   8  

DataScrapingCrawling

Data Scraping 정리 자료

6   8   8  

lazada-scraper

https://www.lazada.sg/ using scrapy

6   8   8  

Framework

IoTCrawler Framework

12   8   8  

pattern-grab

🤛🏻 Regular Expression Data Grabber

0   7   7  

golang-scraping-colly

Exemples de récupération de données non structurées avec le framework...

0   7   7  

nutch-webapp

Apache Nutch is an extensible and scalable web crawler

5   7   7  

instagramProfileCrawler

Get latest media from instagram profile without API

0   7   7  

Cars.com-Crawling

A python crawler for cars.com

1   7   7  

leo-bot

📢 디스코드 공식 리오봇 📢

1   7   7  

jsonld-extract

A damn simple tool to extract json-ld metadata from webpage using jque...

0   7   7  

LicencePlateScraper

Système automatique pour constituer un dataset de plaque d'immatricula...

3   7   7  

minigun-requests

Web scraping API to outsource tons of GET & xpath to cloud computing

0   7   7  

web-scraping-template

🚀 THIS WEB SCRAPING TEMPLATE PROVIDES YOU WITH A GREAT STARTING POINT...

3   7   7  

dotlas_odyssey

⛵️ A take-home assignment for the full-time Data Engineering position...

2   7   7  

quotes-crawler

Quotes crawler using scrapy and python.

0   6   6  

telegramBot_instaDP

A simple BOT Telegram to downloading Instagram profiles photo

0   6   6  

Cloud_Player_V2

You can use the cloudplayer tool to listen to the music of the singer...

0   6   6  

fiverr_scraper

This repo contains a Python script that crawls gig information from th...

2   6   6  

crawlee-web-scraping-tutorial

This article covers everything you need to get started with Crawlee. L...

2   6   6  

firecrawler

A lightweight frontend for self-hosted Firecrawl instances

5   6   6  

Web-Crawler

Web Crawler with Python

0   6   6  

Slic

Single line image classifier

3   6   6  

leetcode-summary-crawler

A leetcode crawler built with selenium and requests. Generate a revis...

0   6   6  

ScienceProject

🔭🌦 과학프로젝트, 날씨에 학교를 더하다 (with Django)

1   6   6  

proxypool

A proxy poll: get free and high quality proxies

4   6   6  

chrome-php

A PHP Wrapper for Chrome Headless. Get the DOM of any webpage.

8   6   6  

DBpia_crawler

국내 논문 서지정보 사이트 DBpia 크롤링 프로그램

2   6   6  

crawling-scraping-scripts

Collection of brazilian soccer data crawling/scraping scripts.

0   6   6  

GitHub_Crawling_TextMining_Project

Data collection and processing for intelligent technology ecosystem an...

2   6   6  

actual-deeplearning

파이썬을 이용한 머신러닝, 딥러닝 실전개발 입문

14   6   6  

knu-lms-scheduler

:mortar_board: 공주대학교 온라인 강의 시스템 편의성 향상 프로그램

1   6   6