Most popular crawling repositories and open source projects

popular_restaurants_from_officials

서울시 공무원의 업무추진비를 분석하여 진짜 맛집 찾기 프로젝트

7   26   26  

covid-social-analysis

ML-based analysis of social media users' attentiveness under the COVID...

2   26   26  

puppet-master

Puppeteer as a service hosted on Saasify.

8   25   25  

CrowLeer

Powerful C++ web crawler based on libcurl

3   25   25  

aioscpy

An asyncio + aiolibs crawler imitate scrapy framework

6   25   25  

Mimo-Crawler

A web crawler that uses Firefox and js injection to interact with webp...

2   24   24  

crawlkit

A crawler based on Phantom. Allows discovery of dynamic content and su...

7   23   23  

crawling-framework

Easily crawl news portals or blog sites using Storm Crawler.

3   22   22  

html-article-extractor

A web page content extractor

1   22   22  

amharic_spell_corrector

Amharic Spelling Corrector based on SymSpell - Spelling corrector whic...

9   22   22  

zcrawl

An open source web crawling platform

4   21   21  

trend-monitoring

실시간 트렌드 데이터 분석/모니터링 시스템 tremo

4   21   21  

SlackWebhooksGithubCrawler

Search for Slack Webhooks token publicly exposed on Github

1   21   21  

scrapy-requests

Scrapy middleware to handle javascript pages using requests-html

1   21   21  

proxycrawl-node

ProxyCrawl Node library for scraping and crawling

5   20   20  

DarkWeb-Crawling-Indexing

A DarkWeb Crawler based off the open-source TorSpider. Indexing with s...

14   20   20  

crawler

Crawl your own website with various clients for SEO and indexing purpo...

4   19   19  

xXx___dead___xXx

b̶̡̪̬͒l̸̰̗̝̀ỏ̷̡̩g̴͇̑g̶̲̱̽͐i̵̹͗n̶̤̥͂̅̆g̴̮̾̅͜ ̷̧͎͆i̷̛͒͜͠n̸̥̺͒ ̶͚͚͊̿͜t̸̆...

1   19   19  

proxycrawl-php

ProxyCrawl PHP library for scraping and crawling websites

5   19   19  

path-finder-rl

Method For Establishing Database For Global Value Chain For Parts Proc...

15   19   19  

go-scrapy

Web crawling and scraping framework for Golang

4   18   18  

GlassFrog

Keyword Search & Information Gathering Tool

2   18   18  

crawl-original-google-images

python scripts for crawling original image from Google Images

2   18   18  

billboard-json

🎧 Get json type billboard hot 100 chart

2   18   18  

scrapy-fieldstats

A Scrapy extension to log items coverage when the spider shuts down

4   17   17  

old_ver_bot

파이썬 슬랙 크롤링 봇입니다. It's slack bot made by python+flask+bs4....

7   17   17  

udemy-crawler

Crawling Udemy course info and save into JSON format.

5   17   17  

mida

MIDA: A Tool for Measuring the Internet

4   17   17  

SentimentPoliticalCompass

framework to analyze newspapers with respect to their political convic...

3   17   17  

XML-Parser

A Node.js XML DOM, Parser & Stringifier.

8   17   17  

app-crawler

crawling App by uiautomator2 & mitmproxy

8   17   17  

ragno

Common Lisp Web crawling library based on Psychiq.

2   16   16  

pyReptile

web crawling & scraping framework for Python

7   16   16  

img-cli

An interactive Command-Line Interface Build in NodeJS for downloading...

3   16   16  

deephotel

scraping TripAdvisor, Booking.com with Scrapy

10   16   16  

XingDumper

Python 3 script to dump company employees from XING API

3   16   16  

scrapper

Web scraper with a simple REST API living in Docker and using a Headle...

2   16   16  

crawlly

A simple web crawller in go

12   15   15  

kasthack.osp

Генератор сырых дампов пользователей VK.

5   15   15  

facebook-scraper-for-non-english-user

crawling facebok page

4   15   15  

the-seinfeld-chronicles

A dataset for textual analysis on arguably the best written comedy tel...

1   14   14  

web-search-engine-UIC

CS 582 Information Retrieval at University of Illinois at Chicago. Mul...

4   14   14  

translators

🇻🇨 구글, 파파고, 카카오 번역기 비교 서비스

0   14   14  

DDMKL

한국 현대문학 박사학위 논문 서지 데이터 분석

0   13   13  

product-integrations

Code examples and general information

6   13   13  

pumba

Fetch, store and access user agent strings for different browsers

2   12   12  

PyCarGr

PyCarGr - Unofficial car.gr API

12   12   12  

ProductHunt-scraper

Producthunt.com famous website scraper script. Scrap all offers and sa...

7   12   12  

scraper

All In One API to easily scrape data from any website, without worryin...

3   12   12  

WeiboStockAnalysisCharts

crawling china stock recommendation from Sina Weibo, create pyecharts...

3   11   11