Most popular crawler repositories and open source projects

app-crawler

crawling App by uiautomator2 & mitmproxy

8   17   17  

papercut

Papercut is a scraping/crawling library for Node.js built on top of JS...

1   17   17  

Deep_miner

Webcrawler written in Python. This crawler does dig in till the 3 leve...

5   17   17  

SearchGar

SearchGar - An actual Search Engine made using Python

2   17   17  

konect-extr

Network dataset extraction library – part of the KONECT project by Jér...

10   17   17  

AnimeDl

⚡️An API for downloading or streaming your favorite anime.

1   17   17  

water-melon

Water Melon is simple melon.com api sdk for php

0   17   17  

crawlersuseragents

Python script to check if there is any differences in responses of an...

2   17   17  

noscrape

obfuscate text via node to make scraping your content really difficult...

5   17   17  

Spider

爬虫相关

4   16   16  

CodeSchool-Downloader

💯 Automation Download codeschool all videos, build with python and sel...

11   16   16  

pyReptile

web crawling & scraping framework for Python

7   16   16  

QZone-Crawler

基于Scrapy的QQ空间照片/相册爬虫。A crawler for QZone photos && albums...

3   16   16  

vozer

CLI tool to crawl images and URLs from VOZ (https://forums.voz.vn) thr...

6   16   16  

hepcrawl

Scrapy project for feeds into INSPIRE-HEP

29   16   16  

img-cli

An interactive Command-Line Interface Build in NodeJS for downloading...

3   16   16  

Crawler_Demo

This is my Crawler exercises.

7   16   16  

penjabarberita

Extract the article list from its raw news HTML

2   16   16  

uiautomator2_spider

通过uiautomator2实现的爬虫

13   16   16  

olx

Web crawler / web scraper micro-service for downloading advertisements...

1   16   16  

Code-Crawler

Automatic tool using for crawling code to find low-hang fruit vulnerab...

9   16   16  

sherlock

🔍 Wikidot crawler. Built for the SCP Foundation

3   16   16  

Meme-Crawler

【停止维护】【小鸡词典】【B站】【微博】【谷歌图片】梗相关爬虫，并生成...

5   16   16  

deephotel

scraping TripAdvisor, Booking.com with Scrapy

10   16   16  

playstore_crawler

Python scalable Play Store crawler

5   16   16  

EHentai-Crawler

Clone a panda yourself.

1   16   16  

yande-crawler

yande.re 爬虫 免代理 支持R18

2   16   16  

spatie-crawler-toolkit-for-laravel

A toolkit for Spatie's Crawler and Laravel.

6   16   16  

PaperWebCrawler

IEEE XPLORE等文献网站的爬虫工具/Crawler for Paper Website like IEEE XP...

4   16   16  

insecrawl

Automated downloading of still images from Insecam.org

0   16   16  

reddit-graph-releases

Releases for the reddit-graph project

2   16   16  

scrapper

Web scraper with a simple REST API living in Docker and using a Headle...

2   16   16  

scrapy_django_tianya

Scrapy, tianya, 天涯; scrapy django增量抓取天涯莲蓬鬼话全部帖子

6   16   16  

scrapy-zyte-api

Zyte API integration for Scrapy

12   16   16  

premium-downloader

7   16   16  

auto-internet-letter

군바리 친구들을 위한 자동으로 편지 보내기

0   15   15  

httpmapper

👁️ Tool for Data Extraction and Web Requests.

4   15   15  

crawler_click_tutorial

click tutorial ( crawler ) use python

4   15   15  

Crawler_pubg.op.gg

This is a web crawler for pubg.op.gg, written by Ruichong Liu. 绝地求...

3   15   15  

kasthack.osp

Генератор сырых дампов пользователей VK.

5   15   15  

recipe-crawler

🕷 🍖 爬取"下厨房"菜谱数据 https://www.xiachufang.com/

8   15   15  

sentry

Parallelized web crawler written in Golang

6   15   15  

crawler_CIA_CREST

R-crawler for CIA website (CREST)

5   15   15  

spider_for_Books

图书爬虫，已囊括当当、京东……目前字典内容包括了书名、作者、出版社、出版...

6   15   15  

aCrawler

🔍 A powerful web-crawling framework, based on aiohttp.

3   15   15  

QuestionClustering

Clasificador de preguntas escrito en python 3 que fue implementado en...

2   15   15  

Wheres-My-Offer

University Admission Portal Checker

5   15   15  

crawlero

Distributed web crawlers. Fault tolerance, user-agent randomizer, Rabb...

1   15   15  

BangumiSpider

对Bangumi.tv进行爬虫

5   15   15  

chromebot

Run headless Chrome using Go.

2   15   15