Most popular crawler repositories and open source projects

crawler-set

各种网站爬虫合集，持续更新中....

16 17 17

spider_for_Books

图书爬虫，已囊括当当、京东……目前字典内容包括了书名、作者、出版社、出版...

6 17 17

photo-spider-scrapy

10 photo website spiders, 10 个国外图库的 scrapy 爬虫代码

10 17 17

Douban-Crawler

抓取豆瓣小组相关信息（小组、用户、帖子）。

8 17 17

images-downloader

A Node.js module for downloading a single image or multiple images to...

10 17 17

news-sentiment-analysis

The spider crawls moneycontrol.com and economictimes.com to fetch news...

5 17 17

deephotel

scraping TripAdvisor, Booking.com with Scrapy

10 17 17

Deep_miner

Webcrawler written in Python. This crawler does dig in till the 3 leve...

5 17 17

SearchGar

SearchGar - An actual Search Engine made using Python

2 17 17

konect-extr

Network dataset extraction library – part of the KONECT project by Jér...

10 17 17

AnimeDl

⚡️An API for downloading or streaming your favorite anime.

1 17 17

water-melon

Water Melon is simple melon.com api sdk for php

0 17 17

crawlersuseragents

Python script to check if there is any differences in responses of an...

2 17 17

hepcrawl

Scrapy project for feeds into INSPIRE-HEP

29 16 16

Crawler_Demo

This is my Crawler exercises.

7 16 16

penjabarberita

Extract the article list from its raw news HTML

2 16 16

uiautomator2_spider

通过uiautomator2实现的爬虫

13 16 16

locust

Distributed web data discovery and collection framework built for serv...

1 16 16

olx

Web crawler / web scraper micro-service for downloading advertisements...

1 16 16

Code-Crawler

Automatic tool using for crawling code to find low-hang fruit vulnerab...

9 16 16

sherlock

🔍 Wikidot crawler. Built for the SCP Foundation

3 16 16

Meme-Crawler

【停止维护】【小鸡词典】【B站】【微博】【谷歌图片】梗相关爬虫，并生成...

5 16 16

playstore_crawler

Python scalable Play Store crawler

5 16 16

EHentai-Crawler

Clone a panda yourself.

1 16 16

yande-crawler

yande.re 爬虫免代理支持R18

2 16 16

spatie-crawler-toolkit-for-laravel

A toolkit for Spatie's Crawler and Laravel.

6 16 16

PaperWebCrawler

IEEE XPLORE等文献网站的爬虫工具/Crawler for Paper Website like IEEE XP...

4 16 16

insecrawl

Automated downloading of still images from Insecam.org

0 16 16

scrapy_django_tianya

Scrapy, tianya, 天涯; scrapy django增量抓取天涯莲蓬鬼话全部帖子

6 16 16

premium-downloader

7 16 16

go-scrapy

Web crawling and scraping framework for Golang

2 16 16

Spider

爬虫相关

4 16 16

CodeSchool-Downloader

💯 Automation Download codeschool all videos, build with python and se...

11 16 16

pyReptile

web crawling & scraping framework for Python

7 16 16

QZone-Crawler

基于Scrapy的QQ空间照片/相册爬虫。A crawler for QZone photos && albums...

3 16 16

vozer

CLI tool to crawl images and URLs from VOZ (https://forums.voz.vn) thr...

6 16 16

auto-internet-letter

군바리 친구들을 위한 자동으로 편지 보내기

0 15 15

httpmapper

👁️ Tool for Data Extraction and Web Requests.

4 15 15

crawler_click_tutorial

click tutorial ( crawler ) use python

4 15 15

Crawler_pubg.op.gg

This is a web crawler for pubg.op.gg, written by Ruichong Liu. 绝地求...

3 15 15

kasthack.osp

Генератор сырых дампов пользователей VK.

5 15 15

recipe-crawler

🕷 🍖 爬取"下厨房"菜谱数据 https://www.xiachufang.com/

8 15 15

sentry

Parallelized web crawler written in Golang

6 15 15

crawler_CIA_CREST

R-crawler for CIA website (CREST)

5 15 15

img-cli

An interactive Command-Line Interface Build in NodeJS for downloading...

3 15 15

aCrawler

🔍 A powerful web-crawling framework, based on aiohttp.

3 15 15

QuestionClustering

Clasificador de preguntas escrito en python 3 que fue implementado en...

2 15 15

Wheres-My-Offer

University Admission Portal Checker

5 15 15

crawlero

Distributed web crawlers. Fault tolerance, user-agent randomizer, Rabb...

1 15 15

BangumiSpider

对Bangumi.tv进行爬虫

5 15 15