Most popular crawler repositories and open source projects

water-melon

Water Melon is simple melon.com api sdk for php

0 17 17

crawlersuseragents

Python script to check if there is any differences in responses of an...

2 17 17

noscrape

obfuscate text via node to make scraping your content really difficult...

5 17 17

arachnod

High performance crawler for Nodejs

2 17 17

shub_cli

A CLI for dealing with the features of ScrapingHub

0 17 17

webhunger

WebHunger is an extensible, full-scale crawler framework that supports...

4 17 17

TripAdvisor-Crawling-Suite

Fetching hotel data from TripAdvisor.

7 17 17

udemy-crawler

Crawling Udemy course info and save into JSON format.

5 17 17

Hackerrank-Solution-Crawler

🐍 Crawls solutions of hackerrank and stores as local files.

8 17 17

Spider

爬虫相关

4 16 16

CodeSchool-Downloader

💯 Automation Download codeschool all videos, build with python and se...

11 16 16

pyReptile

web crawling & scraping framework for Python

7 16 16

QZone-Crawler

基于Scrapy的QQ空间照片/相册爬虫。A crawler for QZone photos && albums...

3 16 16

vozer

CLI tool to crawl images and URLs from VOZ (https://forums.voz.vn) thr...

6 16 16

hepcrawl

Scrapy project for feeds into INSPIRE-HEP

29 16 16

img-cli

An interactive Command-Line Interface Build in NodeJS for downloading...

3 16 16

Crawler_Demo

This is my Crawler exercises.

7 16 16

penjabarberita

Extract the article list from its raw news HTML

2 16 16

uiautomator2_spider

通过uiautomator2实现的爬虫

13 16 16

olx

Web crawler / web scraper micro-service for downloading advertisements...

1 16 16

Code-Crawler

Automatic tool using for crawling code to find low-hang fruit vulnerab...

9 16 16

sherlock

🔍 Wikidot crawler. Built for the SCP Foundation

3 16 16

Meme-Crawler

【停止维护】【小鸡词典】【B站】【微博】【谷歌图片】梗相关爬虫，并生成...

5 16 16

deephotel

scraping TripAdvisor, Booking.com with Scrapy

10 16 16

playstore_crawler

Python scalable Play Store crawler

5 16 16

EHentai-Crawler

Clone a panda yourself.

1 16 16

yande-crawler

yande.re 爬虫免代理支持R18

2 16 16

spatie-crawler-toolkit-for-laravel

A toolkit for Spatie's Crawler and Laravel.

6 16 16

PaperWebCrawler

IEEE XPLORE等文献网站的爬虫工具/Crawler for Paper Website like IEEE XP...

4 16 16

insecrawl

Automated downloading of still images from Insecam.org

0 16 16

reddit-graph-releases

Releases for the reddit-graph project

2 16 16

scrapper

Web scraper with a simple REST API living in Docker and using a Headle...

2 16 16

scrapy_django_tianya

Scrapy, tianya, 天涯; scrapy django增量抓取天涯莲蓬鬼话全部帖子

6 16 16

scrapy-zyte-api

Zyte API integration for Scrapy

12 16 16

premium-downloader

7 16 16

crawler_CIA_CREST

R-crawler for CIA website (CREST)

5 15 15

spider_for_Books

图书爬虫，已囊括当当、京东……目前字典内容包括了书名、作者、出版社、出版...

6 15 15

aCrawler

🔍 A powerful web-crawling framework, based on aiohttp.

3 15 15

QuestionClustering

Clasificador de preguntas escrito en python 3 que fue implementado en...

2 15 15

Wheres-My-Offer

University Admission Portal Checker

5 15 15

crawlero

Distributed web crawlers. Fault tolerance, user-agent randomizer, Rabb...

1 15 15

BangumiSpider

对Bangumi.tv进行爬虫

5 15 15

chromebot

Run headless Chrome using Go.

2 15 15

DotnetThirdPartyNotices

A .NET tool to generate file with third party legal notices

1 15 15

NetCrawlerDetect

A .net standard port of JayBizzle's CrawlerDetect project (https://git...

1 15 15

Crawler_Web_Js

Dùng scrapy-splash kết hợp lua script để crawl các trang web sử dụng J...

13 15 15

Studybyte

Studybyte is a search engine designed to help students find educationa...

2 15 15

rforseo

Guide to use R for SEO

2 15 15

crawler-google-scholar

This bot crawls and downloads statistics and pictures from google scho...

1 15 15

fedimapper

An API for the Fediverse - The Software behind the Fediverse Almanac

1 15 15