Most popular crawler repositories and open source projects

local-api-client-csharp

This .NET Standard package provides convenient access to the Local API...

2   31   31  

squirm

This was the night of the crawling terror!

2   31   31  

PTTmineR

Parallel Searching and Crawling Data from PTT 🚀

4   31   31  

Search_Ads_Web_Service

Online search advertisement platform & Realtime Campaign Monitoring [M...

1   31   31  

pysaint

[deprecated] 유세인트 파이썬 클라이언트

3   31   31  

Spydan

A web spider for shodan.io without using the Developer API.

8   31   31  

spiderable-middleware

🤖 Prerendering for JavaScript powered websites. Great solution for PW...

4   31   31  

see

Search Engine in Erlang

3   31   31  

instagram_scraper

Extract instagram users informations from hashtags. This scraper can e...

14   31   31  

bet365API

The latest way to get bet365 data odds, with a delay of 0.2 seconds be...

8   31   31  

octopus

Recursive and multi-threaded broken link checker

11   31   31  

images-grabber

🖼️ Get all images from pixiv/twitter/deviantart

3   31   31  

Mechanize.NET

Stateful programmatic web browsing, based on Python-Mechanize, which i...

9   30   30  

invana-bot

A Web Crawler that scrapes using YAML and python code.

9   30   30  

TTBot2.0

app版本今日头条 用户登录/个人主页/关注列表/粉丝列表/评论点赞收藏 关键...

16   30   30  

utsusemi

A tool to generate a static website by crawling the original site.

2   30   30  

SINA_Spider

新浪微博爬虫：登录、关键词微博查询、微博监控

20   30   30  

TaobaoAnalysis

练习NLP，分析淘宝评论的项目

7   30   30  

goGamer

巴哈姆特自訂API

2   30   30  

instagram-downloader

Node.js/Express app to retrive instagram video/image download urls

13   30   30  

igxe-c5-buff-csgo-skins-sale-data-catch

Automatically get the csgo skins sale data on igxe.cn and buff and c5g...

2   30   30  

iranian-calendar-events

Fetch Iranian calendar events (Jalali, Hijri and Gregorian) from time....

2   29   29  

integrada.minhabiblioteca.com.br

Download de livros para PDF/EPUB - Integrada.minhabiblioteca / vitalso...

10   29   29  

scrapit

Scraping scripts for various websites.

1   29   29  

DouyuBarrage

(2020年最新)斗鱼弹幕抓取及实时弹幕数据可视化，分为crawler(弹幕抓取)，s...

12   29   29  

tse-client

A client for fetching stock data from the Tehran Stock Exchange (TSETM...

10   29   29  

CrowLeer

Powerful C++ web crawler based on libcurl

4   29   29  

fmovies-crawler

A web scraper that allows you to fetch Movies and TV series informatio...

3   29   29  

reactor-crw

Simple content crawler for joyreactor.cc

2   29   29  

Bayesian-Stock-Market-Sentiment

A stock market text sentiment analysis website. A股舆情分析, web-crawl...

10   29   29  

GooglePlayWebServiceAPI

Tiny script to crawl information of a specific application in the Goog...

10   28   28  

apkpure_download

a py module to download apk from apkpure.com

13   28   28  

cache-warmup

🔥 Composer package to warm up caches of URLs located in XML sitemaps

2   28   28  

crawler

爬取tumblr关注博主图片

7   28   28  

telegram_bbbot

Telegram Bug Bounty Bot

5   28   28  

python-crawl

Library to crawl and extract internal links from domain

8   28   28  

vietnam-ecommerce-crawler

Crawling the data from lazada, websosanh, compare.vn, cdiscount and cu...

9   28   28  

Moodle-Downloader

A Moodle Crawler that downloads course content from Moodle (eg. lectur...

5   28   28  

google-image-downloader

A script to download images from images.google.com

17   28   28  

bot_do_bandejao

🤖🍴 A Python script that scrapes UFMG's restaurants menus and publish...

0   28   28  

scrapy-picture-spider

The project is a spider that uses scrapy and beautifulsoup4 for crawl...

7   28   28  

frisbee

Collect email addresses by crawling search engine results.

7   28   28  

fess-crawler

Web/FileSystem Crawler Library

18   28   28  

BitInsight

:earth_africa: Bittorrent Network Overview through Infohash Indexing,...

6   28   28  

AzureSearchCrawler

A simple web crawler, using Abot, that indexes page contents into Azur...

11   28   28  

advanced-php-crawler

新浪博客文章/wenku8轻小说文库爬虫，可抓取图片保存，一键制作电子书。kin...

11   28   28  

job-funnel-ts

Automated tool for scraping job postings into a .xlsx files inspired b...

0   27   27  

ytpriv

YT metadata exporter

7   27   27  

ds-video-helper

群晖Video Station助手，自动获取豆瓣电影信息，并填写Video Station视频信...

4   27   27  

scrapy_pro

关于5000+站点的scrapy爬虫开发，涉及一些技术架构搭建以及各种反爬方案，...

13   27   27