Most popular crawler repositories and open source projects

instagram-profilecrawl

📝 quickly crawl the information (e.g. followers, tags etc...) of an i...

239 1040 1040

crawly

Crawly, a high-level web crawling & scraping framework for Elixir.

120 1037 1037

grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dy...

113 1036 1036

sqliv

massive SQL injection vulnerability scanner

382 1029 1029

mzitu

👧 美女写真套图爬虫（二）

346 1018 1018

article-extractor

To extract main article from given URL with Node.js

100 990 990

Bili23-Downloader

跨平台的 B 站视频下载工具，支持 Windows、Linux、macOS 三平台，下载 B...

79 976 976

kimuraframework

Kimurai is a modern web scraping framework written in Ruby which works...

143 971 971

ast-hook-for-js-RE

浏览器内存漫游解决方案（探索中...）

309 967 967

Pxer

A tool for pixiv.net. 人人可用的P站爬虫

111 959 959

crawler

A high performance web crawler / scraper in Elixir.

90 948 948

stormcrawler

A scalable, mature and versatile web crawler based on Apache Storm

266 931 931

BT-btt

磁力網站U3C3介紹以及域名更新

84 929 929

SecCrawler

一个方便安全研究人员获取每日安全日报的爬虫和推送程序，目前爬取范围包括...

141 923 923

TumblThree

A Tumblr Blog Backup Application

129 918 918

chatWeb

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main...

137 910 910

magnet-dht

✌️ Python3 BitTorrent DHT crawler

284 907 907

zhihu-crawler

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展...

382 907 907

scrapyrt

HTTP API for Scrapy spiders

162 864 864

XSRFProbe

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Too...

175 862 862

skrape.it

A Kotlin-based testing/scraping/parsing library providing the ability...

66 852 852

spidr

A versatile Ruby web spidering library that can spider a site, multipl...

108 815 815

till

DataHen Till is a companion tool to your existing web scraper that ins...

22 814 814

Lulu

[Unmaintained] A simple and clean video/music/image downloader 👾

142 810 810

easy-scraping-tutorial

Simple but useful Python web scraping tutorial code.

546 802 802

pic-gather

🛑 image collector, which supports custom acquisition source configura...

212 801 801

sperm

浏览过的精彩逆向文章汇总，值得一看

226 791 791

BaiduImageSpider

一个超级轻量的百度图片爬虫

390 781 781

creeper

:paw_prints: Creeper - The Next Generation Crawler Framework (Go)

57 780 780

fetchbot

A simple and flexible web crawler that follows the robots.txt policies...

99 774 774

BaiduSpider

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图...

186 765 765

icrawler

A multi-thread crawler framework with many builtin image crawlers prov...

163 759 759

xxl-crawler

A lightweight web crawler framework.（Java爬虫框架）

315 740 740

seo-audits-toolkit

SEO & Security Audit for Websites. Lighthouse & Security Headers crawl...

147 737 737

PyPtt

The best PTT library

99 709 709

bookcorpus

Crawl BookCorpus

94 694 694

xeHentai

Doujinshi downloader 绅士漫画下载

84 692 692

TumblThree

A Tumblr and Twitter Blog Backup Application

82 682 682

linkedin-profile-scraper-api

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in J...

171 680 680

course-crawler

🎓 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下...

199 677 677

ArrowDL

ArrowDL (Arrow Downloader) is a download manager for Windows, MacOS an...

38 670 670

FileMasta

A search application to explore, discover and share online files

72 663 663

crawler

K 哥爬虫代码分享，JS 逆向，爬虫进阶。关注公众号：K哥爬虫

238 662 662

Scavenger

Crawler (Bot) searching for credential leaks on paste sites.

122 649 649

gOSINT

OSINT Swiss Army Knife

80 649 649

spider_collection

python爬虫，目前库存：网易云音乐歌曲爬取，B站视频爬取，知乎问答爬取，...

164 647 647

NetDiscovery

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间...

150 643 643

Weibo-Analyst

Social media (Weibo) comments analyzing toolbox in Chinese 微博评论分...

171 640 640

fbcrawl

A Facebook crawler

231 624 624

DouYin

API of DouYin for Humans used to Crawl Popular Videos and Musics

260 621 621