Most popular crawler repositories and open source projects

FooProxy

稳健高效的评分制-针对性- IP代理池 + API服务，可以自己插入采集器进行代...

59   222   222  

WebVideoBot

Web crawler.

44   221   221  

ZhihuSpider

多线程知乎用户爬虫，基于python3

81   221   221  

91porn-crawler

91 porn crawler. 自动爬取并下载你想要的91porn热门视频。Automatically d...

39   220   220  

Sitemap-Generator-Crawler

PHP script to recursively crawl websites and generate a sitemap. Zero...

89   219   219  

google-group-crawler

[Deprecated] Get (almost) original messages from google group archives...

37   214   214  

go-movies

golang spider Crawler 爬虫 电影

66   214   214  

gf-secrets

Secret and/or credential patterns used for gf.

47   213   213  

InfinityCrawler

A simple but powerful web crawler library for .NET

33   213   213  

nudecrawler

Crawl telegra.ph searching for nudes!

18   213   213  

goribot

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang...

30   211   211  

ptt-alertor

:loudspeaker: Ptt 文章通知機器人！Notify Ptt Article in Realtime

24   210   210  

indonesian-NLP-resources

data resource untuk NLP bahasa indonesia

53   209   209  

AllNewsSpider

澎湃新闻，新浪新闻，腾讯新闻，搜狐新闻，新闻联播，泰晤士报，纽约时报，...

45   209   209  

scrapedin-linkedin-crawler

Crawler for LinkedIn full profiles 2019

70   207   207  

news-crawl

News crawling with StormCrawler - stores content as WARC

26   207   207  

crawlab-lite

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

69   206   206  

laosj

golang light-weight image crawler

39   205   205  

N2H4

네이버 뉴스 수집을 위한 도구

74   205   205  

web-page-monitor

Web Site Page Changes Monitor. 网站网页页面更新变更监控提醒。

35   204   204  

KoreaNewsCrawler

대량의 뉴스 데이터를 수집하기 위해 만들어진 뉴스 크롤러입니다.

102   202   202  

VideoServer

以Node.js基于express以及爬虫实现的视频资源后端

6   201   201  

dorkscout

DorkScout - Golang tool to automate google dork scan against the entie...

18   201   201  

weibo_wordcloud

根据关键词抓取微博数据，再生成词云

68   197   197  

galer

A fast tool to fetch URLs from HTML attributes by crawl-in.

33   197   197  

robots-txt

Determine if a page may be crawled from robots.txt, robots meta tags a...

29   196   196  

NoSmoke

A cross platform UI crawler which scans view trees then generate and e...

61   195   195  

Pinkerton

🕵️ Pinkerton is an JavaScript file crawler and secret finder developed...

30   194   194  

black-widow

GUI based offensive penetration testing tool (Open Source)

42   191   191  

JavPy

Enjoy driving on a Javascriptive (originally Pythonic) way to Japanese...

35   190   190  

crawler-js-hook-framework-public

JS逆向Hook工具集，开源部分工具到这里

82   190   190  

crawler_shopee_public

蝦皮非同步爬蟲 + 競品賣家分析

64   189   189  

instagram-crawler

Crawl instagram photos, posts and videos for download.

15   188   188  

th-music-video-generator

Touhou Project random music video generator/player, crawling image and...

44   188   188  

web-bee

🐝 Web vertical crawler framework for fun

35   187   187  

digger

Digger is a powerful and flexible web crawler implemented by pure gola...

72   187   187  

CSharpCrawler

C#爬虫示例程序，想学习爬虫入门知识的可以看过来。后续会慢慢加入更多爬虫...

49   186   186  

zhihu_fun

基于 Selenium 的知乎关键词爬虫

39   185   185  

leetcode-spider

用 node.js 爬你自己的 leetcode 解题源码

50   185   185  

zhihu-crawler-people

A simple distributed crawler for zhihu && data analysis

91   184   184  

sensitivefilescan

75   183   183  

Crawler-for-Github-Trending

🕷️ A node crawler for github trending.

19   180   180  

xvideos

xvideos API library

61   180   180  

packagist-mirror

📦✂️📋📦 Create a mirror of packagist.org metadata for use locally with c...

65   180   180  

datmusic-api

51   178   178  

nCov2019_data_crawler

疫情数据爬虫，2019新型冠状病毒数据仓库，轨迹数据，同乘数据，报道

32   177   177  

spoon

🥄 A package for building specific Proxy Pool for different Sites.

23   176   176  

tiktok-downloader

Tiktok Downloader/Scraper using requests & bs4

71   176   176  

kuaishou-crawler

As you can see, a kuaishou crawler

65   172   172  

search

An Open Source Search Engine

105   172   172