Most popular crawler repositories and open source projects

DotnetThirdPartyNotices

A .NET tool to generate file with third party legal notices

1   15   15  

NetCrawlerDetect

A .net standard port of JayBizzle's CrawlerDetect project (https://git...

1   15   15  

Crawler_Web_Js

Dùng scrapy-splash kết hợp lua script để crawl các trang web sử dụng J...

13   15   15  

Studybyte

Studybyte is a search engine designed to help students find educationa...

2   15   15  

rforseo

Guide to use R for SEO

2   15   15  

crawler-google-scholar

This bot crawls and downloads statistics and pictures from google scho...

1   15   15  

fedimapper

An API for the Fediverse - The Software behind the Fediverse Almanac

1   15   15  

venom

Tool designed for fast crawl and extract endpoints

4   15   15  

crawler-client

crawler dev tools using electron webview

3   14   14  

small-spider-project

日常爬虫

6   14   14  

spider-picture

Node 批量抓取并下载某站点的图片

2   14   14  

roph-rewards

Scripts for claiming free items from Ragnarok Online Philippines websi...

1   14   14  

web_crawler

爬蟲練習(youtube,dcard,kkbox,發票,ptt) 🕷️

3   14   14  

rovers

Rovers is a service to retrieve repository URLs from multiple reposito...

13   14   14  

scrapy-bhinneka-crawler

Scraping bhinneka.com, just for fun

9   14   14  

alipay_crawler

支付宝爬虫,alipay crawler

4   14   14  

getSeoSitemap

PHP library to get the sitemap. It crawls a whole website checking all...

5   14   14  

octopus_spider

基于Scala Akka的分布式主题网络爬虫

2   14   14  

supermonkey

A crawler for automated Android UI testing.

23   14   14  

ZhihuAnalyse

知乎用户爬虫数据分析

6   14   14  

weibo_search

【工具】基于selenium的微博搜索爬虫

7   14   14  

Spider

:dizzy: Spider is a PHP library with easily module integration for cra...

1   14   14  

locust

Distributed web data discovery and collection framework built for serv...

1   14   14  

-Competitive-Coding-Problem-Classifier-and-Recommender

Competitive Coding Problem Classifier and Problem Recommendation

6   14   14  

eynyCrawlerMega

eyny 電影 Mega and Google 連結爬蟲 use python

7   14   14  

framler

[DEPRECATED] AutoCrawler - automate extracting main information from w...

3   14   14  

proxycrawl-ruby

ProxyCrawl API ruby gem for scraping and crawling

1   14   14  

nutch-in-java

How to use Apache Nutch without command line

4   14   14  

BiQuKan

基于python2.7的笔趣看小说网站爬取（http://www.biqukan.com/）

8   14   14  

wallpaperCrawler

自动从网络中爬取壁纸，并发送至你的邮箱。

2   14   14  

Twitter-Friend-Connections

Visualizing Twitter Friend Connections

1   14   14  

Taiwan-Stock-Knowledge-Graph

A knowledge graph about Taiwan stock

3   14   14  

instagram-crawler

Short Ruby scripts to download images and videos from Instagram by cra...

0   13   13  

doffy

a web auto run lib base on chrome headless

0   13   13  

robots.txt

:robot: robots.txt as a service. Crawls robots.txt files, downloads an...

1   13   13  

AioCrawler

Async crawler framework based on aiohttp and asyncio for running fast.

3   13   13  

pyparazzi

Pyparazzi is an scanner that searches websites for links.

0   13   13  

QQZoneParse

模拟登陆QQ空间，获取好友信息，并做分析（年龄分布、性别分布、地址分布等...

3   13   13  

chatper15_net_io_img_crawler

第15章 Kotlin 文件IO操作与多线程

3   13   13  

HorizonSpider

The spider for ZeroNet search engine Horizon

0   13   13  

axegrinder

Crawl websites for accessibility issues from the command line.

6   13   13  

tumblrcrawl

Simple tumblr crawler to download images and videos

7   13   13  

crawler

一个php爬虫

4   13   13  

BeFree

大概就是爬取YouTube之类一些墙外的一些热门内容到一些大陆能访问的网站

4   13   13  

scraper

Scraper

7   13   13  

InstagramLocationScraper

3   13   13  

80s_spider

www.80s.tw 爬虫，用 pyspider，只爬电影、电视剧、动漫、综艺，爬取后存储...

3   13   13  

node-fetch-dom

Magic utility that extract javascript global variables from a remote h...

0   13   13  

GithubCrawler

分布式Github爬虫

7   13   13  

sephora_goods_alarm

监控丝芙兰是否补货的爬虫脚本

6   13   13