Most popular crawler repositories and open source projects

ACM_difficult_words_list

为中国ACM选手提供的单词表!-This is difficult words list for chinese ac...

5   13   13  

proxy_manager

Ruby proxy manager. Gem for easy usage proxy in parser/web bots.

0   13   13  

news_crawler

News crawler là một công cụ giúp bạn có thể crawl dữ liệu của một tran...

5   13   13  

crawlItem

用于爬取淘宝天猫网页的谷歌插件

8   13   13  

lastfm

🔌 Last.fm webservice client for php.

6   13   13  

Crawler

This is a crawler with a tool of Jsoup. Furthermore. Moreover, there i...

4   13   13  

python-flightradar

Python airline/flights data crawler

2   13   13  

goose-starter-kit

This is a starter kit for redco/goose-parser

1   12   12  

scrapy_tecnoblog

Projeto Scrapy para coleta de notícias em https://tecnoblog.net/ - Web...

9   12   12  

Marsvin

Structural Crawler framework written in PHP

5   12   12  

JudgeGirl-Scoreboard

A Fancy Scoreboard for JudgeGirl

2   12   12  

khs-screens

Evidence denních dat o COVID-19 z krajských hygienických stanic. Autom...

0   12   12  

fastcrawler

一个快速，简单，基于多线程的网络爬虫框架

9   12   12  

headless-crawler

A crawler implemented using a headless browser (Chrome).

0   12   12  

shelob

Archive of shelob. Replaced by https://github.com/mlcdf/sc-backup

3   12   12  

renren-dumps

人人网数据备份器

3   12   12  

crawler

Web crawler based on Puppeteer

1   12   12  

crawlerUtils

Utils for programming web crawler

3   12   12  

warmcache

A simple tool to scan your website to keep your cache hot & ready. Hel...

0   12   12  

Crawler

A PHP flexible web crawler that can login into a website.

10   12   12  

APKCrawler

Android APK Crawler

4   12   12  

EveryClass-collector

Spider part of EveryClass

3   12   12  

SpiderX

php多线程，可定制爬虫框架

2   12   12  

crawl-reuters

A simple Scrapy script for crawling Reuters news articles (Python 3)

5   12   12  

GithubCrawler

Crawl github data using API and no-API

2   12   12  

selenium-image-crawler

Selenium Image Crawler

8   12   12  

venom-tutorial

A tutorial based on your preferred open source focused crawler for the...

0   12   12  

limit-up-stock-crawler

📈 沪深股市涨停板数据爬虫

5   12   12  

express-middleware-seo

Webpage pre-rendering middleware, base on headless chrome⚡️

2   12   12  

dead-link-crawler

An efficient, asynchronous crawler that identifies broken links on a g...

1   12   12  

web-master

Web mastering tools for my personal services

1   12   12  

predator

High-performance crawler framework based on fasthttp.

1   12   12  

worker

Containerized Ferret worker

7   12   12  

WebCrawler

工作中用到的一些python爬虫，结合业务场景说明使用，主要爬取豌豆荚、应用...

4   12   12  

flatcrawl-processors

A set of processors that will instantly inform users via a set of chan...

1   11   11  

In-One-File-Manager

Desktop File Manager for Windows

2   11   11  

instacrawler

KMU CS Capstone Design project: Instagram Meta Search Engine

0   11   11  

scrapio

Simple and easy-to-use scraper and crawler in Go.

1   11   11  

url2vec

Graph clustering and Node embeddings with word2vec

6   11   11  

full-proxy

小全代理是一个优秀的HTTP(S)隧道代理产品，基于分享原则，永久免费，优化...

0   11   11  

imhodump

Экспорт оценок из imhonet.ru

10   11   11  

axe-seeder

⚓️ crawler for the AXE network

55   11   11  

ccrawl

Simple CORPORA list crawler

1   11   11  

DiSec

Distributed Image Search Engine Crawler

1   11   11  

robotparser-scala

robotparser-scala implements a parser for the robots.txt file format i...

2   11   11  

SofPythonBot

This Telegram-Bot answers python questions by using stackoverflow subj...

6   11   11  

triplesKB

根据军事科技领域词词表采集人员,机构数据集,并构建人员-关键词,机构-关键...

6   11   11  

crawl_freess

用于爬取ssr地址，仅供学习🤖

3   11   11  

andromeda

Global graph of Deno modules and their interdepencies

0   11   11  

StateMapper

Worldwide, collaborative, public data reviewing and monitoring tool. R...

3   11   11