crawler

大麦抢票脚本案例

An automated website accessibility scanner and cli

Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.

:computer: Quickly crawl the information (e.g. followers, tags, etc...) of an instagram profile. No login required!

Enlarge training dataset by searching images with specified keywords in google and download the presented images

Download Videos Skill Share per ID or per Class

Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with...

java framework for prerender

Python client library for Diffbot APIs

GraphQuery is a query language and execution engine tied to any backend service.

🎓a Better BlackBoard for PKUers. 北京大学教学网命令行工具（🖥️Win/🐧Linux/🍏Mac）

某东商品价格监控：自定义商品价格，降价邮件/微信提醒。技术：Python爬虫/IP代理池/JS接口爬取/Selenium页面爬取

百度贴吧吧务管理器✨删帖机✨使用aiohttp封装大量贴吧核心API

（已跑路）百度贴吧吧务管理工具，自动扫描帖子并处理违规帖

A dungeon crawler

🌱 goClone - clone websites in seconds

Amazon商品引流的 python 爬虫

Findpapers: A tool for helping researchers who are looking for related works

Public Opinion Mining System of Taiwanese Forums

一只优雅的正方教务系统爬虫。

Sample of using proxies to crawl baidu search results.

下载搜狗、百度、QQ输入法的词库文件的 python 爬虫，可用于构建不同行业的词汇库

爬虫代理IP池服务，可供其他爬虫程序通过restapi获取

LFITester is a Python3 program that automates the detection and exploitation of Local File Inclusion (LFI) vulnerabilities on a server.

✴️ An experimental graph database

Processes XML sitemaps and extracts URLs. Includes features such as support for both plain XML and compressed XML files, multiple input sources, prote...

Cross-platform persistent and distributed web crawler :link:

AI-powered web scraping CLI. Describe what you want, get a production-ready Scrapy spider. Write once, reuse forever.

Powerful dork searcher and vulnerability scanner for windows platform

wxpath - declarative web crawling with XPath; a Web Query Language (WQL)

针对某亿些小说网站的爬虫

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

Collect, Download, Organize and Share your Favorite Anime Artworks.

Golang pkg to quickly return a preview of a webpage (title/description/images)

A command-line utility designed to recursively spider webpages for URLs. It works by actively traversing websites - following links embedded in webpag...

Project thu thập điểm chuẩn đại học 2014 - 2018 và phân tích dữ liệu

Proxy List Scrapper

qcrawl - fast async web crawling & scraping framework for Python.

Scrapy + Puppeteer

gocrawler, go分布式爬虫框架

Spider ported to Python

A complimentary proxy to help to use SPM with headless browsers

爬虫, http代理, 模拟登陆!

This package is a complete tool for creating a large dataset of images (specially designed -but not only- for machine learning enthusiasts). It can cr...

Simple Weibo Scraper

✨ BOSE IS SWISS ARMY KNIFE 🔪 FOR BOT DEVELOPMENT. THE ULTIMATE BOT DEVELOPMENT FRAMEWORK. 🤖

Get Aliexpress product details in JSON

A command-line tool to crawl websites using puppeteer.