Awesome list dedicated to digital and data preservation tools, sources, services and so on.
基于小红书web端的请求封装,JS实现
CobWeb is a Python library for web scraping. The library consists of two classes: Spider and Scraper.
Best manga-viewer on windows for crawling/downloading/browsing exhentai.
A collection of webnovel sources offering varying amounts of scraping capability.
Web/FileSystem Crawler Library
Crawler4U, a general purpose focused crawler
FF14 国服官网自动签到脚本
自動抓取 ptt 表特板圖片
highly customizable news collector
A three-part toolkit for browser session capture, automated crawling, and offline reconstruction of modern web applications.
Free proxy list by country: verified SOCKS5, SOCKS4, HTTP and HTTPS proxies, rechecked every 30 minutes, as TXT, JSON and CSV.
다나와 크롤러 - PC부품 크롤링
File Crawler index files and search hard-coded credentials
✨ Article Crawler is a package used to crawl articles with Markdown format from a specific webpage and store them locally in HTML / Markdown formats.
Build web scraping agents using AI to auto-extract the data from websites, capture screenshot, generate pdf from URL and web crawling with Agenty
Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex sc...
將一個 E-Hentai 畫廊下載並轉換成 PDF,方便在 Kindle 上閱讀 以及在 iPad 上閱讀並作筆記,,,
🔎 scan the internet to find "private" proxies.
Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Cachin...
A short introduction to scraping with Python with given steps and an example scraper script.
Generic altcoin DNS seeder. Compatible with virtually any cryptocurrency cloned from bitcoin. Built-in lightweight DNS server ~ Cloudflare DNS support...
A Tox DHT network crawler
A multithreaded tool for downloading search results of Baidu image search.
对抗cloudflare载入页反爬虫防护(已失效)
A collection of pentesting web scanners
华数杯2024C题数据集收集过程
基于 Xray-core、glider 的代理池工具
a py module to download apk from apkpure.com
Auto crawl RSS feeds using Github Action
一个将runoob.com转换为PDF的爬虫
A collection of short projects, you could try and implement these as short projects or use them as part of a larger project.
Get facebook events from location with Python 3
an unofficial facebook api
[DEPRECATED] Simple, flexible, delightful web crawler/spider package
PoliteWebScanner is a learning-grade web vulnerability scanner plus AI-assisted report viewer. It crawls safely (queue-based, robots-aware, rate-limit...
基于 Selenium 和 Tkinter 的爬取淘宝商品的Web自动化工具
Download all content from Medium and Dev.to to local folder
A multithreaded web crawler using two mechanism - single lock and thread safe data structures
🍋 Python基础、Pygame游戏编程、Python算法与面试题、四种常用的Python Web框架、爬虫、数据可视化、机器学习。一共七个Python大方向!
A Minimal Yet Powerful Crawler for Extracting all The Internal/External/Fuzz-able Links from a website
publiccode.yml crawler for the Open Source software catalog of Developers Italia
Based on Swoole,a PHP DHT crawler, which have insane productivity(依托于swoole的PHP版本的DHT爬虫,有着奇高的效率)
Fetches PubMed article IDs (PMIDs) from email inbox, then crawls PubMed, Google Scholar and Sci-Hub for respective PDF files.
Simple for use node html crawler (spider) of site web pages
:beetle:简单轻便的Java爬虫框架,只要会一点简单的正则表达式和简单的css选择器就能轻松的采集数据。
[Obsolete] imooc web crawler in Node.js(使用 Node.js 编写的慕课网爬虫)
The fast website crawler
마루마루 다운로더 신규 프로젝트
HttpClient + Jsoup + Queue