Topic

crawler

Repositories (1501)

awesome-digital-preservation
awesome-digital-preservation ruarxive

Awesome list dedicated to digital and data preservation tools, sources, services and so on.

39
xhs-js
xhs-js saifeiLee JavaScript

基于小红书web端的请求封装,JS实现

39
CobWeb-lnx
CobWeb-lnx GoncaloMark Python

CobWeb is a Python library for web scraping. The library consists of two classes: Spider and Scraper.

39
ExHentaiReader
ExHentaiReader AndyHsiehTA HTML

Best manga-viewer on windows for crawling/downloading/browsing exhentai.

39
novelsave_sources
novelsave_sources m-haisham Python

A collection of webnovel sources offering varying amounts of scraping capability.

39
fess-crawler
fess-crawler codelibs Java

Web/FileSystem Crawler Library

39
crawler
crawler crawlerclub Go

Crawler4U, a general purpose focused crawler

39
FF14AutoSignIn
FF14AutoSignIn renchangjiu Python

FF14 国服官网自动签到脚本

39
auto_crawler_ptt_beauty_image
auto_crawler_ptt_beauty_image twtrubiks Python

自動抓取 ptt 表特板圖片

39
aristotle
aristotle egcodes Python

highly customizable news collector

39
Browser.Reverse.Engineering.Toolkit
Browser.Reverse.Engineering.Toolkit Dvurechensky JavaScript

A three-part toolkit for browser session capture, automated crawling, and offline reconstruction of modern web applications.

38
free-proxy-health-list
free-proxy-health-list xyzs996 Python

Free proxy list by country: verified SOCKS5, SOCKS4, HTTP and HTTPS proxies, rechecked every 30 minutes, as TXT, JSON and CSV.

38
Danawa-Crawler
Danawa-Crawler sammy310 Python

다나와 크롤러 - PC부품 크롤링

38
filecrawler
filecrawler helviojunior Python

File Crawler index files and search hard-coded credentials

38
article_crawler
article_crawler tychozzz Python

✨ Article Crawler is a package used to crawl articles with Markdown format from a specific webpage and store them locally in HTML / Markdown formats.

38
scrapingai
scrapingai Agenty TypeScript

Build web scraping agents using AI to auto-extract the data from websites, capture screenshot, generate pdf from URL and web crawling with Agenty

38
sneakpeek
sneakpeek flulemon Python

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex sc...

38
EH-PDF
EH-PDF Galgamer-org Python

將一個 E-Hentai 畫廊下載並轉換成 PDF,方便在 Kindle 上閱讀 以及在 iPad 上閱讀並作筆記,,,

38
ProxyScan
ProxyScan Its-Vichy Go

🔎 scan the internet to find "private" proxies.

38
papercut
papercut armand1m TypeScript

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Cachin...

38
Python-scraper-tutorial
Python-scraper-tutorial Decodo Python

A short introduction to scraping with Python with given steps and an example scraper script.

38
generic-seeder
generic-seeder team-exor C++

Generic altcoin DNS seeder. Compatible with virtually any cryptocurrency cloned from bitcoin. Built-in lightweight DNS server ~ Cloudflare DNS support...

38
toxcrawler
toxcrawler JFreegman C

A Tox DHT network crawler

38
BaiduImageCrawler
BaiduImageCrawler flexwang-zz Python

A multithreaded tool for downloading search results of Baidu image search.

38
AntiCloudFlare
AntiCloudFlare s045pd HTML

对抗cloudflare载入页反爬虫防护(已失效)

38
WebRecon
WebRecon flashnuke Python

A collection of pentesting web scanners

37
Data-Collection-Process-for-the-2024-Huashu-Cup-C-Problem
Data-Collection-Process-for-the-2024-Huashu-Cup-C-Problem Diraw Python

华数杯2024C题数据集收集过程

37
xray_pool
xray_pool allanpk716 Go

基于 Xray-core、glider 的代理池工具

37
apkpure_download
apkpure_download batrukanya Python

a py module to download apk from apkpure.com

37
github-action-rss-crawler
github-action-rss-crawler minhhungit HTML

Auto crawl RSS feeds using Github Action

37
crawlhtmltopdf
crawlhtmltopdf osdodo Python

一个将runoob.com转换为PDF的爬虫

37
Mini-Projects
Mini-Projects nazaninsbr Python

A collection of short projects, you could try and implement these as short projects or use them as part of a larger project.

37
python-facebook-bot
python-facebook-bot tudoanh Python

Get facebook events from location with Python 3

37
Facebooker
Facebooker gpwork4u Python

an unofficial facebook api

37
NodeSpider
NodeSpider Bin-Huang TypeScript

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

37
Web_Vulnerability_Scanner-AI
Web_Vulnerability_Scanner-AI SagarBiswas-MultiHAT Python

PoliteWebScanner is a learning-grade web vulnerability scanner plus AI-assisted report viewer. It crawls safely (queue-based, robots-aware, rate-limit...

36
taobao-crawler-selenium
taobao-crawler-selenium YoungZM339 Python

基于 Selenium 和 Tkinter 的爬取淘宝商品的Web自动化工具

36
medup
medup miry Crystal

Download all content from Medium and Dev.to to local folder

36
Web-Crawler
Web-Crawler kshru9 C++

A multithreaded web crawler using two mechanism - single lock and thread safe data structures

36
Python
Python fmw666 Python

🍋 Python基础、Pygame游戏编程、Python算法与面试题、四种常用的Python Web框架、爬虫、数据可视化、机器学习。一共七个Python大方向!

36
Damn-Small-URL-Crawler
Damn-Small-URL-Crawler r3dxpl0it Python

A Minimal Yet Powerful Crawler for Extracting all The Internal/External/Fuzz-able Links from a website

36
publiccode-crawler
publiccode-crawler italia Go

publiccode.yml crawler for the Open Source software catalog of Developers Italia

36
crazyDhtSpider
crazyDhtSpider ixiaofeng PHP

Based on Swoole,a PHP DHT crawler, which have insane productivity(依托于swoole的PHP版本的DHT爬虫,有着奇高的效率)

36
PyperGrabber
PyperGrabber pykong Python

Fetches PubMed article IDs (PMIDs) from email inbox, then crawls PubMed, Google Scholar and Sci-Hub for respective PDF files.

36
node-html-crawler
node-html-crawler safonovpro JavaScript

Simple for use node html crawler (spider) of site web pages

36
vw-crawler
vw-crawler vector4wang Java

:beetle:简单轻便的Java爬虫框架,只要会一点简单的正则表达式和简单的css选择器就能轻松的采集数据。

36
imooc-crawler
imooc-crawler monkeym4ster JavaScript

[Obsolete] imooc web crawler in Node.js(使用 Node.js 编写的慕课网爬虫)

36
gargantua
gargantua andreaskoch Go

The fast website crawler

36
MMDownloader
MMDownloader occidere Java

마루마루 다운로더 신규 프로젝트

36
NetEaseCloudMusicCrawler
NetEaseCloudMusicCrawler timelessmemory Java

HttpClient + Jsoup + Queue

36