crawler
Repositories (1232)
Bitextor generates translation memories from multilingual websites
This is a Multi-thread crawler for Tumblr.
Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探测工具
dynamic crawler for web vulnerability scanner
节点爬取,筛选, 支持Clash,base64订阅解析,自动生成可用的ss, ssr, v2ray, trojan节点. 已集成Github Action,每天8-24,定时更新.
多线程知乎用户爬虫，基于python3
OWASP D4N155 - Intelligent and dynamic wordlist using OSINT
Simple news aggregator displaying top stories in real time
Web Site Page Changes Monitor. 网站网页页面更新变更监控提醒。
Python3 script to continuously download all images/webms of multiple 4chan thread simultaneously - without installation
一款开源的安全评估工具支持常见的 web 安全问题扫描和自定义 POC。此外，该工具还具备机器学习的漏洞检测和自动化测试功能。
crawler framework, distributed crawler extractor
Universal scraping tool, which allows you to extract data using multiple environments
JS逆向研究
Search emails from a domain through search engines
微博超级话题爬虫，微博词频统计+情感分析+简单分类，新增肺炎超话爬取数据
稳健高效的评分制-针对性- IP代理池 + API服务，可以自己插入采集器进行代理IP的爬取，针对你的爬虫的一个或多个目标网站分别生成有效的IP代理数据库，支持Mongo...
Web crawler.
GUI based offensive penetration testing tool (Open Source)
91 porn crawler. 自动爬取并下载你想要的91porn热门视频。Automatically download your "favorite" 91porn hot movies.
[Deprecated] Get (almost) original messages from google group archives. Your data is yours.
PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.
네이버 뉴스 수집을 위한 도구
Secret and/or credential patterns used for gf.
A simple but powerful web crawler library for .NET
[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。
:loudspeaker: Ptt 文章通知機器人！Notify Ptt Article in Realtime
data resource untuk NLP bahasa indonesia
澎湃新闻，新浪新闻，腾讯新闻，搜狐新闻，新闻联播，泰晤士报，纽约时报，BBCNews，旨在爬取所有新闻门户网站的新闻，禁止将所得数据商用！
Crawler for LinkedIn full profiles 2019
News crawling with StormCrawler - stores content as WARC
golang light-weight image crawler
Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Experience for effectively fetching Facebook data by Querying Graph API with Account-based Token and Operating undetectable scraping Bots to extract C...
대량의 뉴스 데이터를 수집하기 위해 만들어진 뉴스 크롤러입니다.
DorkScout - Golang tool to automate google dork scan against the entiere internet or specific targets
以Node.js基于express以及爬虫实现的视频资源后端
根据关键词抓取微博数据，再生成词云
A fast tool to fetch URLs from HTML attributes by crawl-in.
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
A cross platform UI crawler which scans view trees then generate and execute UI test cases.
🐝 Web vertical crawler framework for fun
Enjoy driving on a Javascriptive (originally Pythonic) way to Japanese AV!
JS逆向Hook工具集，开源部分工具到这里
自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向
蝦皮非同步爬蟲 + 競品賣家分析
Crawl instagram photos, posts and videos for download.
Digger is a powerful and flexible web crawler implemented by pure golang
基于 Selenium 的知乎关键词爬虫
C#爬虫示例程序，想学习爬虫入门知识的可以看过来。后续会慢慢加入更多爬虫相关的知识。