crawler

🕷️ A node crawler for github trending.

As you can see, a kuaishou crawler

Extract web archive data using Wayback Machine and Common Crawl

疫情数据爬虫，2019新型冠状病毒数据仓库，轨迹数据，同乘数据，报道

A Telegram crawler made in Python to automatically search groups and channels and collect any type of data from them.

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library des...

Interactive CLI Web Crawler

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

Projeto de calculo de Imposto de Renda em operacoes na bovespa automaticamente. Tags:canal eletronico do investidor, CEI, selenium, bovespa, IRPF, IR,...

:spider: The pipeline for the OSCAR corpus

轻量、异步、开箱即用的社交媒体聚合解析库

A simple, fast and reliable Coursera crawling & downloading tool

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

This script scrapes the HTML from different web pages to get the information from the video (XVideos, PornHub, RedTube) and you can use it in your own...

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

字体混淆服务

利用文本挖掘技术进行新闻热点关注问题分析

GitHub Star 收藏资源库，自动同步个人收藏项目。收录 2000+ 个精选开源项目，覆盖开发框架、组件、SDK、模板、API、脚本、工具及各类实用资源，帮助开发者快速...

🥄 A package for building specific Proxy Pool for different Sites.

MM131网站图片爬取 :rotating_light:

推特爬虫

🕸 Modular, multithreaded, puppeteer-based crawler

Powerful mutable web directory fuzzer to bruteforce existing and/or hidden files or directories.

无cookie版微博爬虫，可以连续爬取一个或多个新浪微博用户信息、用户微博及其微博评论转发。

中国大陆大学列表爬虫

CeWLeR - Custom Word List generator Redefined. CeWL alternative in Python, based on the Scrapy framework.

(2020年最新)斗鱼弹幕抓取及可视化管理平台第二版，提供弹幕抓取、弹幕实时发送速度可视化、抓取记录查询、弹幕下载、自定义关键词统计、铁粉统计、高光时刻自动...

由Python编写的全异步实现的动漫之家(dmzj)漫画批量下载器（爬虫）

Easy way to brute-force web directory.

A Java componentized distributed crawler framework. 一个Java版本的组件化的分布式通用爬虫

Domain names collector - Crawl websites and collect domain names along with their availability status.

Crawl some picture for fun

一款分布式爬虫平台，帮助你更好的管理和开发爬虫。 内置一套爬虫定义规则（模版），可使用模版快速定义爬虫，也可当作框架手动开发爬虫。(兴趣使然的项目，用的...

🔥 Shadowsocks 账号爬虫

:star2::octocat: powered by python3( simple learning of spider) 百度文库；网易云歌曲； 豆瓣电影； GitHub； 京东； QQ空间； 天气； vip解析助手； TED文...

微博数据采集，后续会加上知乎，贴吧，小红书，抖音，快手等主流媒体内容

Quickly reads webpages and converts to markdown for fast, token efficient web scraping

简单、易用、高效 一个有态度的开源.Net Http请求框架!可以用制作爬虫，api请求等等。

Golang爬虫 爬取豆瓣电影Top250

CrawlAI RAG is an AI-powered website intelligence platform that allows users to crawl entire websites, index their content, and ask natural-language q...

Automatically constructing corpus for automatic speech recognition from YouTube videos

Note: The website is rewritten in https://github.com/Liu233w/ojhunt-lite

Collect XSS vulnerable parameters from entire domain.

A fast, modern and intelligent proxy rotator perfect for crawling and scraping public data.

Have time.ir in shell!

Amazon S3 bucket finder and crawler.

保存百度贴吧帖子到本地，并且支持图片, 视频, 语音等内容。与本项目配套的阅读器 TiebaReader(https://github.com/Sorceresssis/TiebaReader)