crawler

Python的基础练习代码与各种爬虫代码

Social media (Weibo) comments analyzing toolbox in Chinese 微博评论分析工具, 实现功能: 1.微博评论数据爬取; 2.分词与关键词提取; 3.词云与词频统计; 4.情...

爬取菜鸟教程网站并转PDF__python_crawer_by_chrome

A Facebook crawler

API of DouYin for Humans used to Crawl Popular Videos and Musics

What do people have in their dotfiles?

带你了解一下Golang的市场行情

Free Web Scraping Tool with Java

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

Locally saves webpages to your hard disk with images, css, js & links as is.

LinkedIn Scraper (currently working 2020)

Headless Chrome For Java （Java 爬虫）

小红书数据采集、网站图片、视频资源批量下载工具，颜值超高的数据采集工具（批量下载，视频提取，图片，去水印等）

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

a reliable high-level web crawling & scraping framework for Node.js.

swiss army knife for hackers

🌈Python3网络爬虫实战：QQ音乐歌曲、京东商品信息、房天下、破解有道翻译、构建代理池、豆瓣读书、百度图片、破解网易登录、B站模拟扫码登录、小鹅通、荔枝微课

Crawler for Nintendo Switch eShop

Crawljax

:newspaper: Let ChatGPT Summarize Hacker News for You

a new crawler based on python with more function including Network fingerprint search

A framework for creating semi-automatic web content extractors

Open-source Enterprise Grade Search Engine Software

收集各种免费的 Python 爬虫项目

Html网页正文提取

Simple yet powerful automation stuffs.

👩 美女写真套图爬虫（一）

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

This repository contains all the code I use in my YouTube tutorials.

🕵️ Python project to crawl for JavaScript files and search for secrets like API keys, authorization tokens, hardcoded credentials, etc.

dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok

Open source SEO audit tool.

:musical_note: 缓存文件转换为 MP3 文件

Second-order subdomain takeover scanner

A search engine that doesn't track you.

台灣股票即時爬蟲。Taiwan Stock Exchange Real Time Crawler

Videodl: A lightweight video downloader written by pure python.

Crawl GitHub APIs and store the discovered orgs, repos, commits, ...

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

今日头条机器人，支持用户登陆、关注、取消关注、获取关注粉丝、发文、发悟空问答、点赞、评论、采集各种类型新闻讯息等，使用今日头条网页版API实现

Android 本地网络小说爬虫，基于jsoup及xpath

JavaScript + BeautifulSoup = JSSoup

🕸️ Crawl in the web network

《爬虫逆向进阶实战》书籍代码库

A non API python program to crawl public photos, posts or followers

微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等...

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy