Topic

crawler

Repositories (1501)

aliexpress-product-scraper
aliexpress-product-scraper sudheer-ranga JavaScript

Get Aliexpress product details as a json response including feedbacks, variants, shipping info, description, images, etc.,

331
SpotifyScraper
SpotifyScraper AliAkhtari78 Python

Extract public Spotify data — tracks, albums, artists, playlists, podcasts & lyrics — without the official API. Sync + async, typed models, one depend...

327
Laravel-Crawler-Detect
Laravel-Crawler-Detect JayBizzle PHP

A Laravel wrapper for CrawlerDetect - the web crawler detection library

326
awesome-java-crawler
awesome-java-crawler rockswang

本仓库收集整理爬虫相关资源,开发语言以Java为主

325
wencai
wencai GraySilver JavaScript

This is a wencai crawler.(i问财的策略回测接口的Pythonic工具包)

324
extractor
extractor lightfeed TypeScript

Use LLMs to robustly extract web data

322
oddish
oddish puppylpg Python

Crawl csgo skin info from `buff.163.com` and steam, then find the most suitable one to buy from the former and to sell to the latter.

321
gplay-scraper
gplay-scraper Mohammedcha Python

GPlay Scraper is a powerful Python Google Play scraper library for extracting comprehensive app data from the Google Play Store. Scrape Google Play St...

315
4chan-downloader
4chan-downloader Exceen Python

Python3 script to continuously download all images/videos of multiple 4chan threads simultaneously - without installation

312
crawler
crawler infinilabs Go

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

312
xvideos
xvideos rodrigogs TypeScript

xvideos API library

310
Python-Web-Scraping-Tutorial
Python-Web-Scraping-Tutorial oxylabs Python

In this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move...

309
site-audit-seo
site-audit-seo viasite JavaScript

Web service and CLI tool for SEO site audit: crawl site, lighthouse all pages, view public reports in browser. Also output to console, json, csv, xlsx

307
js-reverse
js-reverse freedom-wy HTML

JS逆向研究

304
Fast-LianJia-Crawler
Fast-LianJia-Crawler CaoZ Python

直接通过链家 API 抓取数据的极速爬虫,宇宙最快~~ 🚀

302
bitextor
bitextor bitextor Python

Bitextor generates translation memories from multilingual websites

301
LinkedIn-Scraper
LinkedIn-Scraper TufayelLUS Python

A LinkedIn Scraper to scrape up to 1k LinkedIn profiles(due to LinkedIn limit) from company profile links and save their e-mail addresses if available...

300
weibo-topic-spider
weibo-topic-spider czy1999 Python

微博超级话题爬虫,微博词频统计+情感分析+简单分类,新增肺炎超话爬取数据

297
ComicCrawler
ComicCrawler eight04 Python

An image crawler written in Python.

297
line-bot-tutorial
line-bot-tutorial twtrubiks Python

line-bot-tutorial use python flask

294
pychromeless
pychromeless jairovadillo Python

Python Lambda Chrome Automation (naming pending)

293
Sasila
Sasila da2vin Python

一个灵活、友好的爬虫框架

292
PixivCrawler
PixivCrawler cwher Python

Pixiv Utils implemented in Python, including Pixiv Crawler and Mosaic Puzzles, support for rankings, personal bookmarks, artist works and keyword sear...

291
Gorecon
Gorecon devanshbatham Go

Gorecon is a All in one Reconnaissance Tool , a.k.a swiss knife for Reconnaissance , A tool that every pentester/bughunter might wanna consider into...

287
Instagram-Bot
Instagram-Bot mustafadalga Python

An Instagram bot developed using the Selenium Framework

284
th-music-video-generator
th-music-video-generator Jasonnor JavaScript

Touhou Project random music video generator/player, crawling image and video from websites to generate MV.

279
laravel-seo-scanner
laravel-seo-scanner backstagephp PHP

Scan your Laravel application routes for SEO improvements suggestions.

275
go-movies
go-movies hezhizheng Go

golang spider Crawler 爬虫 电影

275
Strong-Web-Crawler
Strong-Web-Crawler microfisher C#

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。

274
D4N155
D4N155 OWASP Shell

OWASP D4N155 - Intelligent and dynamic wordlist using OSINT

272
weiboPicDownloader
weiboPicDownloader yAnXImIN Java

免登录下载微博图片 爬虫 Download Weibo Images without Logging-in

271
rotating-tor-http-proxy
rotating-tor-http-proxy zhaow-de Shell

A multi-arch image provides one HTTP proxy endpoint with many concurrent tunnels to the Tor network.

269
onecomic
onecomic hardwarecode Python

一本漫画

268
crypto-crawler-rs
crypto-crawler-rs crypto-crawler Rust

A rock-solid cryptocurrency crawler library.

266
ptt-alertor
ptt-alertor Ptt-Alertor Go

:loudspeaker: Ptt 文章通知機器人!Notify Ptt Article in Realtime

266
algoliasearch-netlify
algoliasearch-netlify algolia TypeScript

Official Algolia Plugin for Netlify. Index your website to Algolia when deploying your project to Netlify with the Algolia Crawler

266
RuiJi.Net
RuiJi.Net zhupingqi C#

crawler framework, distributed crawler extractor

264
antch
antch antchfx Go

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

264
galer
galer dwisiswant0 Go

A fast tool to fetch URLs from HTML attributes by crawl-in.

262
Github-spider
Github-spider chenjiandongx Python

Github 仓库及用户分析爬虫

261
siteone-crawler-gui
siteone-crawler-gui janreges Svelte

SiteOne Crawler GUI is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for develope...

260
robots-txt
robots-txt spatie PHP

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

259
Selenops
Selenops zntfdr Swift

A Swift Web Crawler 🕷

258
weibo_terminator_workflow
weibo_terminator_workflow lucasjinreal Python

Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!

257
ok_ip_proxy_pool
ok_ip_proxy_pool cwjokaka Python

🍿爬虫代理IP池(proxy pool) python🍟一个还ok的IP代理池

254
arachnid
arachnid zrashwani PHP

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

254
FileSensor
FileSensor Xyntax Python

Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探测工具

253
SpideyX
SpideyX RevoltSecurities Python

SpideyX a multipurpose Web Penetration Testing tool with asynchronous concurrent performance with multiple mode and configurations.

252
CSharpCrawler
CSharpCrawler zhaotianff C#

C#爬虫示例程序,想学习爬虫入门知识的可以看过来。后续会慢慢加入更多爬虫相关的知识。

252
InfinityCrawler
InfinityCrawler TurnerSoftware C#

A simple but powerful web crawler library for .NET

251