Most popular crawler repositories and open source projects

zhihu-login

知乎模拟登录，支持提取验证码和保存 Cookies

140 355 355

Rcrawler

An R web crawler and scraper

92 353 353

supercrawler

A web crawler. Supercrawler automatically crawls websites. Define cust...

66 351 351

91porn-api

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新）及在线web...

28 350 350

spidy

The simple, easy to use command line web crawler.

69 349 349

magic_google

Google search results crawler, get google search results that you need

111 345 345

tsec

台灣上市上櫃股票爬蟲 Taiwan Stock Exchange Crawler

169 344 344

hQuery.php

An extremely fast web scraper that parses megabytes of invalid HTML in...

71 342 342

xcrawler

快速、简洁且强大的PHP爬虫框架

51 338 338

Search-Engines-Scraper

Search google, bing, yahoo, and other search engines with python

113 338 338

Moodle-DL

Moodle-DL downloads course content fast from Moodle (eg. lecture pdfs)

56 338 338

ppspider

web spider built by puppeteer, support task-queue and task-scheduling...

77 336 336

polite

Be nice on the web

13 327 327

tiktok-downloader

Tiktok Downloader/Scraper using requests & bs4

85 321 321

telegram-crawler

🕷 Automatically detect changes made to the official Telegram sites, cl...

37 318 318

Free_Proxy_Website

获取免费socks/https/http代理的网站集合

76 316 316

lightnovel_epub

🍭 epub generator for (light)novels (轻)小说 epub 生成器，支持站点：轻...

21 314 314

Laravel-Crawler-Detect

A Laravel wrapper for CrawlerDetect - the web crawler detection librar...

27 313 313

crawler

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

83 309 309

nudecrawler

Crawl telegra.ph searching for nudes!

28 305 305

line-bot-tutorial

line-bot-tutorial use python flask

151 298 298

Sasila

一个灵活、友好的爬虫框架

69 296 296

pychromeless

Python Lambda Chrome Automation (naming pending)

124 294 294

chinese-fund-crawler

中国场外基金数据爬取&汇总分析

159 290 290

PulsarRPA

Automate webpages at scale, scrape web data completely and accurately...

59 287 287

Instagram-Bot

An Instagram bot developed using the Selenium Framework

84 281 281

Python-Web-Scraping-Tutorial

In this Python Web Scraping Tutorial, we will outline everything neede...

30 280 280

awesome-java-crawler

本仓库收集整理爬虫相关资源，开发语言以Java为主

65 276 276

oddish

Crawl csgo skin info from `buff.163.com` and steam, then find the most...

74 276 276

Fast-LianJia-Crawler

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀

100 274 274

th-music-video-generator

Touhou Project random music video generator/player, crawling image and...

44 274 274

Strong-Web-Crawler

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码...

154 272 272

sitemap-generator-cli

Creates an XML-Sitemap by crawling a given site.

41 268 268

go-movies

golang spider Crawler 爬虫电影

81 268 268

Github-spider

Github 仓库及用户分析爬虫

91 266 266

scrapper

Web scraper with a simple REST API living in Docker and using a Headle...

41 266 266

weiboPicDownloader

免登录下载微博图片爬虫 Download Weibo Images without Logging-in

55 264 264

Gorecon

Gorecon is a All in one Reconnaissance Tool , a.k.a swiss knife for Re...

50 263 263

antch

Antch, a fast, powerful and extensible web crawling & scraping framewo...

41 263 263

algoliasearch-netlify

Official Algolia Plugin for Netlify. Index your website to Algolia whe...

10 259 259

weibo_terminator_workflow

Update Version of weibo_terminator, This is Workflow Version aim at Ge...

78 258 258

Selenops

A Swift Web Crawler 🕷

17 257 257

ok_ip_proxy_pool

🍿爬虫代理IP池(proxy pool) python🍟一个还ok的IP代理池

68 256 256

arachnid

Crawl all unique internal links found on a given website, and extract...

59 255 255

bitextor

Bitextor generates translation memories from multilingual websites

44 255 255

wencai

This is a wencai crawler.（i问财的策略回测接口的Pythonic工具包）

108 252 252

Tumblr_Crawler

This is a Multi-thread crawler for Tumblr.

76 251 251

FileSensor

Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探...

80 250 250

chromium_for_spider

dynamic crawler for web vulnerability scanner

47 250 250

Sub

节点爬取,筛选, 支持Clash,base64订阅解析,自动生成可用的ss, ssr, v2ray,...

99 249 249