Most popular crawler repositories and open source projects

hQuery.php

An extremely fast web scraper that parses megabytes of invalid HTML in...

71 342 342

xcrawler

快速、简洁且强大的PHP爬虫框架

51 338 338

Search-Engines-Scraper

Search google, bing, yahoo, and other search engines with python

113 338 338

Moodle-DL

Moodle-DL downloads course content fast from Moodle (eg. lecture pdfs)

56 338 338

scrapy-zyte-smartproxy

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

89 337 337

media-scraper

Scrapes all photos and videos in a web page / Instagram / Twitter / Tu...

47 332 332

seonaut

Open source SEO auditing tool.

55 329 329

second-order

Second-order subdomain takeover scanner

65 328 328

polite

Be nice on the web

13 327 327

tiktok-downloader

Tiktok Downloader/Scraper using requests & bs4

85 321 321

Free_Proxy_Website

获取免费socks/https/http代理的网站集合

76 316 316

ppspider

web spider built by puppeteer, support task-queue and task-scheduling...

74 315 315

CrawlerTutorial

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

101 312 312

spidy

The simple, easy to use command line web crawler.

66 311 311

Laravel-Crawler-Detect

A Laravel wrapper for CrawlerDetect - the web crawler detection librar...

27 311 311

gopa

GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://inde...

82 296 296

pychromeless

Python Lambda Chrome Automation (naming pending)

124 294 294

Sasila

一个灵活、友好的爬虫框架

76 293 293

line-bot-tutorial

line-bot-tutorial use python flask

149 291 291

chinese-fund-crawler

中国场外基金数据爬取&汇总分析

159 290 290

PulsarRPA

Automate webpages at scale, scrape web data completely and accurately...

59 287 287

Instagram-Bot

An Instagram bot developed using the Selenium Framework

84 280 280

awesome-java-crawler

本仓库收集整理爬虫相关资源，开发语言以Java为主

65 276 276

oddish

Crawl csgo skin info from `buff.163.com` and steam, then find the most...

74 276 276

Fast-LianJia-Crawler

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀

100 274 274

th-music-video-generator

Touhou Project random music video generator/player, crawling image and...

44 274 274

Strong-Web-Crawler

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码...

154 272 272

sitemap-generator-cli

Creates an XML-Sitemap by crawling a given site.

41 268 268

weiboPicDownloader

免登录下载微博图片爬虫 Download Weibo Images without Logging-in

55 264 264

Gorecon

Gorecon is a All in one Reconnaissance Tool , a.k.a swiss knife for Re...

50 263 263

algoliasearch-netlify

Official Algolia Plugin for Netlify. Index your website to Algolia whe...

10 259 259

weibo_terminator_workflow

Update Version of weibo_terminator, This is Workflow Version aim at Ge...

78 258 258

Selenops

A Swift Web Crawler 🕷

17 257 257

bitextor

Bitextor generates translation memories from multilingual websites

44 255 255

arachnid

Crawl all unique internal links found on a given website, and extract...

64 252 252

wencai

This is a wencai crawler.（i问财的策略回测接口的Pythonic工具包）

108 252 252

Tumblr_Crawler

This is a Multi-thread crawler for Tumblr.

76 251 251

FileSensor

Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探...

80 250 250

chromium_for_spider

dynamic crawler for web vulnerability scanner

47 250 250

antch

Antch, a fast, powerful and extensible web crawling & scraping framewo...

42 250 250

Sub

节点爬取,筛选, 支持Clash,base64订阅解析,自动生成可用的ss, ssr, v2ray,...

99 249 249

spider

The fastest web crawler and indexer

34 246 246

Github-spider

Github 仓库及用户分析爬虫

88 241 241

ComicCrawler

An image crawler written in Python.

46 238 238

4chan-downloader

Python3 script to continuously download all images/webms of multiple 4...

32 237 237

wscan

一款开源的安全评估工具支持常见的 web 安全问题扫描和自定义 POC。此外，...

27 236 236

woid

Simple news aggregator displaying top stories in real time

122 235 235

RuiJi.Net

crawler framework, distributed crawler extractor

45 234 234

ok_ip_proxy_pool

🍿爬虫代理IP池(proxy pool) python🍟一个还ok的IP代理池

70 233 233

QQMusicSpider

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论...

61 229 229