Most popular crawler repositories and open source projects

91porn-api

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web...

34   346   346  

magic_google

Google search results crawler, get google search results that you need

111   345   345  

tsec

台灣上市上櫃股票爬蟲 Taiwan Stock Exchange Crawler

169   344   344  

hQuery.php

An extremely fast web scraper that parses megabytes of invalid HTML in...

71   342   342  

xcrawler

快速、简洁且强大的PHP爬虫框架

51   338   338  

Search-Engines-Scraper

Search google, bing, yahoo, and other search engines with python

113   338   338  

Moodle-DL

Moodle-DL downloads course content fast from Moodle (eg. lecture pdfs)

56   338   338  

ppspider

web spider built by puppeteer, support task-queue and task-scheduling...

77   336   336  

media-scraper

Scrapes all photos and videos in a web page / Instagram / Twitter / Tu...

47   332   332  

seonaut

Open source SEO auditing tool.

55   329   329  

second-order

Second-order subdomain takeover scanner

65   328   328  

polite

Be nice on the web

13   327   327  

tiktok-downloader

Tiktok Downloader/Scraper using requests & bs4

85   321   321  

Free_Proxy_Website

获取免费socks/https/http代理的网站集合

76   316   316  

lightnovel_epub

🍭 epub generator for (light)novels (轻)小说 epub 生成器，支持站点：轻...

21   314   314  

CrawlerTutorial

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

101   312   312  

spidy

The simple, easy to use command line web crawler.

66   311   311  

Laravel-Crawler-Detect

A Laravel wrapper for CrawlerDetect - the web crawler detection librar...

27   311   311  

crawler

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

82   308   308  

nudecrawler

Crawl telegra.ph searching for nudes!

28   305   305  

Sasila

一个灵活、友好的爬虫框架

69   296   296  

pychromeless

Python Lambda Chrome Automation (naming pending)

124   294   294  

line-bot-tutorial

line-bot-tutorial use python flask

149   291   291  

chinese-fund-crawler

中国场外基金数据爬取&汇总分析

159   290   290  

PulsarRPA

Automate webpages at scale, scrape web data completely and accurately...

59   287   287  

Python-Web-Scraping-Tutorial

In this Python Web Scraping Tutorial, we will outline everything neede...

30   280   280  

Instagram-Bot

An Instagram bot developed using the Selenium Framework

84   280   280  

awesome-java-crawler

本仓库收集整理爬虫相关资源，开发语言以Java为主

65   276   276  

oddish

Crawl csgo skin info from `buff.163.com` and steam, then find the most...

74   276   276  

Fast-LianJia-Crawler

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀

100   274   274  

th-music-video-generator

Touhou Project random music video generator/player, crawling image and...

44   274   274  

Strong-Web-Crawler

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码...

154   272   272  

sitemap-generator-cli

Creates an XML-Sitemap by crawling a given site.

41   268   268  

weiboPicDownloader

免登录下载微博图片 爬虫 Download Weibo Images without Logging-in

55   264   264  

Gorecon

Gorecon is a All in one Reconnaissance Tool , a.k.a swiss knife for Re...

50   263   263  

antch

Antch, a fast, powerful and extensible web crawling & scraping framewo...

41   262   262  

algoliasearch-netlify

Official Algolia Plugin for Netlify. Index your website to Algolia whe...

10   259   259  

weibo_terminator_workflow

Update Version of weibo_terminator, This is Workflow Version aim at Ge...

78   258   258  

Selenops

A Swift Web Crawler 🕷

17   257   257  

arachnid

Crawl all unique internal links found on a given website, and extract...

59   255   255  

bitextor

Bitextor generates translation memories from multilingual websites

44   255   255  

ok_ip_proxy_pool

🍿爬虫代理IP池(proxy pool) python🍟一个还ok的IP代理池

68   254   254  

wencai

This is a wencai crawler.（i问财的策略回测接口的Pythonic工具包）

108   252   252  

Tumblr_Crawler

This is a Multi-thread crawler for Tumblr.

76   251   251  

FileSensor

Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探...

80   250   250  

chromium_for_spider

dynamic crawler for web vulnerability scanner

47   250   250  

Sub

节点爬取,筛选, 支持Clash,base64订阅解析,自动生成可用的ss, ssr, v2ray,...

99   249   249  

D4N155

OWASP D4N155 - Intelligent and dynamic wordlist using OSINT

48   247   247  

scrapper

Web scraper with a simple REST API living in Docker and using a Headle...

37   243   243  

web-page-monitor

Web Site Page Changes Monitor. 网站网页页面更新变更监控提醒。

40   242   242