Most popular crawler repositories and open source projects

unfx-proxy-parser

Unfx Proxy Parser - Nextgen proxy parser with deep links crawler. Foll...

16   48   48  

local-api-client-python

Official Python library for interacting with Kameleo Client

6   48   48  

Awesome-Scrapy

一个基于Scrapy的数据采集爬虫代码库

16   47   47  

tors

⏬ Yet another torrent searching application for your command line

5   47   47  

URLBrute-Py

Tool to brute website sub-domains and dirs.

8   47   47  

httpseed

Cartographer: A new type of seed for the Bitcoin network

25   47   47  

wishlist

Read an Amazon wishlist programmatically with Python

13   47   47  

crawler_JD_what_worthy_buying

爬取京东商品所有评论，利用情感分析，判断商品是否值得买

13   47   47  

SearchX

基于规则的跨平台一站式聚合搜索工具

15   47   47  

codes-scratch-crawler

读书笔记《自己动手写网络爬虫》，自己敲的代码。主要记录了网络爬虫的基本...

21   46   46  

scrapy-admin

A django admin site for scrapy

12   46   46  

maman

Rust Web Crawler saving pages on Redis

6   46   46  

crawler

Chromium / Puppeteer site crawler

5   46   46  

gscholar-citations-crawler

Crawl all your citations from Google Scholar

11   46   46  

scaling-to-distributed-crawling

Repository for the Mastering Web Scraping in Python: Scaling to Distri...

9   46   46  

webrtc-local-ip-leak

Oh no, stop this. You can see my local IP address 😲! Use `foundation`...

5   46   46  

local-api-client-typescript

Official JavaScript/TypeScript library for interacting with Kameleo Cl...

3   45   45  

scrapy-kafka-redis

Distributed crawling/scraping, Kafka And Redis based components for S...

13   45   45  

jason-the-miner

⛏ A versatile Web scraper for Node.js

11   45   45  

bthello

Python3 DHT 磁力种子爬虫 种子解析 种子搜索 演示地址

26   45   45  

Web-crawler-engineer-for-Python

Web-crawler-engineer-for-Python

23   44   44  

crawler-jsoup-maven

This is a crawler(reptile)

24   44   44  

dbworld-search

:mag: 简单的搜索引擎, django 框架

18   44   44  

seenreq

Generate an object for testing if a request is sent, request is Mikeal...

9   44   44  

bluebird

Unofficial Python client for Twitter

14   44   44  

shopify-app-store-scraper

Crawler behind the Shopify App Marketplace dataset

12   44   44  

jadwalsholatorg

Parsed data from website https://jadwalsholat.org

19   44   44  

FunUtils

Some codes i wrote to help me with me with my daily errands ;)

51   43   43  

copyheaders

方便的从浏览器复制浏览器头

9   43   43  

anilist-crawler

Crawl data from anilist API and store in MariaDB.

3   43   43  

douyin-crawler

抖音爬虫. 通过手机代理爬取用户的作品和用户的喜欢

11   43   43  

gogetcrawl

Extract web archive data using Wayback Machine and Common Crawl

7   43   43  

FF14AutoSignIn

FF14 国服官网自动签到脚本

7   43   43  

webmagician-ui

An admin UI project for a configurable web crawler platform

14   42   42  

Broken-Link-Crawler

:robot: Python bot that crawls your website looking for dead stuff

13   42   42  

nhentai-imgcollect

:rocket: 使用PyQt5图形界面的Python多线程nhentai爬虫

10   42   42  

steam-discount

steam 特惠游戏榜单（自动刷新）

15   42   42  

aio-vextractor

解析视频 网站/APP/H5 页面视频信息。支持抖音、腾讯视频、YouTube、Instag...

11   41   41  

ronin-web

ronin-web is a collection of useful web helper methods and commands.

6   41   41  

PSGameSpider

自动爬取所有PlayStationStore中的所有游戏封面，自动生成网页并索引 # # #...

5   41   41  

js-cookie-monitor-debugger-hook

js cookie逆向利器：js cookie变动监控可视化工具 & js cookie hook打条件...

10   41   41  

noscrape

This repository is deprecated

8   41   41  

spider.npm

网络爬虫类库,基本可以实现自定义规则大部分网站

7   41   41  

python-facebook-bot

Get facebook events from location with Python 3

16   41   41  

Crawler

:snake:A collection of simple Python crawlers.

15   41   41  

AntiCloudFlare

对抗cloudflare载入页反爬虫防护（已失效）

21   41   41  

crawler

nodejs 爬虫框架. crawler framework for nodejs

9   41   41  

PyTse

TseTmc Crawler

8   41   41  

wx-crawl

微信公众号文章爬虫

12   41   41  

stock_linebot_public

The project for Linebot

9   41   41