Most popular crawler repositories and open source projects

m3u8Downloader

meijuba.net,Python crawler,M3U8格式视频下载，桌面应用

22 56 56

devsearch

A web search engine built with Python which uses TF-IDF and PageRank t...

13 56 56

SearchEngineScrapy

Scrape data from Google.com, Bing.com, Baidu.com, Ask.com, Yahoo.com,...

16 56 56

actor-facebook-scraper

Scrape public Facebook pages, posts, reviews and comments

32 56 56

skweez

Fast website scraper and wordlist generator

5 56 56

telegram-groups-crawler

A Telegram crawler made in Python to automatically search groups and c...

23 55 55

JMComic-Crawler-Python

Python API For JMComic (禁漫天堂)

141 55 55

talospider

talospider - A simple,lightweight scraping micro-framework

4 55 55

MyCrawler

我的爬虫合集

3 55 55

All-IT-eBooks-Spider

[Updated] A simple python crawler for my tutorial blog at http://www.j...

34 55 55

bolsa

Biblioteca feita em Python com o objetivo de facilitar o acesso a dado...

16 55 55

simple_bank_korea

simple crawler for Korean banks with Transactions

9 55 55

nest-crawler

An easiest crawling and scraping module for NestJS

8 55 55

kalel

Kal El Network Stress Test and Penetration Testing Toolkit

16 54 54

tool-gin

基于go-gin框架建立减少冗余动作项目，如：下载一些工具

20 54 54

instagram-hashtag-crawler

Crawl Instagram hashtags

19 54 54

browser-as-a-service

A web browser :earth_americas: hosted as a service, to render your Jav...

12 54 54

alipay-crawler

支付宝账单爬虫

16 53 53

go-crawler-distributed

分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微...

8 53 53

OpenCrawler

Open Crawler || Open Source Crawler

5 53 53

Deepminer

Deep web crawler and search engine

13 52 52

WebTable

A python package that takes tables from a web page and processes them...

2 52 52

rarbgcli

RARBG command line interface for scraping the rarbg.to torrent search...

10 52 52

flink-crawler

Continuous scalable web crawler built on top of Flink and crawler-comm...

18 52 52

findopendata

A search engine for Open Data

6 52 52

site-mirror-py

[码云](https://gitee.com/generals-space/site-mirror-py) 通用爬虫, 仿站...

18 52 52

facebook-messenger-bot-tutorial

facebook-messenger-bot-tutorial use Python Django

12 51 51

fb-page-chat-download

Python script to download messages from a Facebook page to a CSV file

33 51 51

baidu-chain-dog

百度莱茨狗爬虫。

15 51 51

GPlayCrawler

13 51 51

price-monitoring

Node.js price monitoring library, leveraging the power of x-ray and ni...

7 51 51

tw-stock-telegram-bot

台股機器人，提供即時個股及大盤報價、走勢、新聞、盤後資料等 Telegram bo...

7 51 51

crawler

Libraries and scripts for crawling the TYPO3 page tree. Used for re-ca...

81 51 51

crawler_shopee

Shopee coin getter is a script to collect daily shopee coins.

16 51 51

crawler-userscript

一个基于 Tampermonkey 插件平台开发的爬虫。主要目的是最大限度模拟用户环...

18 51 51

open-gov-crawlers

Parse government documents into well formed JSON

4 51 51

TwitterCrawler

抓取twitter数据，可根据时间、话题、用户名等条件抓取数据，twitter爬虫

15 50 50

Crawling-CV-Conference-Papers

Crawling CV conference papers with Python.

6 50 50

kepub

Crawl novels from sfacg, ciweimao, esjzone, lightnovel and masiro; gen...

14 50 50

billboard-json

🎧 Get json type billboard hot 100 chart

6 50 50

Timbr_V1

A web service that turns an arbitrary web page into structural JSON da...

1 50 50

bloodhound

36 50 50

nasty

NASTY Advanced Search Tweet Yielder

8 50 50

snapcrawl

Crawl a website and take screenshots

10 50 50

html-query

A fluent and functional approach to querying HTML

11 49 49

scrapy.dart

Scrapy, a fast high-level web crawling & scraping framework for dart a...

8 49 49

logo-scrape

🕷🚀 Scrapes/Crawls the logo from a provided url(s)/website for your No...

10 49 49

Mini-Spider

简单、实用的爬虫工具，仅需四步创建属于你的爬虫程序！

23 48 48

rolling-news

获取滚动新闻

13 48 48

DouYinSDK

抖音 SDK，数据采集，爬虫抓取不是梦

9 48 48