Most popular crawler repositories and open source projects

PyperGrabber

Fetches PubMed article IDs (PMIDs) from email inbox, then crawls PubMe...

7 27 27

Android-Apps-Downloader

📱 A tool to download android apps from Google Play Store and Xiaomi A...

22 27 27

generic-seeder

Generic altcoin DNS seeder. Compatible with virtually any cryptocurren...

80 27 27

YourLesson

深圳大学抢课系统

6 27 27

spider

A web spider framework

7 27 27

selenium_facebook_scraper

A simple python3 script used to download a users's friend list from fa...

15 27 27

botcity-framework-web-python

BotCity Framework Web - Python

17 27 27

pinscrape

A simple library to scrape Pinterest images written in Python

3 27 27

iranian-news-agencies-crawler

a crawler to fetch last news from Iranian(Persian) news agencies.

2 27 27

AyugeSpiderTools

scrapy 扩展库：其主要功能使 scrapy 开发不用在意 item，pipeline，middle...

3 27 27

n46-crawler

Nogizaka46 Blog Crawler - 乃木坂46卒業成員部落格備份程式

6 26 26

froxy

Hide your IP with free proxies using Froxy 🔄

2 26 26

FTPSearcher

Asynchronous file scanner and downloader for FTP servers. Also takes I...

4 26 26

get-site-urls

🔗 Get all of the URL's from a website.

8 26 26

cambridge

Terminal version of Cambridge Dictionary by default. Also supports Mer...

4 26 26

BestBuy-Parser

A personal tool using Python's Scrapy framework to scrape Best Buy's p...

0 26 26

narr

2 26 26

serverless-crawler-demo

Serverless Architecture Crawler demo

8 26 26

PY-Login

模拟登录各类网站，操作 API 完成各种不可描述的事情

10 26 26

pimcore-lucene-search

Pimcore Website Indexer (powered by Zend Search Lucene)

18 26 26

od-database-crawler

OD-Database Go crawler

5 26 26

LeetCodeCrawler

A tool for crawling the description and accepted submitted code of pro...

6 26 26

nivinEdu

拟物校园，一个开源的高校教务移动化解决方案。

10 26 26

tor-ip-rotation-python-example

An example of Tor IP rotation in Python

18 26 26

USPTO-PatFT-Web-Crawler

Crawler for fetching information of US Patents and PDF bulk download

16 26 26

douyin-sdk

联系微信（1764328791）、抖音SDK、抖音数据、抖音直播数据、抖音直播Api、...

9 26 26

PySitemap

Simple sitemap generator with Python 3

32 26 26

wallstreetcnScrapy

a crawler for wallstreetcn,finance.sina by Scrapy-新浪财经，同花顺财经...

3 25 25

tucan-tools

Nomen est omen. It exports tucan grades/vv etc.

4 25 25

zhihu-crawler

轻量级知乎爬虫，支持问题、收藏夹和本月最热

11 25 25

ProxyCrawler

代理爬虫服务，爬取代理IP并保存到 Redis 中, topshelf+Quartz.Net+redis

11 25 25

CrawlerDetectBundle

A Symfony bundle for the Crawler-Detect library (detects bots/crawlers...

11 25 25

CyberCrowl

CyberCrowl is a python Web path scanner tool

10 25 25

Real_Time_Social_Media_Mining

DevOps pipeline for Real Time Social/Web Mining

9 25 25

crazyDhtSpider

Based on Swoole,a PHP DHT crawler, which have insane productivity（依...

13 25 25

social-media-archiver

A Node.js template to be implemented to archive post from any social m...

5 25 25

novelsave_sources

A collection of webnovel sources offering varying amounts of scraping...

2 25 25

marmot

💐Marmot A Golang HTTP Download

13 25 25

MedicalKG

医疗知识图谱构建实战，通过爬虫获取百度百科数据，使用Mongodb存储结构化...

4 25 25

kontests

Competitive programming contests schedule

8 24 24

Pinterest-Crawler

Downloads HD images from pinterest by keywords

3 24 24

dorker

Better Google Dorking with Dorker.

13 24 24

convertible-bond-crawler

宁稳网(旧富投网)、集思录可转债数据&策略分析

11 24 24

Techweekly

高可配的技术周报邮件推送工具

4 24 24

realestate-scraper

A scraper that gathers data from real estate ads

16 24 24

FacePlusPlus-Stars-Library-Images-Crawler

Face++ starlib 明星库头像标注集爬虫及图片集合，用于face recognition tr...

17 24 24

PaperCrawler

Crawler used to crawl papers

9 24 24

AndroidValidatorCrawler

Kotlin library, Validator box that can inspect any type of form, provi...

3 24 24

dht

一个DHT爬虫

6 24 24

spider-mooc

本爬虫程序旨在从中国大学MOOC爬取相关课程的评论信息

6 24 24