Most popular crawler repositories and open source projects

PY-Login

模拟登录各类网站，操作 API 完成各种不可描述的事情

10 26 26

pimcore-lucene-search

Pimcore Website Indexer (powered by Zend Search Lucene)

18 26 26

od-database-crawler

OD-Database Go crawler

5 26 26

LeetCodeCrawler

A tool for crawling the description and accepted submitted code of pro...

6 26 26

nivinEdu

拟物校园，一个开源的高校教务移动化解决方案。

10 26 26

tor-ip-rotation-python-example

An example of Tor IP rotation in Python

18 26 26

USPTO-PatFT-Web-Crawler

Crawler for fetching information of US Patents and PDF bulk download

16 26 26

douyin-sdk

联系微信（1764328791）、抖音SDK、抖音数据、抖音直播数据、抖音直播Api、...

9 26 26

PySitemap

Simple sitemap generator with Python 3

32 26 26

Crawling-Emails

Very simple bash script to crawl email addresses from a specific websi...

14 26 26

n46-crawler

Nogizaka46 Blog Crawler - 乃木坂46卒業成員部落格備份程式

6 26 26

FTPSearcher

Asynchronous file scanner and downloader for FTP servers. Also takes I...

4 26 26

get-site-urls

🔗 Get all of the URL's from a website.

8 26 26

cambridge

Terminal version of Cambridge Dictionary by default. Also supports Mer...

4 26 26

BestBuy-Parser

A personal tool using Python's Scrapy framework to scrape Best Buy's p...

0 26 26

narr

2 26 26

wallstreetcnScrapy

a crawler for wallstreetcn,finance.sina by Scrapy-新浪财经，同花顺财经...

3 25 25

tucan-tools

Nomen est omen. It exports tucan grades/vv etc.

4 25 25

zhihu-crawler

轻量级知乎爬虫，支持问题、收藏夹和本月最热

11 25 25

ProxyCrawler

代理爬虫服务，爬取代理IP并保存到 Redis 中, topshelf+Quartz.Net+redis

11 25 25

CrawlerDetectBundle

A Symfony bundle for the Crawler-Detect library (detects bots/crawlers...

10 25 25

CrowLeer

Powerful C++ web crawler based on libcurl

3 25 25

CyberCrowl

CyberCrowl is a python Web path scanner tool

10 25 25

Real_Time_Social_Media_Mining

DevOps pipeline for Real Time Social/Web Mining

9 25 25

crazyDhtSpider

Based on Swoole,a PHP DHT crawler, which have insane productivity（依...

13 25 25

social-media-archiver

A Node.js template to be implemented to archive post from any social m...

5 25 25

novelsave_sources

A collection of webnovel sources offering varying amounts of scraping...

2 25 25

marmot

💐Marmot A Golang HTTP Download

13 25 25

MedicalKG

医疗知识图谱构建实战，通过爬虫获取百度百科数据，使用Mongodb存储结构化...

4 25 25

Techweekly

高可配的技术周报邮件推送工具

4 24 24

realestate-scraper

A scraper that gathers data from real estate ads

16 24 24

FacePlusPlus-Stars-Library-Images-Crawler

Face++ starlib 明星库头像标注集爬虫及图片集合，用于face recognition tr...

17 24 24

PaperCrawler

Crawler used to crawl papers

9 24 24

soccer-scrape

:page_with_curl: Scrape football data from Bet365

22 24 24

AndroidValidatorCrawler

Kotlin library, Validator box that can inspect any type of form, provi...

3 24 24

Mimo-Crawler

A web crawler that uses Firefox and js injection to interact with webp...

2 24 24

dht

一个DHT爬虫

6 24 24

spider-mooc

本爬虫程序旨在从中国大学MOOC爬取相关课程的评论信息

6 24 24

images-grabber

🖼️ Get all images from pixiv/twitter/deviantart

2 24 24

kontests

Competitive programming contests schedule

8 24 24

scrapingant-client-python

ScrapingAnt API client for Python.

3 24 24

Python-Web-Scraping-Tutorial

In this Python Web Scraping Tutorial, we will outline everything neede...

6 24 24

Pinterest-Crawler

Downloads HD images from pinterest by keywords

3 24 24

dorker

Better Google Dorking with Dorker.

13 24 24

convertible-bond-crawler

宁稳网(旧富投网)、集思录可转债数据&策略分析

11 24 24

Amipy

A micro asynchronous Python website crawler framework .（Python微型异...

11 23 23

crawlerr

A simple and fully customizable web crawler/spider for Node.js with se...

7 23 23

onionstack

A Pictorial Book of Tor Hidden Services.

3 23 23

WebCrawler

one web crawler frame based on golang

15 23 23

little-python

little python projects, 一些小的python项目.

12 23 23