Most popular crawler repositories and open source projects

js-cookie-monitor-debugger-hook JSREI JavaScript

js cookie逆向利器：js cookie变动监控可视化工具 & js cookie hook打条件断点

41 10 41

noscrape schoenbergerb TypeScript

This repository is deprecated

41 8 41

aristotle egcodes Python

highly customizable news collector

40 5 40

php-crawler elboletaire PHP

:spider: A simple crawler (spider) writen in php just for fun, with zero dependencies

40 19 40

laundry endquote JavaScript

Data laundering tools

40 4 40

USTBCrawlers nladuo Python

那些年，我爬过的北科。一个由浅入深的定向爬虫教程。

40 7 40

ncrawler kant2002 C#

Web Crawler written in C#

40 14 40

HttpProxy asche910 Java

JAVA实现的IP代理池，支持HTTP与HTTPS两种方式

40 26 40

sponge spypunk Kotlin

sponge is a website crawler and links downloader command-line tool

40 2 40

MahjongKit erreurt Python

Riichi Mahjong Kit: (1) Game log crawler (sqlite3, json, bs4); (2) Game log preprocessor; (3) Deterministic algorithms library

40 9 40

lezhin-comics-downloader ImSejin Java

📥 Downloader for lezhin comics

40 3 40

scrapingant-client-python ScrapingAnt Python

ScrapingAnt API client for Python.

40 5 40

ICLR2023-OpenReviewData fedebotu Jupyter Notebook

Crawl & Visualize ICLR 2023 Data from OpenReview

40 3 40

cewler roys Python

CeWLeR - Custom Word List generator Redefined. CeWL alternative in Python, based on the Scrapy framework.

40 2 40

insecres kkomelin Go

A console tool that finds insecure resources on HTTPS sites

39 2 39

SpiderWho lanrat Python

A very fast whois crawler

39 14 39

podcastcrawler podcastcrawler PHP

PHP library to find podcasts

39 10 39

WebCrawler zhk0603 C#

一个轻量级、快速、多线程、多管道、灵活配置的网络爬虫。

39 14 39

LuoguCrawler himself65 Python

一个python爬虫来爬取洛谷各种信息

39 15 39

AppCrawler tongtzeho Python

Android应用市场网络爬虫

39 19 39

crawel MrXujiang JavaScript

基于Apify+node+react搭建的有点意思的爬虫平台

39 16 39

dijnet-bot juzraai JavaScript

Az összes számlád még egy helyen :)

39 3 39

Domainker BitTheByte Python

BugBounty Tool

38 21 38

TripAdvisor_crawler Tang-Li-Jen Python

Python Crawler: Scrape Data From Tripadvisor

38 10 38

BaiduImageCrawler flexwang-zz Python

A multithreaded tool for downloading search results of Baidu image search.

38 20 38

leboncoin-crawler rfussien HTML

Crawler for leboncoin.fr

38 17 38

ArticleSpider hackfengJam Python

Crawling zhihu, jobbole, lagou by Scrapy, and using Elasticsearch+Django to build a Search Engine website --- README_zh.md (including: implementation...

38 10 38

CrawlerSamples VAllens C#

This is a Puppeteer+AngleSharp crawler console app samples, used C# 7.1 coding and dotnet core build.

38 13 38

tiktok-crawler hackertogether Python

This is a Tiktok Crawler App.

38 16 38

DeadPool Ryuchen Python

该项目是一个使用celery作为主体框架的爬虫应用，能够灵活的添加爬虫任务，并且同时运行多站点的爬虫工作，所有组件都能够原生支持规模并发和分布式，加上celery...

38 15 38

SeleniumLogin CharlesPikachu Python

Login some website using selenium.

38 17 38

crawlerdetect x-way Go

Golang module to detect bots and crawlers via the user agent

38 5 38

papercut armand1m TypeScript

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Cachin...

38 2 38