Most popular crawler repositories and open source projects

FF14AutoSignIn renchangjiu Python

FF14 国服官网自动签到脚本

40 6 0

lolcrawler jonaslejon Python

Headless web crawler for bugbounty and penetration-testing/redteaming

40 10 1

doogle safesploitOrg PHP

Doogle is a search engine and web crawler which can search indexed websites and images

40 19 6

Raven Symbolexe Go

Raven is a powerful and customizable web crawler written in Go.

40 7 1

wayurls alwalxed Go

CLI tool for fetching URLs from Wayback Machine, Common Crawl, and VirusTotal.

40 4 40

crawlee-one JuroOravec TypeScript

Production-ready web scraping in a single function call. Built on Crawlee.

40 6 2

acon WillyEverGreen Python

The intelligence layer for any web scraper. Pair with Scrapling, Playwright, or httpx to crawl smarter.

40 1 0

AntiCloudFlare s045pd HTML

对抗cloudflare载入页反爬虫防护（已失效）

39 21 0

auto_crawler_ptt_beauty_image twtrubiks Python

自動抓取 ptt 表特板圖片

39 18 3

crawler crawlerclub Go

Crawler4U, a general purpose focused crawler

39 6 4

fess-crawler codelibs Java

Web/FileSystem Crawler Library

39 18 12

Android-Apps-Downloader harismuneer Python

📱 A utility for downloading Android apps from the Google Play Store and Xiaomi App Store (the Chinese App Store).

39 21 2

selenium_facebook_scraper Mhmd-Hisham Python

A simple python3 script used to download a users's friend list from facebook.

39 21 4

papercut armand1m TypeScript

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Cachin...

39 2 1

novelsave_sources m-haisham Python

A collection of webnovel sources offering varying amounts of scraping capability.

39 4 2

CobWeb-lnx GoncaloMark Python

CobWeb is a Python library for web scraping. The library consists of two classes: Spider and Scraper.

39 2 2

proxy-in-a-box naiba Go

Automatic proxy pool for web scraping - crawls, validates, and rotates proxies with rate limiting and MITM support

39 4 1

python-facebook-bot tudoanh Python

Get facebook events from location with Python 3

38 15 1

BaiduImageCrawler flexwang-zz Python

A multithreaded tool for downloading search results of Baidu image search.

38 21 3

integrada.minhabiblioteca.com.br tharyckgusmao JavaScript

Download de livros para PDF/EPUB - Integrada.minhabiblioteca / vitalsource

38 10 9

generic-seeder team-exor C++

Generic altcoin DNS seeder. Compatible with virtually any cryptocurrency cloned from bitcoin. Built-in lightweight DNS server ~ Cloudflare DNS support...

38 96 7

ProxyScan Its-Vichy Go

🔎 scan the internet to find "private" proxies.

38 6 38

ExHentaiReader AndyHsiehTA HTML

Best manga-viewer on windows for crawling/downloading/browsing exhentai.

38 2 3

EH-PDF Galgamer-org Python

將一個 E-Hentai 畫廊下載並轉換成 PDF，方便在 Kindle 上閱讀以及在 iPad 上閱讀並作筆記，，，

38 2 38

article_crawler tychozzz Python

✨ Article Crawler is a package used to crawl articles with Markdown format from a specific webpage and store them locally in HTML / Markdown formats.

38 5 1

geckordp jpramosi Python

A client implementation of Firefox DevTools over remote debug protocol in python

38 15 1

WebRecon flashnuke Python

A collection of pentesting web scanners

38 3 2

xhs-js saifeiLee JavaScript

基于小红书web端的请求封装,JS实现

38 3 1

LLM-Web-Crawler buildship-ai TypeScript

Web Scraper and Crawler for LLM Apps and AI Workflows with NoCode / LowCode. Plug and play with your own logic and customize it flexibly and scalably...

38 8 3

NodeSpider Bin-Huang TypeScript

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

37 4 2

toxcrawler JFreegman C

A Tox DHT network crawler

37 12 7

Facebooker gpwork4u Python

an unofficial facebook api

37 9 2

Mini-Projects nazaninsbr Python

A collection of short projects, you could try and implement these as short projects or use them as part of a larger project.

37 14 2

Python fmw666 Python

🍋 Python基础、Pygame游戏编程、Python算法与面试题、四种常用的Python Web框架、爬虫、数据可视化、机器学习。一共七个Python大方向！

37 6 0

crawlhtmltopdf osdodo Python

一个将runoob.com转换为PDF的爬虫

37 11 37

xray_pool allanpk716 Go

基于 Xray-core、glider 的代理池工具

37 6 37

Data-Collection-Process-for-the-2024-Huashu-Cup-C-Problem Diraw Python

华数杯2024C题数据集收集过程

37 1 1

instagram-data-scraper Z786ZA

Instagram Data Scraper analyze profile

37 0 0

sneakpeek flulemon Python

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex sc...

37 0 1