crawler

uforall is a fast url crawler this tool crawl all URLs number of different sources, alienvault,WayBackMachine,urlscan,commoncrawl

Continuous scalable web crawler built on top of Flink and crawler-commons

A web browser :earth_americas: hosted as a service, to render your JavaScript web pages as HTML

网页解析器，用于网络爬虫解析页面, 不懂网页解析也能写爬虫

Scrapfly Python SDK for headless browsers and proxy rotation

Python script to download messages from a Facebook page to a CSV file

Riichi Mahjong Kit: (1) Game log crawler (sqlite3, json, bs4); (2) Game log preprocessor; (3) Deterministic algorithms library

基于规则的跨平台一站式聚合搜索工具

分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微服务架构，通过消息队列实现消息的异步发送，使用到的框架包括：redigo, gorm, goquer...

Deep web crawler and search engine

A Content Discovery and Development Platform. Empowering Cybersecurity, AI, Marketing, and Finance professionals and researchers to discover, analyze,...

百度莱茨狗爬虫。

支付宝账单爬虫

Scrapy, a fast high-level web crawling & scraping framework for dart and Flutter

抓取twitter数据，可根据时间、话题、用户名等条件抓取数据，twitter爬虫

search & get datas from youtube no google account needed

Open datasets of companies & websites grouped by technologies they use (CSV & JSON). Discover who uses Shopify, Stripe, Woocommerce, HubSpot, and more...

facebook-messenger-bot-tutorial use Python Django

A web service that turns an arbitrary web page into structural JSON data and easy-to-use APIs with just a few clicks

A fluent and functional approach to querying HTML

NASTY Advanced Search Tweet Yielder

简单、实用的爬虫工具，仅需四步创建属于你的爬虫程序！

爬虫学习仓库，适合零基础的人学习，对新手比较友好

Crawl novels from sfacg, ciweimao, esjzone, lightnovel and masiro; generate, append and extract epub

Armiarma is a Libp2p open-network crawler with a current focus on Ethereum's CL network

Next Crawler 是使用Playwright + Next.js + Prisma等主流技术搭建的网页数据采集器，通过可视化的UI进行配置，即可周期性的通过Playwright驱动浏览器爬取网页数...

12306查票助手，一键查询沿途所有站点，先上车后补票，让你的出行更省心。

Tool to brute website sub-domains and dirs.

API para recuperar informações sobre FII

A simple web crawler, using Abot, that indexes page contents into Azure Search.

⚡ A subdomain enumeration tool leveraging diverse techniques, designed for advanced pentesting operations

Crawl 100%-discount games on steam

Python script to get the leaderboard along with corresponding team details of the Dream11 contest we are participating in an excel sheet as soon as th...

Chromium / Puppeteer site crawler

抖音爬虫. 通过手机代理爬取用户的作品和用户的喜欢

抖音 SDK，数据采集，爬虫抓取不是梦

The project for Linebot

:rocket: 使用PyQt5图形界面的Python多线程nhentai爬虫

Extract instagram users informations from hashtags. This scraper can extract emails addresses from Bio section and business email.

🕷🚀 Scrapes/Crawls the logo from a provided url(s)/website for your Node.js applications.

Official Python library for interacting with Kameleo Client

读书笔记《自己动手写网络爬虫》，自己敲的代码。主要记录了网络爬虫的基本实现，网页去重的算法，网页指纹算法，文本信息挖掘

Cartographer: A new type of seed for the Bitcoin network

那些年，我爬过的北科。一个由浅入深的定向爬虫教程。

Crawler dos dados metereológicos de estações convencionais do INMET (BDMEP)

一个基于Scrapy的数据采集爬虫代码库

Simples crawler para obter resultados dos jogos de futebol

This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.