crawler

Some codes i wrote to help me with me with my daily errands ;)

Distributed crawling/scraping, Kafka And Redis based components for Scrapy

:mag: 简单的搜索引擎, django 框架

Crawl data from anilist API and store as JSON file

Android应用市场网络爬虫

BotCity Framework Web - Python

Repository for the Mastering Web Scraping in Python: Scaling to Distributed Crawling blogpost with the final code.

a crawler to fetch last news from Iranian(Persian) news agencies.

Official Supadata MCP Server - Adds powerful video & web scraping to Cursor, Claude and any other LLM clients.

⏬ Yet another torrent searching application for your command line

A django admin site for scrapy

⛏ A versatile Web scraper for Node.js

This is a crawler(reptile)

TseTmc Crawler

Official JavaScript/TypeScript library for interacting with Kameleo Client

Python tools for ethical hacking

A python package that takes tables from a web page and processes them to get high quality tables

Anatomy and Visualization of the Network structure of the Dark web using multi-threaded crawler

Retro Environment Canada Weather Channel for your browser

Bíblia católica "Ave Maria" em formato JSON

网络爬虫类库,基本可以实现自定义规则大部分网站

Rust Web Crawler saving pages on Redis

方便的从浏览器复制浏览器头

Generate an object for testing if a request is sent, request is Mikeal's request.

一个轻量级、快速、多线程、多管道、灵活配置的网络爬虫。

微信公众号文章爬虫

Unofficial Python client for Twitter

🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.

Web-crawler-engineer-for-Python

:robot: Python bot that crawls your website looking for dead stuff

Login some website using selenium.

:cyclone: crawl bilibili user info and video info for data analysis | BiliBili爬虫

Nogizaka46 Blog Crawler - 乃木坂46卒業成員部落格備份程式

ScrapingAnt API client for Python.

A fast and powerful web scraping library

A multithreaded tool for searching and downloading images from popular search engines. It is straightforward to set up and run!

An admin UI project for a configurable web crawler platform

Web Crawler written in C#

JAVA实现的IP代理池，支持HTTP与HTTPS两种方式

This is a Tiktok Crawler App.

一个python爬虫来爬取洛谷各种信息

Pre-rendering for JavaScript websites that delivers SSR-level SEO, enhanced link previews, and performance via effortless middleware integration — ide...

A stock market text sentiment analysis website. A股舆情分析, web-crawler, bayesian algorithm, SQL, django, data-visualization.

ronin-web is a collection of useful web helper methods and commands.

This repository is deprecated

project(NSFW): hentai contents combined with multi sources daily

:spider: A simple crawler (spider) writen in php just for fun, with zero dependencies

nodejs 爬虫框架. crawler framework for nodejs

正方教务管理系统学生版的自动登录、选课、信息获取

Crawler for leboncoin.fr