Topic

crawler

Repositories (1501)

trusted-ips-whitelist
trusted-ips-whitelist sefinek JavaScript

Collection of IP addresses used by legitimate crawlers and services: Googlebot, Bingbot, AhrefsBot, UptimeRobot, Pingdom, Cloudflare, Bunny CDN, Strip...

47
Awesome-Scrapy
Awesome-Scrapy Threekiii Python

一个基于Scrapy的数据采集爬虫代码库

47
AppCrawler
AppCrawler tongtzeho Python

Android应用市场网络爬虫

47
codes-scratch-crawler
codes-scratch-crawler duoan Java

读书笔记《自己动手写网络爬虫》,自己敲的代码。主要记录了网络爬虫的基本实现,网页去重的算法,网页指纹算法,文本信息挖掘

47
GooFish-AIMonitor
GooFish-AIMonitor tristanwqy Python

🐟 闲鱼/Goofish 智能监控控制台:AI 大模型选品 + 收藏降价·售罄即时提醒,全本地运行不外传。Xianyu second-hand deal monitor — LLM-filtered picks, price-d...

46
douyin_one_spider
douyin_one_spider mashukui Python

抖音数据采集GUI工具:关键词作品与评论、主页作品批量采集、无水印视频下载、链接与UID互转,CSV导出,Windows/macOS开箱即用

46
DarkSpider
DarkSpider PROxZIMA Python

Anatomy and Visualization of the Network structure of the Dark web using multi-threaded crawler

46
scaling-to-distributed-crawling
scaling-to-distributed-crawling ZenRows HTML

Repository for the Mastering Web Scraping in Python: Scaling to Distributed Crawling blogpost with the final code.

46
botcity-framework-web-python
botcity-framework-web-python botcity-dev Python

BotCity Framework Web - Python

46
anilist-crawler
anilist-crawler soruly TypeScript

Crawl data from anilist API and store as JSON file

46
dbworld-search
dbworld-search heqin-zhu HTML

:mag: 简单的搜索引擎, django 框架

46
FunUtils
FunUtils HoussemCharf Python

Some codes i wrote to help me with me with my daily errands ;)

46
acon
acon WillyEverGreen Python

The intelligence layer for any web scraper. Pair with Scrapling, Playwright, or httpx to crawl smarter.

45
TaiwanLotteryCrawler
TaiwanLotteryCrawler stu01509 Python

Taiwan Lottery Crawler 台灣 樂透 彩券 爬蟲

45
WebTable
WebTable AtomEcho Python

A python package that takes tables from a web page and processes them to get high quality tables

45
local-api-client-typescript
local-api-client-typescript kameleo-io TypeScript

Official JavaScript/TypeScript library for interacting with Kameleo Client

45
crawlerdetect
crawlerdetect moskrc Python

🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.

45
PyTse
PyTse miladj Python

TseTmc Crawler

45
crawler-jsoup-maven
crawler-jsoup-maven bluetata Java

This is a crawler(reptile)

45
scrapy-admin
scrapy-admin liangWenPeng Python

A django admin site for scrapy

45
tors
tors murat Ruby

⏬ Yet another torrent searching application for your command line

45
scrapy-kafka-redis
scrapy-kafka-redis tenlee2012 Python

Distributed crawling/scraping, Kafka And Redis based components for Scrapy

45
jason-the-miner
jason-the-miner mawrkus JavaScript

⛏ A versatile Web scraper for Node.js

45
crawl4weibo
crawl4weibo Praeviso Python

An out-of-the-box Weibo scraper Python library, based on a successfully tested solution, usable without cookies.

44
maw
maw memvid TypeScript

Crawl any website into a single searchable file. Query it forever, offline.

44
proxy-in-a-box
proxy-in-a-box naiba Go

Automatic proxy pool for web scraping - crawls, validates, and rotates proxies with rate limiting and MITM support

44
MCPDocSearch
MCPDocSearch alizdavoodi Python

This project provides a toolset to crawl websites wikis, tool/library documentions and generate Markdown documentation, and make that documentation se...

44
scrapy-diario-oficial-da-uniao
scrapy-diario-oficial-da-uniao sinayra Python

Script Python para buscar o conteúdo do Diário Oficial da União

44
scrapy-zyte-api
scrapy-zyte-api scrapy-plugins Python

Zyte API integration for Scrapy

44
python-hacking-tools
python-hacking-tools cristianzsh Python

Python tools for ethical hacking

44
n46-crawler
n46-crawler janelin612 JavaScript

Nogizaka46 Blog Crawler - 乃木坂46卒業成員部落格備份程式

44
WebCrawler
WebCrawler zhk0603 C#

一个轻量级、快速、多线程、多管道、灵活配置的网络爬虫。

44
maman
maman spk Rust

Rust Web Crawler saving pages on Redis

44
spider.npm
spider.npm Ireoo JavaScript

网络爬虫类库,基本可以实现自定义规则大部分网站

44
bluebird
bluebird labteral Python

Unofficial Python client for Twitter

44
redbooks
redbooks xiaofuqing13 Python

小红书爬虫工具 - 支持关键词搜索、主页推荐、博主主页爬取,图片视频批量下载,评论采集,Excel导出,SQLite存储,GUI图形界面 | Python + DrissionPage

43
PaperWebCrawler
PaperWebCrawler yagol2020 Java

IEEE XPLORE等文献网站的爬虫工具/Crawler for Paper Website like IEEE XPLORE

43
CygnusX1
CygnusX1 datnnt1997 Python

A multithreaded tool for searching and downloading images from popular search engines. It is straightforward to set up and run!

43
GooglePlayWebServiceAPI
GooglePlayWebServiceAPI BaseMax PHP

Tiny script to crawl information of a specific application in the Google play/store base on PHP.

43
scrapingant-client-python
scrapingant-client-python ScrapingAnt Python

ScrapingAnt API client for Python.

43
Android-Apps-Downloader
Android-Apps-Downloader harismuneer Python

📱 A utility for downloading Android apps from the Google Play Store and Xiaomi App Store (the Chinese App Store).

43
tse-client
tse-client m-ahmadi JavaScript

A client for fetching stock data from the Tehran Stock Exchange (TSETMC). Works in Browser, Node and as CLI.

43
BilibiliCrawler
BilibiliCrawler cgDeepLearn Python

:cyclone: crawl bilibili user info and video info for data analysis | BiliBili爬虫

43
SeleniumLogin
SeleniumLogin CharlesPikachu Python

Login some website using selenium.

43
wx-crawl
wx-crawl xuziping Java

微信公众号文章爬虫

43
tiktok-crawler
tiktok-crawler hackertogether Python

This is a Tiktok Crawler App.

43
Broken-Link-Crawler
Broken-Link-Crawler healeycodes Python

:robot: Python bot that crawls your website looking for dead stuff

43
seenreq
seenreq mike442144 JavaScript

Generate an object for testing if a request is sent, request is Mikeal's request.

43
copyheaders
copyheaders jin10086 Python

方便的从浏览器复制浏览器头

43
spiderable-middleware
spiderable-middleware veliovgroup JavaScript

Pre-rendering for JavaScript websites that delivers SSR-level SEO, enhanced link previews, and performance via effortless middleware integration — ide...

43