Topic

crawler

Repositories (1501)

Web-crawler-engineer-for-Python
Web-crawler-engineer-for-Python zhangslob Python

Web-crawler-engineer-for-Python

43
TiebaScraper
TiebaScraper TiebaMeow Python

基于 aiotieba 的高性能百度贴吧异步爬虫工具

42
doogle
doogle safesploitOrg PHP

Doogle is a search engine and web crawler which can search indexed websites and images

42
Bayesian-Stock-Market-Sentiment
Bayesian-Stock-Market-Sentiment wangys96 Python

A stock market text sentiment analysis website. A股舆情分析, web-crawler, bayesian algorithm, SQL, django, data-visualization.

42
ronin-web
ronin-web ronin-rb Ruby

ronin-web is a collection of useful web helper methods and commands.

42
scalpel
scalpel lewoudar Python

A fast and powerful web scraping library

42
LuoguCrawler
LuoguCrawler himself65 Python

一个python爬虫来爬取洛谷各种信息

42
CrawlerSamples
CrawlerSamples VAllens C#

A small, practical C# sample that crawls a modern JavaScript-rendered page with Puppeteer Sharp, then parses and extracts structured data with AngleSh...

42
leboncoin-crawler
leboncoin-crawler rfussien HTML

Crawler for leboncoin.fr

42
ncrawler
ncrawler kant2002 C#

Web Crawler written in C#

42
webmagician-ui
webmagician-ui Jkanon TypeScript

An admin UI project for a configurable web crawler platform

42
dijnet-bot
dijnet-bot juzraai JavaScript

Az összes számlád még egy helyen :)

42
browserpaw
browserpaw GoldenLoaf24h TypeScript

BrowserClaw - Control your everyday Chrome browser from AI agents, without losing logins or focus

41
12306
12306 wan300 Python

🚂 基于 Python FastAPI + Vue 3 + Electron 开发的 12306 自动化抢票助手。支持自动登录、余票监控、订单提交及桌面化集成,提供直观的 GUI 管理界面。仅供学习...

41
LLM-Web-Crawler
LLM-Web-Crawler buildship-ai TypeScript

Web Scraper and Crawler for LLM Apps and AI Workflows with NoCode / LowCode. Plug and play with your own logic and customize it flexibly and scalably...

41
AutoTBOXDataSystem
AutoTBOXDataSystem DolorHunter Java

汽车TBOX数据采集及分析系统设计与实现

41
TikHub-API-Python-SDK-V2
TikHub-API-Python-SDK-V2 TikHub Python

TikHub-API-Python-SDK-V2

41
crawlee-one
crawlee-one JuroOravec TypeScript

Production-ready web scraping in a single function call. Built on Crawlee.

41
noscrape
noscrape schoenbergerb TypeScript

This repository is deprecated

41
crawel
crawel MrXujiang JavaScript

基于Apify+node+react搭建的有点意思的爬虫平台

41
medium-stat-box
medium-stat-box kylemocode TypeScript

Practical pinned gist which show your latest medium status 📌

41
UniversityRecruitment-sSurvey
UniversityRecruitment-sSurvey Maicius Python

用严肃的数据来回答“什么样的企业会到什么样的大学招聘”?

41
Spider
Spider xiantang Python

web crawler

41
HttpProxy
HttpProxy asche910 Java

JAVA实现的IP代理池,支持HTTP与HTTPS两种方式

41
ZUCC_ZhenFangHelper
ZUCC_ZhenFangHelper zhouzaihang Python

正方教务管理系统学生版的自动登录、选课、信息获取

41
crawler
crawler axetroy TypeScript

nodejs 爬虫框架. crawler framework for nodejs

41
php-crawler
php-crawler elboletaire PHP

:spider: A simple crawler (spider) writen in php just for fun, with zero dependencies

41
grab_beautiful_girls_pictures
grab_beautiful_girls_pictures cunxi1992 Python

抓取MM131美女写真图片,并将其保存至本地指定的文件夹中。

41
krisha.kz
krisha.kz andprov Python

Rental ad parser

40
weread-mp-fetcher
weread-mp-fetcher Pengyf04 JavaScript

通过微信读书抓取公众号最新文章。零依赖,复用你已登录的 Chrome,输出 mp.weixin 原文直链。

40
wayurls
wayurls alwalxed Go

CLI tool for fetching URLs from Wayback Machine, Common Crawl, and VirusTotal.

40
Raven
Raven Symbolexe Go

Raven is a powerful and customizable web crawler written in Go.

40
lolcrawler
lolcrawler jonaslejon Python

Headless web crawler for bugbounty and penetration-testing/redteaming

40
d00r
d00r CYB3RMX Python

Simple directory brute-force tool written with python.

40
Youtube_Scraper
Youtube_Scraper CriticalHunter Python

Scrape data about an entire Channel or just a Playlist, or get stats about your Own Watch History.

40
integrada.minhabiblioteca.com.br
integrada.minhabiblioteca.com.br tharyckgusmao JavaScript

Download de livros para PDF/EPUB - Integrada.minhabiblioteca / vitalsource

40
DeadPool
DeadPool Ryuchen Python

该项目是一个使用celery作为主体框架的爬虫应用,能够灵活的添加爬虫任务,并且同时运行多站点的爬虫工作,所有组件都能够原生支持规模并发和分布式,加上celery...

40
sponge
sponge spypunk Kotlin

sponge is a website crawler and links downloader command-line tool

40
TripAdvisor_crawler
TripAdvisor_crawler Tang-Li-Jen Python

Python Crawler: Scrape Data From Tripadvisor

40
Domainker
Domainker BitTheByte Python

BugBounty Tool

40
SpiderWho
SpiderWho lanrat Python

A very fast whois crawler

40
insecres
insecres kkomelin Go

A console tool that finds insecure resources on HTTPS sites

40
Crawler
Crawler taseikyo Python

:snake:A collection of simple Python crawlers.

40
selenium_facebook_scraper
selenium_facebook_scraper Mhmd-Hisham Python

A simple python3 script used to download a users's friend list from facebook.

40
laundry
laundry endquote JavaScript

Data laundering tools

40
ArticleSpider
ArticleSpider hackfengJam Python

Crawling zhihu, jobbole, lagou by Scrapy, and using Elasticsearch+Django to build a Search Engine website --- README_zh.md (including: implementation...

40
podcastcrawler
podcastcrawler podcastcrawler PHP

PHP library to find podcasts

40
fund_analysis
fund_analysis piginzoo Python

The project is about the Chinese funds, including crawler the data and analysis them.

39
figma-archives
figma-archives gridaco Python

Figma Files Scraper for Research & Studies

39
geckordp
geckordp jpramosi Python

A client implementation of Firefox DevTools over remote debug protocol in python

39