zxuhan

zxuhan

👤 Developer

1 repository on SrcLog

View on GitHub
1 Repos
3 Stars
0 Forks
0 Watchers

Repositories (1)

llm-router zxuhan/llm-router Go

Cache-aware router for OpenAI-compatible LLM servers, in Go. Per-worker radix trees route each request to the worker holding its KV prefix. Validated on 4x A100 + vLLM and Apple Silicon + llama.cpp.

3