Production LLM serving is a three-component problem, not a single-framework choice. KServe v0.15 owns the deployment + autoscaling surface, vLLM with prefix caching owns the request engine and per-replica KV reuse, and LMCache owns the shared KV tier that ties replicas together across the cluster. This page distills the trade-offs across architecture, scaling granularity, GPU placement, routing fanout, cache-aware routing, and K8s + ecosystem fit — and positions the trio against GPUForge V1's existing Ray / vLLM baseline, where "Ray clusters and vLLM inference servers can register with GPUForge as special cluster types" but no KV-cache layer ships out of the box.
Architecture, scaling granularity, GPU placement, routing & request fanout, cache-aware routing, and K8s + ecosystem fit — sourced from each project's documentation (KServe v0.15 release notes, vLLM prefix-caching docs, LMCache storage-backend reference) and from operational reports on production deployments.
| Dimension | KServe v0.15 | vLLM (with prefix caching) | LMCache |
|---|---|---|---|
| Architecture | Kubernetes-native model serving driven by an InferenceService custom resource. v0.15 stabilizes the GAIE-aware reconciliation loop and the new LLMInferenceService shape for serving LLMs end-to-end. Knative Serving sits underneath for scale-to-zero, request-driven autoscaling, and revision routing. KServe controls the deployment plane and model artifact lifecycle; it does not own the request engine. |
High-throughput LLM serving engine. A vLLM worker owns the model weights and the per-request KV-cache, exposes an OpenAI-compatible HTTP server, and runs continuous batching + PagedAttention. The v0.15-era prefix-caching feature hashes each incoming prompt's token prefix and reuses matching KV blocks in resident GPU memory before any new prefill work starts. vLLM is the request engine inside one replica — it has no opinion on the cluster. |
Distributed KV-cache layer that lives in front of (or alongside) vLLM. LMCache stores KV tensors across GPU, CPU (DRAM), and local- or blob-storage tiers, and serves store/retrieve calls from whichever tier can answer. It is BYO storage backend (Redis, InfiniStore, S3, GCS, NVMe) and a sidecar process to the vLLM worker, not a replacement for it. The cluster-shared KV tier is the cache-aware-routing primitive. |
| Scaling granularity | Per-InferenceService revision. Knative concurrency-driven autoscaling, scale-to-zero, and revision-level canary are all first-class. v0.15 adds per-pod queue-depth visibility that GAIE can score against, which closes a long-standing blind spot in the autoscaler. Granularity is coarser than vLLM's internal batching decisions — KServe decides how many pods; vLLM inside each pod decides request scheduling. |
Per-worker request-loop granularity. Continuous batching means each worker can have hundreds of in-flight requests and decide on a per-iteration basis which ones to swap, prefill, or decode. Prefix caching further amplifies throughput: a hit means the worker skips prefill entirely for the matching block range. There is no notion of "replica" inside a single vLLM process — scaling is the cluster scheduler's job, not vLLM's. | KV-tier pool granularity. An LMCache controller brokers keys across tiers and across workers; eviction is LRU + admission-policy-driven within the configured budget. The controller itself is stateless and scales horizontally, but the practical scaling lever is "how much KV you can afford to keep resident in GPU/CPU before falling back to disk" — not "how many vLLM workers you have." LMCache complements vLLM scaling instead of competing with it. |
| GPU placement | One whole GPU per Pod by default (the nvidia.com/gpu: 1 standard). Fractional GPU is technically possible but unusual. v0.15 keeps the standard 1-GPU-per-replica shape; multi-GPU copies are model-level (predictor replicas), not a tensor-parallel cluster. KServe will not re-shard a model across GPUs inside one pod — that is the inference engine's job. |
Supports both 1-GPU and multi-GPU shapes. Tensor parallelism, pipeline parallelism, and hybrid TP+PP are all first-class: vLLM shards a single model across N GPUs in one worker and exposes one OpenAI-shaped endpoint for the whole shard set. Continuous batching operates uniformly across the shard set. vLLM is the component that decides whether your 70B model lives on two, four, or eight GPUs inside one revision. | No direct GPU compute of its own — LMCache is a KV-cache store. GPUs are a tier (resident HBM/DDR in or near the vLLM worker), CPU DRAM is a tier, and local NVMe or object storage is the fallback tier. The placement question is "how much KV to keep where," which trades off per-token latency (more GPU = lower latency, less budget) against cluster-wide cache hit rate (more shared tier = higher hit rate, more egress). |
| Routing / request fanout | Knative route → revision → predictor, with v0.15 adding first-class GAIE endpoints. Canary and traffic split across revisions are declarative YAML. KServe owns fanout at the model level: which revision receives which percentage of traffic. Layered gateways (Istio, Contour, Gateway API) wire in if you need richer routing semantics around the predictor. | vLLM doesn't route — it serves. Every vLLM worker accepts requests from upstream load balancing (KServe's route, a Gateway API EPP, an LMCache lookup-aware proxy, or a vanilla L4 LB) and processes whatever reaches it. Continuous batching + prefix-cache hashes are internal to the worker; the worker advertises an OpenAI-shaped /v1/chat/completions endpoint and is otherwise opaque to the cluster. |
LMCache doesn't route either — it stores. The look-up is a KV-hash query against the tier pool, and the fanout decision happens one layer up (the EPP / Gateway API Inference Extension scores replicas on prefix locality, cost, or latency, then dispatches the request). LMCache's role is to make those replicas score as prefix-local when the answer lives in a shared tier rather than their own HBM. |
| Cache-aware routing | No built-in cache-aware routing in the core CRDs. v0.15 opens the door for GAIE — InferencePool + Endpoint Picker (EPP) that scores replicas on prefix locality, queue depth, and cost — so cache-awareness is a one-line wiring decision instead of a custom stack to install. KServe ships the orchestration surface that makes cache-aware routing composable. |
Built-in prefix caching in v0.15 and onward. vLLM hashes incoming prompt tokens and matches against the resident KV blocks in worker memory; on hit, prefill is skipped for that block range. Per-worker scope — the prefix cache is local to one vLLM process, so requests routed to the "wrong" replica will miss even if the same prefix was just served. Cache locality is routed, not computed, when more than one replica is in play. | The shared-KV tier. LMCache turns per-worker-local prefix caches into a cluster-wide prefix cache: a KV block computed by replica A can be retrieved by replica B on its first prefill. Combined with a prefix-locality-scoring EPP, the system can route a prompt to the replica that will hit the shared tier first, instead of each replica independently re-prefilling the same prefix. LMCache is the missing piece that turns vLLM's local prefix-cache into a cluster hit-rate lever. |
| K8s + ecosystem fit | K8s-first: every component is a CRD, every rollout is a revision, every scale decision is a Knative metric. v0.15 tightens integration with GAIE, the new LLMInferenceService shape, and the model registry. K8s RBAC, NetworkPolicies, and PodDisruptionBudgets work against KServe out of the box. Heaviest install of the three components, steepest on-ramp for teams new to Knative. |
Ecosystem-near: vLLM is a serving engine, not a K8s framework, but it slots into Ray Serve, KServe, and the HF Inference Endpoints ecosystem with minimal glue. The Hugging Face + Ray + vLLM loop (load → compile → serve) is the integration story. Runs as a Pod under kuberay, a KServe predictor, or a bare Deployment — same Python code path. K8s RBAC works against vLLM trivially because vLLM is just a container. |
BYO storage backend. LMCache itself is a process (a controller + sidecars to vLLM workers) so it inherits K8s semantics for free, but the storage configuration (Redis cluster, S3, GCS, InfiniStore) is an operational decision your platform team owns. No native CRDs — you wire the LMCache controller into your serving stack (KServe + GAIE, Ray Serve, or vanilla Deployments) the same way you wire any external dependency. |
Each piece of the stack solves a different bottleneck. The right adoption shape depends on whether your current pain is deployment lifecycle (KServe), per-worker prefill cost (vLLM), or cluster-wide cache hit rate (LMCache) — and whether your infra is already Knative-native or still GPUForge V1's Ray / vLLM baseline.
Your cluster is already Knative / K8s-native and standard InferenceService CRDs cover most of your deployment surface — you want v0.15's LLMInferenceService shape and GAIE wiring without committing to vLLM or a distributed KV tier yet.
You don't need a new orchestration layer — your existing serving stack (Ray Serve, HF Inference Endpoints, vanilla Deployments) already gives you a per-pod shape — and the binding constraint is prefill cost per request, not deployment lifecycle.
The three components compose into a production LLM stack that GPUForge V1's existing Ray / vLLM baseline does not provide on its own. KServe v0.15 owns deploy + GAIE routing, vLLM owns the per-worker engine, and LMCache owns the cluster-shared KV tier. Together they close the cache-aware-routing gap that the previous Ray Serve vs KServe hybrid pillar flagged as the missing layer.