Two production-grade serving frameworks, two different bets: Ray Serve's Python-native actor system gives you full programmatic control over scaling and routing, while KServe's Kubernetes-native CRDs standardize model deployment around Knative revisions and the broader K8s ecosystem. This page distills the trade-offs across architecture, scaling granularity, GPU placement, router model, cache-aware routing, and K8s fit so platform teams can pick the right baseline for live LLM and embedding serving.
Architecture, scaling granularity, GPU placement, routing & request fanout, cache-aware routing, and K8s + ecosystem fit — sourced from each project's documentation and operational reports.
| Dimension | Ray Serve | KServe |
|---|---|---|
| Architecture | Python-native framework built on the Ray actor system. A Serve deployment is a class decorated with @serve.deployment and compiled into an actor pool where each replica is an actor with its own request queues; ingress is handled by a per-controller proxy_actor. State — model weights, prefix stores, KV caches — lives in Ray's distributed object store. No CRDs or YAML required to ship a service. |
Kubernetes-native model serving driven by an InferenceService custom resource. A prediction service wraps a predictor (and optional transformer / explainer) and is reconciled by the KServe controller. Knative Serving sits underneath for scale-to-zero, request-driven autoscaling, and revision routing. Model artifacts are pulled by a sidecar storage initializer from S3 / GCS / OCI registries. |
| Scaling granularity | Per-deployment replicas. Replicas are pure Python objects, so you can scale individual handlers inside one Serve application by writing a custom autoscaling_policy — queue depth, latency SLO, external metric, or any heuristic you can express in code. Sub-component fan-out (preprocessor, retriever, generator) lives in the actor pool and can be re-balanced independently. |
Per-InferenceService, concurrency-driven autoscaling via Knative metrics. Scale-to-zero and scale-from-zero are first-class; revision-level canary deployments are built in. Granularity is coarser than Ray Serve: every component in an InferenceService scales as a single revision unit, because Knative has no native notion of sub-component scaling. |
| GPU placement | Placement groups let you pin specific replicas to specific GPU IDs, NUMA nodes, or fractional GPUs. Multiple replicas can share one physical GPU via nvidia.com/gpu: <fraction> requests. The Ray scheduler handles pack / spread / strict-pack strategies. Runs equally well on bare metal or under kuberay-operator on K8s — the same Python code path either way. |
One whole GPU per Pod by default (the nvidia.com/gpu: 1 standard). Fractional GPU is technically possible but unusual in practice; community reference architectures assume 1 GPU per replica. The NVIDIA GPU Operator is required for K8s deployment. Multi-GPU copies are model-level (predictor replicas), not node-level. |
| Routing / request fanout | The proxy_actor load-balances across replicas with adaptive batching, and proxies to nested Deployments via a name registry. Ingress can be plain HTTP, gRPC, or a FastAPI app — handlers can compose serve.ingress decorators for arbitrary routing logic, prefix routing, or A/B fan-out implemented in pure Python. |
Knative route → revision → predictor. Canary and traffic split across revisions are first-class (Knative TrafficSplit), so A/B and shadow traffic at the model level are declarative YAML. Layered gateways are not in the base CRD: you wire Gateway API / Istio / Contour on top if you need richer routing semantics around the predictor. |
| Cache-aware routing | No out-of-the-box KV-aware router. Ray's distributed object store supports prefix-lookup primitives, so building a per-replica prefix-cache lookup router is a few hundred lines of Python — but you own the cost model, the staleness handling, and the eviction policy. Custom routing logic has full access to Ray's actor handles. | No built-in cache-aware routing in the core CRDs either. The standard add-on for production LLM serving is the Gateway API Inference Extension (GAIE) — InferencePool + an Endpoint Picker (EPP) that scores replicas on prefix locality, cost, or latency. GAIE pairs with llm-d for distributed KV-cache coordination across pods. Functionally richer than Ray Serve's defaults, but it is a separate stack to install and operate. |
| K8s + ecosystem fit | Runs on K8s via the Ray Operator (kuberay), but Serve is not aware of CRDs / Pods / revisions — its control plane is the Ray GCS. Easier to port from local dev to a cluster: the same Python code path serves in both. Ray's ecosystem (Ray Train, Ray Tune, Ray Data) is the integration story rather than the K8s ecosystem. | K8s-first: every component is a CRD, every rollout is a revision, every scale decision is a Knative metric. Tighter integration with Kubeflow, KServe Model Registry, and pipeline runtimes. K8s RBAC, NetworkPolicies, and PodDisruptionBudgets work against KServe out of the box — the trade-off is heavier install, more version drift across K8s minor releases, and a steeper on-ramp for teams new to Knative. |
Rather than picking one as the universal winner, the answer is shaped by where you sit on the trade-off curve between Pythonic programmatic control and Kubernetes-native declarative deployment.
You want Python-native iteration speed and full programmatic control of routing, autoscaling, and replica placement — without the K8s / Knative / CRD tax.
Your cluster is already Knative / K8s-native and standard InferenceService CRDs cover everything you need — model registry, revision rollback, declarative canary, scale-to-zero.
You want KServe's CRDs and revision lifecycle, plus production-grade cache-aware routing for LLM workloads. Layer Gateway API Inference Extension and llm-d on top.