Kubernetes Deployment Guide

Production EKS deployment of GPUForge V1

GPUForge V1 — Kubernetes Deployment Guide

Production deployment of GPUForge V1 on Amazon EKS using eksctl, the NVIDIA GPU Operator, and the DCGM exporter for metrics scraping. This is the canonical guide for customers who want Kubernetes-native scheduling instead of SLURM. Outputs a working EKS cluster with GPU nodegroups, the GPU Operator installed, and a CUDA smoke test that prints the canonical "GPU is wired correctly" signal.

Choosing a deployment target

This guide is the operational EKS walk-through — it assumes you've already picked EKS. For the strategic comparison of GKE vs EKS vs on-prem (pricing tiers, GPU availability, quota lead times, SLURM + K8s fit, air-gapped fit, egress), see /insights/gke-vs-eks-vs-on-prem-for-gpu-workloads and decide which surface your deployment belongs on before running §1–§11 below.

1. Overview & Scope

This guide covers end-to-end Kubernetes deployment of GPUForge workloads on EKS:

EKS cluster creation via eksctl (AWS CLI fallback included)
GPU nodegroups for g4dn (T4), p4d (A100), and p5 (H100) — on-demand and spot variants
NVIDIA GPU Operator Helm install pinned to v26.3.3 with EKS-tuned --set flags
DCGM exporter at 60s polling interval for Prometheus
containerd + CDI runtime configuration
CUDA smoke-test verification

In scope: EKS 1.29, us-east-2, AL2023 NVIDIA AMIs, customers with no mTLS/air-gap requirement.

Out of scope: air-gapped installs, OpenShift, on-prem Kubernetes, GCP/Azure GPU node types, k3s/k0s edge deployments, GPU time-slicing (see NVIDIA GPU Operator docs), MIG partitioning (see nvidia-cdi).

2. Cluster Creation via eksctl

2.1 eksctl (preferred)

eksctl create cluster \
  --name=gpuforge-v1 \
  --version=1.29 \
  --region=us-east-2 \
  --zones=us-east-2a,us-east-2b,us-east-2c \
  --vpc-private-subnets=10.0.1.0/24,10.0.2.0/24,10.0.3.0/24 \
  --vpc-public-subnets=10.0.101.0/24,10.0.102.0/24,10.0.103.0/24 \
  --node-ami-family amazon-linux-2023-x86-64-nvidia \
  --enable-iam \
  --with-oidc \
  --enable-cluster-logging="api,audit,authenticator,controllerManager,scheduler" \
  --tags="Project=gpuforge-v1,Env=prod"

2.2 AWS CLI fallback (when eksctl is unavailable)

# 1. Create VPC and subnets
aws ec2 create-vpc --cidr-block 10.0.0.0/16 \
  --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=gpuforge-v1}]'

aws ec2 create-subnet --vpc-id <vpc-id> --cidr-block 10.0.1.0/24 --availability-zone us-east-2a
aws ec2 create-subnet --vpc-id <vpc-id> --cidr-block 10.0.2.0/24 --availability-zone us-east-2b
aws ec2 create-subnet --vpc-id <vpc-id> --cidr-block 10.0.3.0/24 --availability-zone us-east-2c

# 2. Create the control plane
aws eks create-cluster \
  --name gpuforge-v1 \
  --region us-east-2 \
  --kubernetes-version 1.29 \
  --role-arn arn:aws:iam::<account-id>:role/gpuforge-eks-cluster \
  --resources-vpc-config subnetIds=<subnet-a>,<subnet-b>,<subnet-c>,securityGroupIds=<sg-id>

# 3. Update kubeconfig
aws eks update-kubeconfig --name gpuforge-v1 --region us-east-2

IAM OIDC provider is mandatory. Without it, the GPU Operator's ServiceAccount cannot assume AWS roles for NVLink topology discovery, and nvidia-smi topo -m returns empty instead of the NVLink/PCIe switch matrix.

3. GPU Nodegroups

Three nodegroups — g4dn-mng (general purpose, on-demand T4), p4d-spot (training, spot variant required), p5-mng (H100 on-demand with capacity-block pre-provisioning).

3.1 g4dn-mng — training/inference, on-demand, T4

Field	Value
Machine type	`g4dn.xlarge`
vCPU	4
RAM (GiB)	16
GPU model	NVIDIA T4
GPU count per node	1
EBS size (GiB)	125
On-demand $/hr	0.526
Spot $/hr floor	0.157

# g4dn-mng.yaml — apply with: eksctl create nodegroup -f g4dn-mng.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: gpuforge-v1
  region: us-east-2
nodeGroups:
  - name: g4dn-mng
    instanceType: g4dn.xlarge
    minSize: 1
    maxSize: 10
    desiredCapacity: 2
    amiFamily: amazon-linux-2023-x86-64-nvidia
    iam:
      withAddonPolicies:
        autoScaler: true
    ssh:
      allow: true
    tags:
      k8s.io/cluster-autoscaler/enabled: "true"

3.2 p4d-spot — training, spot variant

Field	Value
Machine type	`p4d.24xlarge`
vCPU	96
RAM (GiB)	1152
GPU model	NVIDIA A100 (40 GB)
GPU count per node	8
EBS size (GiB)	1000
On-demand $/hr	32.77
Spot $/hr floor	9.83

# p4d-spot.yaml — apply with: eksctl create nodegroup -f p4d-spot.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: gpuforge-v1
  region: us-east-2
nodeGroups:
  - name: p4d-spot
    instanceType: p4d.24xlarge
    minSize: 0
    maxSize: 8
    desiredCapacity: 1
    spot: true
    amiFamily: amazon-linux-2023-x86-64-nvidia
    tags:
      workload: training

3.3 p5-mng — H100 training, on-demand, capacity-block pre-provisioned

Field	Value
Machine type	`p5.48xlarge`
vCPU	192
RAM (GiB)	2048
GPU model	NVIDIA H100 (80 GB)
GPU count per node	8
EBS size (GiB)	1500
On-demand $/hr	98.32
Spot $/hr floor	n/a (capacity-block only)

# p5-mng.yaml — apply with: eksctl create nodegroup -f p5-mng.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: gpuforge-v1
  region: us-east-2
nodeGroups:
  - name: p5-mng
    instanceType: p5.48xlarge
    minSize: 0
    maxSize: 4
    desiredCapacity: 0
    amiFamily: amazon-linux-2023-x86-64-nvidia
    capacityReservation:
      capacityReservationId: <cr-h100-block-id>
    tags:
      workload: h100-training

4. GPU Operator Helm Install

Pinned to v26.3.3 (chart version nvidia/gpu-operator). The three flags below are mandatory and not interchangeable — see the rationale block immediately after the install command.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --version=v26.3.3 \
  --create-namespace \
  --namespace gpu-operator \
  --set driver.enabled=false \
  --set toolkit.enabled=false \
  --set node-ami-family amazon-linux-2023-x86-64-nvidia \
  --set dcgm.pollingInterval=60 \
  --set dcgmExporter.serviceMonitor.enabled=true \
  --wait

Why these flags are mandatory (kernel-panic crashloop rationale):

--node-ami-family amazon-linux-2023-x86-64-nvidia — the default AL2 GPU AMI ships kernel 5.10. AL2023 ships kernel 6.1. The 555+ NVIDIA driver line requires kernel 6.1; on kernel 5.10 the node shows Ready but kubelet enters CrashLoopBackOff after the kernel module load fails. There is no in-cluster workaround for this — the AMI must be AL2023 from the start.
--set driver.enabled=false --set toolkit.enabled=false — the AL2023 NVIDIA AMI already ships the NVIDIA driver and nvidia-container-toolkit pre-baked. Leaving the GPU Operator's driver/toolkit enabled triggers an in-place upgrade that breaks the kernel-module ABI against the AMI's pre-baked driver. The end state is the same kernel-panic crashloop as above, but it surfaces ~6 minutes after install rather than at boot.

If you observe node Ready / kubelet CrashLoopBackOff on a fresh EKS GPU nodegroup, the AMI family override was lost — usually because someone ran eksctl create nodegroup against an older base config. Re-create the nodegroup with --node-ami-family amazon-linux-2023-x86-64-nvidia.

5. DCGM Exporter

The GPU Operator bundles nvidia-dcgm-exporter as a DaemonSet by default. The install command in §4 already enables it at 60s polling. Override only if you want a different cadence.

# Verify the DaemonSet rolled out
kubectl -n gpu-operator rollout status ds/nvidia-dcgm-exporter

# Confirm the ServiceMonitor is created (Prometheus picks it up via label)
kubectl -n gpu-operator get servicemonitor

# Confirm metrics are actually scraped
kubectl -n gpu-operator port-forward ds/nvidia-dcgm-exporter 9400:9400 &
curl localhost:9400/metrics | head -20
# Expect lines starting with DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, etc.

DCGM scrape interval is 60 seconds (--set dcgm.pollingInterval=60). Don't shorten it below 15s — DCGM polls every GPU's telemetry registers, and a tight interval causes register contention on p4d.24xlarge (8 A100s) and p5.48xlarge (8 H100s).

6. Containerd + CDI

Required for K8s 1.28+ GPU time-slicing and MIG partitioning. Drop-in config enables CDI on containerd.

# /etc/containerd/config.toml drop-in on GPU nodes
sudo mkdir -p /etc/containerd/config.d
sudo tee /etc/containerd/config.d/cdi.toml > /dev/null <<'EOF'
[plugins."io.containerd.grpc.v1.cri"]
  enable_cdi = true
EOF

# Apply the runtime class registration
cat <<'EOF' | kubectl apply -f -
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia-cdi
handler: nvidia-cdi
overhead:
  podFixed:
    memory: "1Gi"
    cpu: "250m"
EOF

sudo systemctl restart containerd

Pods that need the CDI runtime declare runtimeClassName: nvidia-cdi in their spec. Plain GPU pods can continue using the default runtime class — both resolve to the same NVIDIA device injection under the hood.

7. kubectl + CUDA Smoke Test

The canonical pass/fail signal that the GPU + driver + runtime are wired correctly: run a CUDA nbody simulation in a pod and confirm it prints non-zero FPS.

# Spin up the CUDA sample pod
kubectl run cuda-smoke \
  --image=nvcr.io/nvidia/k8s/cuda-sample:nbody-cuda12.4.0 \
  --restart=Never \
  --limits=nvidia.com/gpu=1 \
  --overrides='{"spec":{"runtimeClassName":"nvidia-cdi"}}'

# Wait for it to run (Run → Completed takes ~30s on T4, ~12s on H100)
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/cuda-smoke --timeout=120s

# Inspect the nbody output — this is the pass signal
kubectl logs pod/cuda-smoke | grep "FPS"
# Expect:  "X devices, 0000:..." then "X Devices ... <n> GFLOP/s ... <m> billion particles/s"

# Clean up
kubectl delete pod cuda-smoke

A non-zero GFLOP/s print confirms:

The AL2023 NVIDIA AMI kernel module loaded (no crashloop).
The containerd CDI runtime injected the GPU into the pod.
The CUDA driver inside the container matched the host driver ABI.

If the pod hangs in ContainerCreating for >2 minutes, kubectl describe pod cuda-smoke will show the actual error — typically a missing nvidia.com/gpu resource on the node (driver not loaded) or an unmatched runtime class.

8. Machine-Type Spec & Pricing

Single source of truth for the supported instance types and their $/hr as of this guide's publication. Always re-verify against the EC2 pricing page before quoting customers — AWS prices change quarterly.

Machine type	vCPU	RAM (GiB)	GPU model	GPU per node	EBS (GiB)	On-demand $/hr	Spot $/hr floor
`g4dn.xlarge`	4	16	NVIDIA T4	1	125	0.526	0.157
`g4dn.12xlarge`	48	192	NVIDIA T4	4	900	4.352	1.31
`p4d.24xlarge`	96	1152	NVIDIA A100 (40 GB)	8	1000	32.77	9.83
`p4de.24xlarge`	96	1152	NVIDIA A100 (80 GB)	8	1000	40.96	12.29
`p5.48xlarge`	192	2048	NVIDIA H100 (80 GB)	8	1500	98.32	n/a

p5.48xlarge is not generally available on the spot market — capacity must be reserved via EC2 Capacity Blocks for ML.

9. Documented Pitfalls

Numbered list of pitfalls operators hit on a real GPUForge-on-EKS deployment. Pitfall #1 is the kernel-panic crashloop described in §4 — it is inlined here because every operator eventually hits it.

Kernel-panic crashloop from wrong AMI family. Default AL2 GPU AMI ships kernel 5.10; the 555+ NVIDIA driver requires kernel 6.1. Node goes Ready / kubelet CrashLoopBackOff. Fix: --node-ami-family amazon-linux-2023-x86-64-nvidia.
Driver/toolkit ABI mismatch. Leaving GPU Operator's driver.enabled=true on an AMI that ships the NVIDIA driver pre-baked triggers an in-place driver upgrade that breaks kernel-module ABI. Fix: --set driver.enabled=false --set toolkit.enabled=false.
DCGM scrape interval too aggressive. Polling <15s on p4d.24xlarge causes register contention; metrics drop to zero for ~30s then recover. Fix: --set dcgm.pollingInterval=60.
Missing IAM OIDC provider. GPU Operator's ServiceAccount cannot assume roles for NVLink topology discovery; nvidia-smi topo -m returns empty. Fix: pass --with-oidc to eksctl create cluster.
CDI not enabled on containerd. K8s 1.28+ GPU time-slicing pods fail to schedule with "no devices found". Fix: drop-in /etc/containerd/config.d/cdi.toml with enable_cdi = true and restart containerd.
RuntimeClass naming inconsistency. Operator's Helm chart installs RuntimeClass=nvidia while documentation often says nvidia-cdi. Pick one and use it verbatim in the pod spec.
p4d.24xlarge EBS too small. Default 100 GiB runs out within hours of training a model with checkpointing. Bump to ≥1 TiB on creation.
GPU Operator chart versions older than 24.x lack support for K8s 1.29 cgroup v2. Pin to v26.3.3 until a newer LTS is documented in this guide.
kubectl wait timeout too short for H100 nodes. p5.48xlarge AMI bring-up + driver load takes ~6 minutes. Default kubectl wait timeout of 30s will fail.
Spot interruption on p4d-spot silently drains the nodegroup. Toleration must include node.kubernetes.io/not-ready for ≥5 min, otherwise workloads thrash. Use the karpenter interruption handler instead of stock CA.
Capacity-block reservation ID lost on cluster recreation. The capacityReservation block in the nodegroup yaml is silently dropped if you re-create the cluster from scratch. Capture the cr-h100-block-id in 1Password or SSM before teardown.
DCGM exporter ServiceMonitor not picked up. Prometheuses configured to scrape by namespace label only will skip the gpu-operator namespace unless --set dcgmExporter.serviceMonitor.enabled=true is passed.

10. GAIE + llm-d Cache-Aware Inference Routing

Optional add-on for inference workloads. Full sub-section breakdown lives in docs/demo-walkthrough.md. The subsections that previously appeared here (10.1 Install Prereqs through 10.7 Pitfalls) have been folded into the demo walkthrough's Step 8/9 + the §9 pitfalls list and are out of scope for this guide; see the walkthrough for the canonical step sequence.

11. End-to-End Bash Recipe

Paste-ready. Save as eks-cluster-bootstrap.sh, chmod +x, run on a workstation with eksctl + helm + aws + kubectl authenticated.

#!/usr/bin/env bash
set -euo pipefail
trap 'echo "failed at line $LINENO" >&2' EXIT

# --- 0. Preconditions ---
: "${AWS_REGION:=us-east-2}"
: "${CLUSTER_NAME:=gpuforge-v1}"
: "${K8S_VERSION:=1.29}"
: "${AMI_FAMILY:=amazon-linux-2023-x86-64-nvidia}"

command -v eksctl >/dev/null  || { echo "eksctl missing"; exit 1; }
command -v helm    >/dev/null  || { echo "helm missing";    exit 1; }
command -v aws     >/dev/null  || { echo "aws cli missing"; exit 1; }
command -v kubectl >/dev/null  || { echo "kubectl missing"; exit 1; }

aws sts get-caller-identity >/dev/null

# --- 1. Cluster (§2) ---
eksctl create cluster \
  --name "${CLUSTER_NAME}" \
  --version "${K8S_VERSION}" \
  --region "${AWS_REGION}" \
  --zones "${AWS_REGION}a,${AWS_REGION}b,${AWS_REGION}c" \
  --vpc-private-subnets=10.0.1.0/24,10.0.2.0/24,10.0.3.0/24 \
  --vpc-public-subnets=10.0.101.0/24,10.0.102.0/24,10.0.103.0/24 \
  --node-ami-family amazon-linux-2023-x86-64-nvidia \
  --enable-iam \
  --with-oidc \
  --enable-cluster-logging="api,audit,authenticator,controllerManager,scheduler" \
  --tags="Project=gpuforge-v1,Env=prod"

aws eks update-kubeconfig --name "${CLUSTER_NAME}" --region "${AWS_REGION}"

# --- 2. GPU nodegroups (§3) ---
for ng in g4dn-mng p4d-spot p5-mng; do
  eksctl create nodegroup -f "${ng}.yaml"
done

# --- 3. GPU Operator + DCGM (§4 + §5) ---
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --version=v26.3.3 \
  --create-namespace \
  --namespace gpu-operator \
  --set driver.enabled=false \
  --set toolkit.enabled=false \
  --set node-ami-family amazon-linux-2023-x86-64-nvidia \
  --set dcgm.pollingInterval=60 \
  --set dcgmExporter.serviceMonitor.enabled=true \
  --wait

kubectl -n gpu-operator rollout status ds/nvidia-gpu-operator --timeout=600s
kubectl -n gpu-operator rollout status ds/nvidia-dcgm-exporter --timeout=300s

# --- 4. Containerd CDI + RuntimeClass (§6) ---
# (Run on each GPU node via SSH or via the AMI's user-data bootstrap.)
for node in $(kubectl get nodes -l node.kubernetes.io/instance-type -o name | cut -d/ -f2); do
  ssh "ec2-user@${node}" 'sudo mkdir -p /etc/containerd/config.d && \
    sudo tee /etc/containerd/config.d/cdi.toml > /dev/null <<EOF
[plugins."io.containerd.grpc.v1.cri"]
  enable_cdi = true
EOF
  sudo systemctl restart containerd'
done

cat <<'EOF' | kubectl apply -f -
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia-cdi
handler: nvidia-cdi
overhead:
  podFixed:
    memory: "1Gi"
    cpu: "250m"
EOF

# --- 5. CUDA smoke test (§7) ---
kubectl run cuda-smoke \
  --image=nvcr.io/nvidia/k8s/cuda-sample:nbody-cuda12.4.0 \
  --restart=Never \
  --limits=nvidia.com/gpu=1 \
  --overrides='{"spec":{"runtimeClassName":"nvidia-cdi"}}'

kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/cuda-smoke --timeout=600s

kubectl logs pod/cuda-smoke | grep -E "GFLOP|particles/s" \
  || { echo "CUDA smoke test did not report GFLOPs"; exit 1; }

kubectl delete pod cuda-smoke --wait=false

echo "GPUForge V1 EKS bootstrap complete. Cluster: ${CLUSTER_NAME} in ${AWS_REGION}."

If the smoke test prints GFLOPs without the trap firing, the cluster is production-ready. If the trap fires, $LINENO points at the exact failing step (cluster create, nodegroup create, GPU Operator install, containerd rollout, or smoke test).