Kubernetes ML sidecars with MIG partitioning: 40–60% inference cost reduction

Meta description: Deploy ML models as Kubernetes sidecar containers with MIG partitioning on A100s and Triton over gRPC to cut inference costs 40% without sacrificing latency.

Tags: kubernetes docker grpc backend microservices

---

TL;DR

Running a dedicated GPU node per inference service is expensive and wasteful. Co-locating ML models as sidecar containers alongside application pods — using MIG partitioning on A100s, Kubernetes device plugins, and Triton Inference Server over gRPC — reduces GPU idle time dramatically and cuts inference infrastructure costs by 40–60% depending on MIG slice configuration.

---

The problem with dedicated GPU nodes

Most teams get inference infrastructure wrong in the same way: they treat GPU nodes like CPU-bound services — one workload, one node. The result is GPU utilization averaging 20–35% across a fleet, while the billing meter runs at 100%.

In my experience building production systems, the breaking point comes when your inference traffic is bursty and co-located with the application generating the requests. You end up paying for full GPU capacity to serve peak load, while burning money during troughs.

The sidecar pattern changes that calculus entirely.

---

The architecture: sidecar inference co-location

Instead of routing inference traffic to a standalone service across the network, you deploy Triton Inference Server as a sidecar container within the same Kubernetes pod as your application. The app talks to Triton over localhost gRPC — sub-millisecond transport latency, no service mesh overhead.

spec:
  initContainers:
    - name: model-sync
      image: amazon/aws-cli:latest
      command: ["aws", "s3", "sync", "s3://my-model-bucket/", "/models/"]
      volumeMounts:
        - name: model-store
          mountPath: /models
  containers:
    - name: app
      image: myapp:latest
      resources:
        requests:
          cpu: "2"
          memory: "4Gi"
    - name: triton
      image: nvcr.io/nvidia/tritonserver:24.01-py3
      args: ["tritonserver", "--model-repository=/models", "--grpc-port=8001"]
      resources:
        requests:
          nvidia.com/mig-2g.20gb: "1"
        limits:
          nvidia.com/mig-2g.20gb: "1"
      volumeMounts:
        - name: model-store
          mountPath: /models
  volumes:
    - name: model-store
      emptyDir: {}

Models reach /models via an init container that syncs from S3 before Triton starts, so the model repository is populated before inference begins. A PVC backed by shared storage or an S3 FUSE mount are both viable alternatives depending on model size and cold-start tolerance.

The transport latency difference is real: gRPC over loopback consistently delivers p99 latencies under 1ms, versus 4–12ms for cross-node calls depending on your CNI.

---

GPU sharing with MIG partitioning

Running one Triton instance per pod doesn’t mean one full A100 per pod. NVIDIA’s Multi-Instance GPU (MIG) feature on A100s partitions a single physical GPU into up to seven isolated instances, each with dedicated memory and compute.

Before Kubernetes can schedule MIG slices, you configure partitions at the node level — either with nvidia-smi mig commands or the NVIDIA MIG Manager DaemonSet. Use the DaemonSet; it reconciles MIG configuration declaratively across your fleet and is far less error-prone at scale than per-node manual commands.

MIG Profile	GPU Memory	Compute %	Ideal For
1g.10gb	10 GB	~14%	Small classification models
2g.20gb	20 GB	~29%	Medium transformers
3g.40gb	40 GB	~43%	Large language models (7B)
7g.80gb	80 GB	100%	Full A100 allocation

The NVIDIA device plugin surfaces MIG slices as extended Kubernetes resources. The scheduler treats each slice as a discrete allocatable unit, which is what makes bin-packing multiple inference sidecars onto a single A100 possible.

---

Kubernetes scheduling and readiness

Bin-packing alone isn’t enough. You need resilience at the scheduling layer. Use topology spread constraints to distribute pods across nodes, and apply the label consistently so the selector resolves correctly:

# Pod template labels
labels:
  app: inference-sidecar

# Topology spread on the pod spec
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: inference-sidecar

Pair this with Triton’s health endpoints to prevent premature traffic routing. Model load times for 7B-parameter models routinely run 45–90 seconds — the initialDelaySeconds on your readiness probe must account for this, or you’ll see silent inference failures at pod startup:

readinessProbe:
  httpGet:
    path: /v2/health/ready
    port: 8000
  initialDelaySeconds: 60
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /v2/health/live
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 10

---

Cost comparison: dedicated vs. sidecar pattern

Assuming a single A100 8x node at ~$8,200/month handling 2,000,000 inferences/month at baseline utilization:

Deployment Model	GPU Utilization	Effective Cost/Inference	vs. Dedicated
Dedicated GPU nodes	22% avg	$0.0041	baseline
Sidecar + MIG (3 slices)	67% avg	$0.0025	~40% lower
Sidecar + MIG (7 slices)	89% avg	$0.0016	~60% lower

Effective cost = node cost ÷ (utilization × throughput capacity). The 40% figure reflects a 3-slice 2g.20gb configuration — the practical starting point for most teams. Seven-slice configurations require homogeneous model sizes to avoid wasted slice capacity.

---

When this doesn’t work

The sidecar pattern isn’t universally applicable. Here are the scenarios that actually bite you in production.

Pod restarts are the biggest gotcha. When your application container crashes or rolls, Triton goes down with it. For models with 60–90 second load times, that’s a real availability hit during rolling updates and crash recovery. Dedicated inference services decouple these lifecycles entirely — if uptime is the priority, that decoupling might be worth the GPU overhead.

Host-level memory pressure is subtler. MIG provides memory isolation between slices, but CPU OOM events and driver contention can degrade GPU throughput indirectly. Monitor nvidia-smi dmon at the node level, not just pod-level metrics. This one catches people off guard.

Model size constraints are hard limits, not guidelines. A 2g.20gb slice caps you at 20GB GPU memory. A 13B parameter model in FP16 needs ~26GB — it won’t fit, full stop. Profile your model footprint before committing to a MIG profile; TensorRT INT8 quantization is the primary lever when you’re memory-constrained.

---

Where to start

1. Audit GPU utilization before touching anything. If your nodes average below 50%, you’re a candidate for MIG-based co-location. Pull CloudWatch or GKE GPU metrics and establish a real baseline first.

2. Start with 2g.20gb MIG profiles. They fit most mid-size transformer workloads, give you three slices per A100, and skip the operational complexity of 7-slice configurations. Tune from there once you have real utilization data.

3. Wire Triton over gRPC on localhost, not HTTP. Protocol Buffers serialization overhead is negligible compared to transport latency gains over loopback. Use the generated gRPC stubs and skip the REST layer entirely for intra-pod communication.