KV-cache poisoning: securing multi-tenant LLM APIs

Meta description: Shared KV-cache pools create cross-tenant leakage risks in LLM APIs. Enforce isolation with cache namespacing, eviction policies, and gRPC interceptors.

---

TL;DR

Semantic caches and shared KV-cache pools in multi-tenant LLM deployments are a largely underappreciated attack surface. Without tenant namespacing, eviction isolation, and prompt sanitization at the gateway layer, one tenant’s inference context can bleed into another’s — through cache poisoning or prompt injection. The gRPC interceptor pattern solves this without meaningful latency overhead. Most teams get this wrong. Here’s what to fix before an incident forces the issue.

---

The problem nobody talks about until it’s too late

The KV-cache — the key-value store that enables efficient attention computation by reusing prior token representations — is a performance primitive. In single-tenant deployments, it’s safe by construction. In a shared, multi-tenant inference cluster, it becomes a liability.

When inference servers share a KV-cache pool without isolation:

• A cached prefix from Tenant A can be matched and served to Tenant B when their prompts share a common structure (a shared system prompt template, for example).
• A malicious actor can craft prompts to populate the cache with poisoned completions that influence subsequent outputs for other tenants.
• Semantic similarity-based caches compound this further — a tenant can retrieve completions generated for a semantically similar but legally distinct query belonging to a different tenant.

In a system serving dozens of tenants with overlapping prompt structures, unintentional prefix reuse isn’t a theoretical edge case. It’s an operational expectation without explicit namespacing.

---

Attack vector taxonomy

These are the leakage paths that matter in practice:

Vector	Mechanism	Severity	Mitigation
Exact prefix cache hit	Shared system prompt matched across tenants	High	Tenant-scoped cache key namespace
Semantic cache collision	Embedding proximity triggers wrong cache entry	High	Per-tenant semantic index partition
Prompt injection via cache	Malicious completion stored, later retrieved	Critical	Sanitization + signed cache entries
Eviction side-channel	Timing analysis of hit/miss to infer tenant activity	Medium	Uniform response timing + noise injection

---

Cache key namespacing: the non-negotiable baseline

The minimum viable fix is namespacing every cache key with a cryptographically derived tenant identifier. Not optional — it’s the foundation everything else builds on.

import hashlib

def build_cache_key(tenant_id: str, prompt: str, model_id: str) -> str:
    tenant_hash = hashlib.sha256(tenant_id.encode()).hexdigest()[:16]
    prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()
    return f"{tenant_hash}:{model_id}:{prompt_hash}"

For semantic caches, this means maintaining a separate embedding index per tenant — not a shared index with a tenant filter. A shared index with a WHERE tenant_id = X clause is still vulnerable to timing attacks and implementation bugs. Partition at the storage layer, not the query layer.

---

Tenant-scoped eviction policies

Eviction is where most implementations break down. A global LRU cache evicts based on system-wide recency, which creates two distinct problems:

1. A high-traffic tenant can evict entries belonging to a low-traffic tenant — a soft denial-of-service vector.
2. A tenant’s cached completions may persist beyond their contractual data retention window.

The correct model is per-tenant eviction quotas with TTL enforcement tied to each tenant’s data policy:

cache_policy:
  tenant_a:
    max_entries: 10000
    ttl_seconds: 3600
    eviction: lru
  tenant_b:
    max_entries: 5000
    ttl_seconds: 900   # stricter retention SLA
    eviction: lru

In practice, this maps to logical keyspace partitions in Redis with per-prefix SCAN-based TTL enforcement, or dedicated keyspaces with eviction policies set per-tenant via CONFIG SET.

---

Prompt sanitization at the gateway layer

Before a request reaches the inference server, route it through a sanitization layer at the gateway. Three reasons this is the right place for it:

1. It’s centralized — no per-service duplication of sanitization logic.
2. It runs before any caching, so poisoned inputs never populate the cache.
3. It can be updated independently of model deployment cycles.

Minimum gateway sanitization checklist:

• Strip sequences attempting to override system prompt context (Ignore all previous instructions...)
• Validate that user-injected content cannot escape its designated role boundary
• Enforce per-tenant prompt length limits
• Log and alert on structures matching known injection signatures

---

The gRPC interceptor pattern for zero-latency enforcement

In my experience building production systems, the cleanest enforcement mechanism is a gRPC server interceptor that handles tenant identity propagation, cache key construction, and policy enforcement in a single auditable layer:

class TenantIsolationInterceptor : ServerInterceptor {
    override fun <Req, Resp> interceptCall(
        call: ServerCall<Req, Resp>,
        headers: Metadata,
        next: ServerCallHandler<Req, Resp>
    ): ServerCall.Listener<Req> {
        val tenantId = headers.get(TENANT_ID_KEY)
            ?: throw StatusRuntimeException(Status.UNAUTHENTICATED)

        val context = Context.current()
            .withValue(TENANT_CONTEXT_KEY, TenantContext(tenantId))

        return Contexts.interceptCall(context, call, headers, next)
    }
}

The tenant context flows through the entire request pipeline — cache key construction, eviction policy lookup, audit logging — without requiring individual services to re-authenticate. The latency overhead for this interceptor chain runs under 1ms at p99, which is negligible against inference latency.

---

Before you onboard your second tenant

Namespace at the storage layer on day one. Cache keys, semantic indexes, eviction quotas — all must be scoped to a cryptographically derived tenant identifier before you onboard your second customer. This isn’t a post-incident retrofit; it’s a day-one architecture requirement.

Sanitize at the gateway, not inside the model. Prompt injection defenses belong in infrastructure, not prompting strategies. A gateway interceptor gives you centralized enforcement, auditability, and the ability to update defenses without touching model deployments.

Treat cache isolation as a compliance obligation. Shared KV-cache pools without per-tenant eviction controls can violate data retention SLAs and cross-tenant confidentiality guarantees. Map cache TTL policies directly to tenant data agreements and enforce them at the infrastructure layer — before your legal team asks why you didn’t.

---

Tags: api backend microservices architecture grpc