gRPC for mobile: deadlines, channel pools, and retry budgets
Meta description: Master gRPC for mobile clients: propagate deadlines through your service mesh, safely pool channels on Android and iOS, and cap retry traffic before a thundering herd takes down your backend.
Tags: grpc mobile android ios backend api
TL;DR
Naive gRPC usage on mobile kills your backend under degraded network conditions. Cascading deadlines, bounded channel pools, and a fleet-level retry budget are what prevent this. Skip any one of them and you will learn the lesson in production.
How 400,000 retries took down a backend for 18 minutes
At 11pm on a Tuesday, a stadium Wi-Fi handoff triggered 400,000 simultaneous retries. The backend was down for 18 minutes. The root cause was three missing lines of config.
The protocol was gRPC — adopted for its type safety and streaming capabilities, then misused at the network boundary. The defaults are not tuned for mobile realities: high latency variance, aggressive TCP resets, and hundreds of thousands of clients that lose connectivity simultaneously.
Most teams treat gRPC like a slightly faster REST call. It isn’t. The deadline, retry, and connection models are fundamentally different, and each interacts with the others in ways that only surface under load.
Deadline propagation: cascade or die
A deadline is not a timeout. A timeout fires client-side and abandons the call. A deadline propagates through every hop of your service mesh via the grpc-timeout header, allowing upstream services to abandon work that will never reach the client anyway.
// Android — set a 2.5s deadline on the stub, not the channel
val response = stub
.withDeadlineAfter(2500, TimeUnit.MILLISECONDS)
.getHealthData(request)Your server deadline must be shorter than your client deadline. If the client gives up at 3s but the backend deadline is 5s, you burn CPU on work that will be discarded.
| Layer | Deadline | Why |
|---|---|---|
| Mobile client | 3000ms | User-facing tolerance |
| API gateway | 2700ms | Buffer for network RTT |
| Backend service | 2400ms | Buffer for internal hops |
| Database query | 1800ms | Leave room for retries |
A 300ms cascade buffer at each hop is conservative. On low-latency internal networks, 150ms is usually enough.
Channel pooling: size it or leak it
gRPC multiplexes requests over a single HTTP/2 connection per channel. On mobile, one channel per backend endpoint is usually correct — but teams reach for pools the moment they see head-of-line blocking under load.
Before you pool, understand the tradeoff: each channel holds a TCP connection, negotiates TLS, and maintains a flow control window. On Android, an unclosed channel will hold a wake lock. On iOS, it will fight with the system’s network interface scheduler.
A practical ceiling for mobile channel pools:
// iOS — pool of 2 channels for the primary API host
let pool = GRPCChannelPool.with(
target: .host("api.example.com", port: 443),
transportSecurity: .tls(GRPCTLSConfiguration.makeClientDefault()),
eventLoopGroup: PlatformSupport.makeEventLoopGroup(loopCount: 2)
)Two channels handle ~200 concurrent streams comfortably. Going beyond 4 per host rarely improves throughput and meaningfully increases your server-side connection count — a constraint that bites hard when you’re running behind a service mesh with connection limits per replica. Always close channels in onDestroy / deinit.
The retry budget: two mechanisms, not one
This is where most implementations break down. There are two distinct concepts here, and conflating them is exactly how you end up with an 18-minute outage.
The client-side retryPolicy (gRPC service config) controls how an individual client retries a failed call:
// gRPC service config — per-call retry behavior, client-side only
val serviceConfig = """{
"retryPolicy": {
"maxAttempts": 3,
"initialBackoff": "0.5s",
"maxBackoff": "10s",
"backoffMultiplier": 2,
"retryableStatusCodes": ["UNAVAILABLE"]
}
}"""This limits retries per call with exponential backoff. It does not limit total retry traffic across your fleet. If 400,000 clients each retry 3 times simultaneously, you have multiplied your load by 3x at the worst possible moment.
Fleet-wide retry budgets require server-side enforcement. Two practical approaches:
- Gateway-level rate limiting: track the ratio of retries (identifiable via the
grpc-previous-rpc-attemptsheader) to total requests and shed excess retry traffic above a target threshold, typically 10% of total RPC volume. - Coordinated client token buckets: clients draw from a shared retry quota distributed via remote config or a lightweight sidecar. More complex, but removes the gateway from the hot path.
The math is simple: if 1% of requests fail at baseline and each client retries up to 3 times, your effective traffic multiplier under a partial outage is 3x with no fleet-level enforcement. With a 10% retry budget at the gateway, that multiplier is capped at 1.1x regardless of fleet size.
Exponential backoff with jitter is necessary. It is not sufficient. The service config governs per-call behavior; the gateway budget is what actually stops the herd.
Three takeaways
-
Set deadlines on the stub, not the channel. Cascade them 10-15% tighter at each service layer so upstream services abandon work before the client times out.
-
Cap your channel pool at 2-4 per host on mobile. Measure server-side connection count before scaling up, and always close channels in
onDestroy/deinit. -
Combine client-side retryPolicy with a server-side retry budget at your gateway. The config handles per-call behavior; the gateway enforces the fleet-wide ceiling that prevents thundering herds.