Record thermal + sched + freq data for 60 seconds

TL;DR

On-device LLM inference on Android falls apart after 5-10 minutes because of thermal throttling. The kernel’s thermal HAL reduces CPU frequencies and gates GPU clocks long before your model finishes generating. I’ll walk through profiling this with Perfetto, using PowerHAL hints, and building an adaptive scheduler that monitors thermal zones to preemptively adjust batch size and thread count. The goal: keeping token generation consistent across 30-minute sessions instead of watching it crater by 60%.

---

The problem nobody benchmarks

Most on-device LLM benchmarks report peak tokens-per-second measured in the first 30 seconds. That number is useless for real workloads. In my experience building production systems, here’s what actually happens during a sustained session on a Snapdragon 8 Gen 3 device:

Time Elapsed	CPU Freq (Big Cores)	GPU Clock	Tokens/sec	Thermal Zone Temp
0-2 min	3.3 GHz	900 MHz	12.4 t/s	38°C
5 min	2.8 GHz	750 MHz	9.1 t/s	44°C
10 min	2.2 GHz	580 MHz	6.2 t/s	51°C
15 min	1.8 GHz	450 MHz	4.1 t/s	55°C
30 min	1.8 GHz	450 MHz	3.8 t/s	56°C

A 69% drop in throughput. Your users don’t experience your peak benchmark. They experience the thermally throttled floor.

How Android’s thermal framework fights you

Android’s thermal management stack operates in layers, and every one of them works against sustained inference:

1. The thermal HAL polls thermal zones (CPU, GPU, skin, battery) and reports severity levels (0-7) to the framework
2. Cooling devices (CPU freq scaling, GPU clock gating, charge rate reduction) activate at configured trip points
3. The kernel’s thermal governor applies the harshest mitigation. It doesn’t negotiate.

Here’s what matters: by the time thermal_zone0 crosses a trip point, the kernel enforces frequency capping immediately. No graceful degradation. Your inference thread goes from 3.3 GHz to 2.2 GHz in a single scheduling tick.

Profiling with Perfetto thermal tracks

Before building anything, you need visibility. Perfetto exposes thermal data through ftrace thermal events:

# Record thermal + sched + freq data for 60 seconds
perfetto -c - --txt <<EOF
buffers: { size_kb: 65536 }
data_sources: { config { name: "linux.ftrace" ftrace_config {
  ftrace_events: "thermal/thermal_temperature"
  ftrace_events: "power/cpu_frequency"
  ftrace_events: "power/gpu_frequency"
  ftrace_events: "sched/sched_switch"
}}}
duration_ms: 60000
EOF

In the Perfetto UI, overlay the thermal_temperature track with cpu_frequency. You’ll see the exact moment throttling kicks in and can identify which thermal zone triggers it.

The adaptive token generation pipeline

Most teams get this wrong because they try to fight the thermal governor. You can’t. The better strategy: degrade gracefully before the kernel forces catastrophic throttling.

The architecture has three components.

1. Thermal zone monitor

class ThermalMonitor(context: Context) {
    private val powerManager = context.getSystemService(PowerManager::class.java)

    fun getCurrentHeadroom(): Float {
        // Returns degrees of headroom before throttling
        // Available API 31+
        return powerManager.getThermalHeadroom(FORECAST_SECONDS) ?: Float.MAX_VALUE
    }

    fun getThermalStatus(): Int = powerManager.currentThermalStatus
}

PowerManager.getThermalHeadroom() (API 31+) is the key API. It returns the predicted thermal headroom in degrees over a forecast window. When this value drops below 5°C, throttling is imminent.

2. Adaptive parameter scheduler

data class InferenceParams(val threads: Int, val batchSize: Int)

fun computeParams(headroom: Float, status: Int): InferenceParams {
    return when {
        headroom > 12f -> InferenceParams(threads = 4, batchSize = 512)
        headroom > 7f  -> InferenceParams(threads = 3, batchSize = 256)
        headroom > 4f  -> InferenceParams(threads = 2, batchSize = 128)
        else           -> InferenceParams(threads = 1, batchSize = 64)
    }
}

The scheduler checks headroom every 2 seconds and adjusts before the kernel intervenes. Reducing threads from 4 to 2 cuts heat output significantly while only reducing throughput by roughly 30%. That’s far better than the 60%+ forced reduction the kernel will impose if you wait.

3. PowerHAL sustained performance hints

// Request sustained performance mode (API 31+)
val perfHintSession = performanceHintManager
    .createHintSession(threadIds, targetDurationNanos)
perfHintSession.reportActualWorkDuration(actualNanos)

PerformanceHintManager lets you signal to the PowerHAL that you prefer consistent clocks over peak clocks. The SoC vendor’s power firmware can then hold mid-range frequencies longer instead of boosting and crashing.

Results with the adaptive pipeline

Time Elapsed	Strategy	Tokens/sec	% of Peak Retained
0-2 min	Naive	12.4 t/s	100%
30 min	Naive	3.8 t/s	31%
0-2 min	Adaptive	10.1 t/s	100%
30 min	Adaptive	7.8 t/s	77%

You trade ~18% peak performance for 2x better sustained throughput at the 30-minute mark. 77% of peak retained versus 31%. For any session-based use case, the adaptive approach wins and it isn’t close.

Where this actually matters

On-device inference has real, specific product applications where cloud latency or data exfiltration are deal-breakers. Think about an offline chat assistant on a plane or in a rural area. It needs sustained multi-turn generation, not a 2-minute demo that melts the phone. Or a mobile IDE with on-device autocomplete that has to stay responsive across an entire dev session, not just the first few minutes.

The case I find most compelling is privacy-constrained document work. Legal briefs, medical records, financial filings. Sensitive text that can’t leave the device, and users who will throw 20-page documents at your model. They will hit the thermal wall.

In every one of these cases, solving sustained performance is the gap between a demo and a product.

What to do with this

Never trust peak benchmarks. Profile your on-device LLM with Perfetto for 30+ minutes. The sustained floor, not the peak, defines what your users actually feel.

Monitor getThermalHeadroom() instead of raw temperature. Reactive throttling is catastrophic. The PowerManager forecast API lets you stay ahead of the kernel’s blunt-force mitigations.

Trade peak for consistency. An adaptive pipeline that voluntarily reduces thread count and batch size before thermal trip points retains 77% of peak throughput at 30 minutes. The naive approach retains 31%. Predictable performance beats flashy benchmarks every time.