Wiring Ollama to Android: local LLM without the cloud

Meta description: Wire Android’s OkHttp to a local Ollama instance, handle SSE streaming, manage network changes, and build resilient ViewModels.

Tags: android kotlin architecture mobile api

---

TL;DR

Ollama exposes an OpenAI-compatible REST API. You can point an Android OkHttp client at a local network Ollama instance, stream tokens via Server-Sent Events (SSE), and build a ViewModel that degrades gracefully when the server is unreachable — all without shipping model weights on-device or paying per-token cloud costs. The tradeoff is latency sensitivity to Wi-Fi quality and zero offline fallback unless you layer in on-device inference.

---

Why local-network inference instead of on-device?

On-device inference with frameworks like MediaPipe LLM or ONNX Runtime sounds appealing until you hit the APK size wall. Shipping a 4-bit quantized 7B model adds 4–6 GB to your distribution, which is a non-starter for most consumer apps. Meanwhile, cloud API costs compound fast at scale.

The middle path: a developer workstation or home server running Ollama, reachable over the local Wi-Fi network. The model lives on hardware with real VRAM, and your Android client treats it like any other REST endpoint.

The common mistake is treating this as a standard request/response call rather than a streaming connection — then wondering why the UI feels frozen waiting for a 2,000-token response to complete.

---

Configuring OkHttp for the Ollama endpoint

Ollama’s OpenAI-compatible endpoint is /v1/chat/completions. Default port is 11434. Your base URL in development will look like http://192.168.1.x:11434.

val client = OkHttpClient.Builder()
    .connectTimeout(5, TimeUnit.SECONDS)
    .readTimeout(0, TimeUnit.MILLISECONDS) // critical for SSE — no read timeout
    .build()

readTimeout(0) is non-negotiable for streaming. A finite timeout will cut the connection mid-generation on longer responses.

Cleartext traffic gotcha: Android blocks plain HTTP by default on API 28+. Add android:usesCleartextTraffic="true" to your <application> tag in AndroidManifest.xml, or define a network security config that permits your local IP range. Skip this and you get a cryptic CLEARTEXT communication not permitted exception on first run.

---

Handling SSE token streaming

Ollama streams tokens as Server-Sent Events when you pass "stream": true. Each data: line is a JSON delta. OkHttp doesn’t have native SSE parsing, but you can implement it cleanly with a Flow:

fun streamCompletion(prompt: String): Flow<String> = callbackFlow {
    val body = buildJsonRequest(prompt, stream = true)
    val request = Request.Builder()
        .url("$baseUrl/v1/chat/completions")
        .post(body)
        .build()

    val call = client.newCall(request)
    call.enqueue(object : Callback {
        override fun onResponse(call: Call, response: Response) {
            response.body?.source()?.use { source ->
                while (!source.exhausted()) {
                    val line = source.readUtf8Line() ?: break
                    if (line.startsWith("data: ") && line != "data: [DONE]") {
                        val delta = parseDelta(line.removePrefix("data: "))
                        trySend(delta)
                    }
                }
            }
            close()
        }
        override fun onFailure(call: Call, e: IOException) = close(e)
    })
    awaitClose { call.cancel() }
}

The two helpers that make this runnable:

private fun buildJsonRequest(prompt: String, stream: Boolean): RequestBody {
    val json = JSONObject().apply {
        put("model", "llama3")
        put("stream", stream)
        put("messages", JSONArray().put(
            JSONObject().apply {
                put("role", "user")
                put("content", prompt)
            }
        ))
    }.toString()
    return json.toRequestBody("application/json".toMediaType())
}

private fun parseDelta(json: String): String {
    return try {
        JSONObject(json)
            .getJSONArray("choices")
            .getJSONObject(0)
            .getJSONObject("delta")
            .optString("content", "")
    } catch (e: JSONException) {
        ""
    }
}

Both use org.json, which ships with Android. No extra dependencies.

---

ViewModel architecture with graceful degradation

The ViewModel handles three states: server reachable, server unreachable, and mid-stream network loss. The catch block must cover IOException broadly, since mid-stream loss surfaces as SocketTimeoutException or a generic IOException — not just ConnectException:

class ChatViewModel(private val repo: OllamaRepository) : ViewModel() {

    private val _uiState = MutableStateFlow<ChatState>(ChatState.Idle)
    val uiState: StateFlow<ChatState> = _uiState.asStateFlow()

    fun send(prompt: String) {
        viewModelScope.launch {
            _uiState.value = ChatState.Streaming("")
            repo.streamCompletion(prompt)
                .catch { e ->
                    _uiState.value = when (e) {
                        is ConnectException -> ChatState.ServerUnreachable
                        is SocketTimeoutException -> ChatState.Error("Connection timed out mid-stream")
                        is IOException -> ChatState.Error("Network interrupted: ${e.message}")
                        else -> ChatState.Error(e.message ?: "Unknown error")
                    }
                }
                .collect { token ->
                    val current = (_uiState.value as? ChatState.Streaming)?.text ?: ""
                    _uiState.value = ChatState.Streaming(current + token)
                }
            if (_uiState.value is ChatState.Streaming) {
                _uiState.value = ChatState.Complete((_uiState.value as ChatState.Streaming).text)
            }
        }
    }
}

In ServerUnreachable, surface a UI nudge: “Local AI server not found — check that Ollama is running on your network.”

---

Managing network changes

Register a ConnectivityManager.NetworkCallback in your repository. The lifecycle is the part most snippets omit: register in init, unregister in close:

class OllamaRepository(
    private val client: OkHttpClient,
    private val connectivityManager: ConnectivityManager
) {
    private var activeCall: Call? = null

    private val networkCallback = object : NetworkCallback() {
        override fun onLost(network: Network) {
            activeCall?.cancel()
        }
    }

    init {
        connectivityManager.registerDefaultNetworkCallback(networkCallback)
    }

    fun close() {
        connectivityManager.unregisterNetworkCallback(networkCallback)
    }
}

Wire close() to ViewModel.onCleared() or your DI scope’s teardown. Without unregistration, you leak the callback and accumulate duplicate cancellations across configuration changes.

Don’t retry automatically on loss. Surface the interruption to the user. Silent retry loops on a lossy home network produce garbled partial responses.

---

Latency and memory tradeoffs

Approach	Time-to-first-token	Peak memory (Android)	Offline support
Cloud API (hosted endpoint)	300–800 ms	~2 MB	No
Ollama local network (Wi-Fi)	80–200 ms	~3 MB	No
On-device inference (quantized 3B)	1,500–4,000 ms	2,500–4,000 MB	Yes

Measured on a Pixel 7 over 5 GHz Wi-Fi against a local RTX 3090 server; your numbers will vary.

Local-network Ollama wins on latency versus cloud (no internet round-trip) and on memory versus on-device. It loses hard the moment the device leaves the local network — there’s no graceful middle ground there.

---

Before you commit to this architecture

Set readTimeout(0) on OkHttp and add usesCleartextTraffic to your manifest. These are the two most common first-run blockers and they’re easy to miss because neither produces an obvious error message pointing to the real cause.

Catch IOException broadly in your ViewModel, then discriminate. ConnectException means the server is down; SocketTimeoutException means the stream died mid-flight. They need different error messages and different recovery paths.

Benchmark first-token latency on your actual hardware before deciding this approach is worth it. A congested 2.4 GHz network can push latency past 500 ms, which erodes the whole reason to go local. If that’s your environment, a cloud API may actually be more consistent.