Adaptive Bitrate Log Streaming in CI/CD: Chunked Transfer Encoding, Server-Sent Events, and the Backpressure Architecture That Makes 10GB Build Logs Browsable in Real Time

TL;DR

Streaming massive CI/CD build logs in real time is a backpressure problem, not a rendering problem. Most teams reach for WebSockets by default, but Server-Sent Events with chunked transfer encoding give you 90% of the capability at a fraction of the operational complexity. Pair that with a ring buffer on the server, virtual scrolling on the client, and an explicit backpressure signaling protocol, and you can browse 10GB build logs without your viewer ever exceeding 150MB of memory.

---

Build logs are unbounded streams

A typical Kotlin multiplatform CI build generates 50,000-200,000 lines of output. Monorepo Gradle builds with parallel task execution can push well past 500,000 lines. Naively loading that into a browser DOM will crash the tab.

In my experience building production systems that ingest this kind of output, the real failure mode isn’t the log size itself. It’s the mismatch between producer speed (the build) and consumer speed (the browser). Without backpressure, a fast build and a slow client become a cascading failure.

Delivery protocol tradeoffs

What most teams get wrong here: they jump straight to WebSockets without considering the operational cost.

Criteria	Chunked Transfer Encoding	Server-Sent Events (SSE)	WebSocket
Direction	Server -> Client	Server -> Client	Bidirectional
Auto-reconnect	No	Yes (built-in)	Manual
HTTP/2 multiplexing	Yes	Yes	No (upgrade required)
Load balancer support	Excellent	Excellent	Requires sticky sessions
Proxy/CDN compatibility	High	High	Medium
Memory overhead per conn	~2KB	~4KB	~8KB
Client complexity	Low	Low	Medium

For log streaming, a unidirectional, append-only data flow, SSE is the correct default. You get automatic reconnection with Last-Event-ID, native HTTP/2 multiplexing, and zero load balancer headaches. WebSockets only make sense if you need bidirectional control (e.g., sending filter queries upstream), and even then, a hybrid approach is simpler.

Server-side: ring buffers and bounded memory

The log ingestion server should never hold the full log in memory. Use a ring buffer with configurable capacity:

class LogRingBuffer(private val capacity: Int = 50_000) {
    private val buffer = ArrayDeque<LogLine>(capacity)
    private var globalOffset: Long = 0

    @Synchronized
    fun append(line: LogLine) {
        if (buffer.size >= capacity) {
            buffer.removeFirst()
            globalOffset++
        }
        buffer.addLast(line)
    }

    fun slice(from: Long, count: Int): List<LogLine> {
        val start = (from - globalOffset).coerceAtLeast(0).toInt()
        return buffer.drop(start).take(count)
    }
}

The ring buffer keeps the last N lines in memory while the full log streams to object storage (S3, GCS) in compressed chunks. Clients connecting mid-build get the tail from the ring buffer and can seek historically via range requests against the stored chunks, similar to how adaptive bitrate video streaming works with segment-based seeking.

Backpressure: the part everyone skips

This is where most log streaming implementations quietly fall apart. Without backpressure signaling, a slow consumer forces the server to buffer indefinitely, eventually OOM-killing the process. The protocol itself is straightforward:

1. The server tracks per-client send queue depth. If the queue exceeds a threshold (say, 1,000 unsent lines), the server switches that client to summary mode, sending one compacted message per N lines instead of every line.
2. The client signals readiness via SSE reconnection with a Last-Event-ID that encodes the last consumed offset.
3. When the client falls behind, the UI shows a “streaming paused, click to catch up” indicator rather than silently dropping lines. Honesty beats invisible data loss.

This is the same pattern used in reactive streams (Publisher/Subscriber with demand signaling), adapted for HTTP.

Client-side: virtual scrolling with anchor-based seek

Rendering 500K DOM nodes is not viable. Virtual scrolling renders only the visible viewport (typically 50-100 lines) plus a small overscan buffer. The key addition for log viewers is anchor-based seek: when a build fails, the client receives a failure-line offset from the server and jumps directly to it.

interface LogViewport {
  visibleStart: number;
  visibleEnd: number;
  anchorOffset: number | null; // jump target for failure line
  totalLines: number;
}

This keeps client memory under 150MB regardless of total log size, because you’re only holding the viewport lines plus a small LRU cache of recently scrolled regions.

How the major players compare

Platform	Max log size before degradation	Streaming protocol	Virtual scrolling	Backpressure handling
GitHub Actions	~500K lines (truncates beyond)	Polling + chunked	No (full DOM)	Truncation
GitLab CI	~1M lines (archived beyond)	SSE	Partial	Archive offload
Buildkite	~10M lines	Chunked + WebSocket	Yes	Adaptive batching

Buildkite’s approach most closely mirrors the architecture described here, and it shows. Their log viewer stays responsive at scales where others truncate or crash.

What to actually do

Default to SSE over WebSockets for log streaming. You get auto-reconnect, HTTP/2 multiplexing, and trivial load balancer configuration. Only introduce WebSockets if you have a proven bidirectional requirement.

Implement explicit backpressure signaling. A ring buffer on the server and demand-based delivery per client prevent the slow-consumer cascade that kills most naive streaming architectures. Budget 50K lines in the ring buffer and stream the rest to object storage.

Virtual scrolling is non-negotiable at scale. Any log viewer rendering more than a few thousand DOM nodes will degrade. Combine virtual scrolling with anchor-based seek so developers land directly on the failure line. That’s the UX that actually saves time during incident response.

“Just show me the logs” sounds simple. It isn’t. It’s a distributed systems problem wearing a friendly face, and the sooner you treat it like one, the less time you’ll spend debugging your debugging tools.