CDC without Debezium: Postgres WAL to Kafka in Go

Meta description: Skip the JVM overhead. Learn how to wire Postgres WAL directly to Kafka using pg_recvlogical and pgoutput — with slot management, LSN checkpointing, and production-safe failure recovery.

Tags: backend microservices api architecture devops

---

TL;DR

Debezium is the industry default for Change Data Capture, but it carries a heavy operational tax: JVM footprint, Kafka Connect infrastructure, and a connector model that’s overkill for focused use cases. By connecting directly to Postgres logical replication via pg_recvlogical and the pgoutput plugin, you can build a lightweight CDC connector in Go or Kotlin that publishes row-level change events to Kafka or Redis Streams, with full control over slot management, LSN checkpointing, and schema evolution.

---

Why roll your own CDC?

Most teams reach for Debezium without realizing they’re signing up for an entire platform, not a library. When your requirements are scoped — single database, a handful of tables, a specific event schema — the operational cost of Kafka Connect clusters and JVM tuning quickly outweighs the benefit.

The numbers:

Approach	Memory footprint	Operational dependencies	Startup time
Debezium on Kafka Connect	512MB–2GB JVM	Kafka Connect cluster + ZooKeeper	10–30s
DIY Go connector (pg_recvlogical)	20–60MB	Only Postgres + Kafka/Redis	<1s
DIY Kotlin connector (coroutines)	80–150MB	Only Postgres + Kafka/Redis	2–4s

For teams running lean infrastructure, or microservices where one service owns one schema, the DIY path is legitimate production engineering — not premature optimization.

---

The architecture: WAL to event bus

Postgres logical replication exposes row-level changes through replication slots. The pgoutput plugin (built into Postgres 10+) serializes these changes into a binary protocol your connector reads over a standard replication connection.

Postgres WAL
    │
    ▼
Replication Slot (pgoutput)
    │
    ▼
pg_recvlogical / libpq replication protocol
    │
    ▼
Connector (Go or Kotlin)
    │
    ├──► Kafka Topic (per-table or unified)
    └──► Redis Streams (low-latency path)

Creating the replication slot

SELECT pg_create_logical_replication_slot(
  'my_cdc_slot',
  'pgoutput'
);

Create a publication for the tables you care about:

CREATE PUBLICATION my_pub FOR TABLE orders, inventory;

Reading changes in Go

conn, _ := pgconn.Connect(ctx, os.Getenv("DATABASE_URL"))

sysident, _ := pglogrepl.IdentifySystem(ctx, conn)
log.Printf("System ID: %s, LSN: %s", sysident.SystemID, sysident.XLogPos)

err := pglogrepl.StartReplication(ctx, conn, "my_cdc_slot", sysident.XLogPos,
    pglogrepl.StartReplicationOptions{
        PluginArgs: []string{
            "proto_version '1'",
            "publication_names 'my_pub'",
        },
    })

---

LSN checkpointing: the detail that makes or breaks you

In my experience, this is where DIY CDC connectors fail in production. The Log Sequence Number is your position in the WAL stream. If your connector crashes without confirming its LSN, it replays events from the last confirmed position — meaning your consumers must handle duplicates, or you implement idempotent writes downstream.

Checkpoint only after the event is durably written to Kafka:

// Confirm LSN only after successful Kafka produce + flush
err = producer.Flush(5000) // wait for broker ack
if err == nil {
    pglogrepl.SendStandbyStatusUpdate(ctx, conn,
        pglogrepl.StandbyStatusUpdate{WALWritePosition: currentLSN})
}

Never checkpoint speculatively. At-least-once delivery is the contract; your downstream systems must be designed for it.

---

Schema evolution without breaking consumers

pgoutput sends column data by position, not by name. When you ALTER TABLE ADD COLUMN, new columns appear at the end and existing offsets are preserved — safe. ALTER TABLE DROP COLUMN or reordering columns will break your decoder.

The fix: maintain a local schema cache keyed by relation OID. When a RelationMessage arrives, diff it against your cache and emit a schema-change event before the data event. Consumers that care about schema can pause and migrate; consumers that don’t can ignore it.

---

Failure recovery

If your connector crashes, resume from the last confirmed LSN. The replication slot holds WAL until you confirm, so nothing is lost.

Kafka going down is trickier. Buffer events locally — a SQLite file or Redis list works — and don’t confirm LSN until Kafka recovers. Don’t lose your position just because the broker is temporarily unavailable.

The hardest case is Postgres failover. Replication slots don’t survive a primary switch by default. Either use the pg_failover_slots extension or rebuild the slot on the new primary and resume from your last checkpointed LSN. Plan for this before it happens in production.

---

Wrapping up

A few things I’d have wanted someone to tell me before building this the first time:

Use pgoutput over wal2json. It’s binary, faster, and built into Postgres without any extension installation. For Go, pglogrepl wraps the protocol cleanly.

Checkpoint after the durable write, never before. This is the single rule that separates a connector you can trust from one that silently loses data. At-least-once delivery is acceptable; data loss is not.

Talk to your DBA team about schema evolution before the first ALTER TABLE hits production. Cache relation OIDs, emit schema-change events, and get explicit agreement on what “column stability” means for your tables. Much easier to sort out upfront than to retrofit.