PostgreSQL WAL shipping: cross-region failover that survives a primary outage
SEO Meta Description: Master PostgreSQL WAL shipping for cross-region failover. Learn segment archiving vs streaming replication, replication slot hazards, lag monitoring with pg_stat_replication, and the promotion runbook that minimizes data loss.
Tags: backend api microservices architecture cleanarchitecture
TL;DR
WAL streaming replication beats segment archiving for RPO under one second, but replication slots will silently eat your disk if a standby goes dark. Monitor pg_stat_replication.write_lag religiously, cap slot retention with max_slot_wal_keep_size, and rehearse your promotion runbook quarterly — because the outage will happen at 3 AM on a Saturday.
Why this is harder than it looks
Most teams treat the standby as a passive mirror rather than as a first-class consumer of a data contract. Martin Fowler’s collaborators Simon Harrer and Arif Wider frame data contracts precisely this way — a dataset is a service, and its consumers depend on the guarantees it makes about structure, latency, and availability. Your standby replica is exactly that kind of consumer. When the primary violates its availability contract, the standby must be promoted without breaking downstream consumers’ expectations.
That framing changes how you architect your replication topology from day one.
Segment archiving vs. streaming replication
| Method | Typical RPO | Lag Visibility | Disk Pressure | Complexity |
|---|---|---|---|---|
| WAL Segment Archiving | Minutes (1 segment = 16 MB) | None (pull-based) | Archive storage only | Low |
| Streaming Replication | Sub-second | pg_stat_replication | Primary disk (slots) | Medium |
| Streaming + Archive fallback | Sub-second (normal), minutes (reconnect) | Full | Both | Medium-High |
Segment archiving ships completed 16 MB WAL files via archive_command. An idle primary may not rotate a segment for minutes — meaning your cross-region standby is perpetually 10–15 minutes behind during low-write periods. Streaming replication sends WAL records as they are written, keeping standby lag typically under 100 ms on a healthy inter-region link.
Production guidance: use streaming replication as the primary channel, with segment archiving as a fallback and for point-in-time recovery. Configure recovery_min_apply_delay only if you need a delayed standby for logical corruption protection.
The replication slot retention hazard
This is the footgun that has taken down production systems I have personally seen.
-- Check active slots and their retained WAL
SELECT slot_name, active, restart_lsn,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;
If your standby disconnects — network partition, maintenance, crash — a replication slot on the primary will hold every WAL segment from that point forward. On a write-heavy system, this means gigabytes per hour accumulating until the primary’s disk fills and PostgreSQL hard-stops accepting writes.
Fix it now:
# postgresql.conf
max_slot_wal_keep_size = 10GB # PostgreSQL 13+
This caps retained WAL per slot. The trade-off: if the standby falls behind more than this, it must be rebuilt from a base backup. That is acceptable. A primary with a full disk is not.
Monitoring standby lag with pg_stat_replication
In my experience, teams alert on replication lag in bytes but miss the latency dimensions that actually predict data loss:
SELECT client_addr,
state,
write_lag,
flush_lag,
replay_lag,
pg_size_pretty(pg_wal_lsn_diff(sent_lsn, replay_lsn)) AS total_lag_bytes
FROM pg_stat_replication;
Alert thresholds I use in production:
replay_lag > 30s→ warning (network degradation)replay_lag > 120s→ critical (standby falling behind)total_lag_bytes > 500MB→ critical (approach slot retention limit)
The promotion runbook that minimizes data loss
Step 1 — Confirm primary is truly unreachable, not just partitioned from your monitoring host. Use at least two independent probes (application health check + direct TCP) before proceeding.
Step 2 — Check standby recovery position:
psql -c "SELECT pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn(), pg_is_in_recovery();"
Step 3 — Promote the standby:
# PostgreSQL 12+
pg_ctl promote -D /var/lib/postgresql/data
# Or via signal file for older versions
touch /var/lib/postgresql/data/promote
Step 4 — Update connection routing (DNS, pgBouncer, or service mesh) to point to the new primary.
Step 5 — Fence the old primary. Before it comes back online, ensure it cannot accept writes — revoke network access or shut it down — to prevent split-brain.
The data loss window equals the gap between pg_last_wal_receive_lsn() and the primary’s last written LSN at failure time. With streaming replication and synchronous_standby_names set for at least FIRST 1, this window shrinks to zero at the cost of write latency.
Three things to do before the next outage finds you
-
Set
max_slot_wal_keep_sizein yourpostgresql.conftoday. An uncapped slot on a disconnected standby is a time bomb under your primary — it will fill the disk and hard-stop writes before you even get paged. -
Alert on
replay_lagin seconds, not just bytes. Lag in bytes hides the latency story. Time-based lag tells you exactly how much data you lose if the primary fails right now. -
Run the promotion runbook quarterly. Treat your standby as a first-class data contract consumer — test that promotion completes in under five minutes, that applications reconnect cleanly, and that the old primary can be safely fenced. Failovers that have never been practiced will surprise you in the worst possible moment.