A 13-year-old PostgreSQL bug, found by running Postgres on Kubernetes with CloudNativePG

Every so often, running Postgres differently teaches the project something it couldn't have learned any other way. This is one of those times.

A cascading standby reconnect bug that has been sitting in PostgreSQL's streaming replication code since version 9.3 (released in 2013) has just been fixed upstream last week. The fix will ship in the next round of minor releases across every supported branch: PostgreSQL 18.5, 17.11, 16.15, 15.19, and 14.24. Once those are out, equivalent releases of EDB Postgres Extended and EDB Postgres Advanced Server will follow. 

The bug was found, reported, and fixed by Marco Nenciarini, Esteemed Senior Principal Engineer at EDB, a Significant PostgreSQL Contributor, and one of the maintainers of CloudNativePG. The bug surfaced precisely because of the kind of architecture CloudNativePG makes trivial to build: a distributed, cascading, streaming-plus-archive PostgreSQL topology spanning multiple regions.

The bug

The fault sits in StartReplication(). Back in 2013, a commit taught streaming standbys to follow a timeline switch, and along with it added a sensible-looking guard: reject a START_REPLICATION request whose requested LSN sits ahead of the upstream's own WAL flush position. Fine for ordinary streaming.

It breaks down for a cascading standby that has just fallen back to archive recovery. Archive recovery consumes whole WAL segments, not individual records, so the moment a standby finishes replaying one segment from the archive, its next read position lands exactly on the following segment boundary: mechanically ahead of the upstream's still-in-flight flush pointer, even though nothing is actually wrong. The reconnect attempt fails outright with an error similar to this:
ERROR: requested starting point 0/A000000 is ahead of the WAL flush position of this server 0/9000000

And nothing retries. A cascading standby that has to fall back to the archive, even briefly, can be locked out of streaming from its upstream indefinitely.

The fix has the walreceiver check the upstream's IDENTIFY_SYSTEM-reported flush position before issuing START_REPLICATION. If the gap is at most one WAL segment (the exact shape of the gap that archive recovery's segment granularity produces), it waits and retries, bounded by wal_receiver_timeout, instead of giving up.

Why CloudNativePG found it

This bug requires a specific topology to trigger: a standby that streams from an upstream but can also fall back to WAL shipping, cascading to yet another node. That's not an exotic setup in the abstract, but building it, running it, and reliably reproducing a transient network hiccup on it, is a different matter entirely. Which is likely why it survived thirteen years of production PostgreSQL.

CloudNativePG's distributed topology turns exactly this architectural shape into a few lines of YAML. A replica cluster runs a designated primary in continuous recovery (via streaming, WAL archive, or both) and mirrors the primary cluster's architecture underneath it: its own local continuous backup target and its own set of replicas kept current through streaming with archive fallback. We call it symmetric on purpose, because the replica cluster is a first-class, promotable peer, not a bolted-on read replica.

Source: CloudNativePG documentation

Stack that on top of cascading replication and a hybrid streaming/archive continuous recovery strategy, and you get precisely the fault condition described above, on demand. You don't need a multi-region production estate to hit it, either. cnpg-playground, the official learning environment for CloudNativePG, reproduces this exact distributed topology with continuous backup via the Barman Cloud Plugin on Docker and Kind on a laptop.

Thirteen years is a long time not to notice

It's worth asking why this sat undiscovered for so long, because the answer says something about how the industry actually operates Postgres day-to-day.

Building a distributed topology across two or three regions, with continuous backup and recovery properly integrated into the replication fabric, has never been a trivial undertaking outside Kubernetes. Day-2 operations do get automated elsewhere, but tool by tool, each in its own realm: HA tooling handles failover and switchover, backup tooling handles backup and recovery, largely unaware of each other. The common pattern is to wire the two together, an HA tool that can call out to a backup tool to bootstrap a new replica, but they remain independently operated systems with no single control loop reconciling HA state with backup state.

That separation leaves room for an imperative escape hatch, a human dropping into a shell to untangle a situation neither tool was designed to resolve on its own.

CloudNativePG doesn't have that luxury and sets itself a harder target as a result: a single component responsible for both HA and backup/recovery coordination, covering day-2 operations declaratively rather than leaving them to be stitched together imperatively. HA is handled by the operator, which relies on Kubernetes primitives; backup and recovery are extensible through CNPG-I but are still coordinated by the same operator, not delegated outside it. That posture is what makes a distributed, cascading, hybrid-replication topology something you can declare and reproduce in minutes rather than architect from scratch, and it's exactly the kind of workload that exposes bugs like this one, because it actually gets exercised.

Outgrowing expectations

CloudNativePG has grown faster than anyone building it expected, and that's been a source of real satisfaction and joy for everyone involved in the project. It's organic growth too, no paid promotion behind it: close to 9,100 GitHub stars and 215 million downloads in the four years since the project was open-sourced.

That pace has a cost too, in the friction that inevitably comes with scaling this fast, but it's friction we're committed to smoothing out as we go, not a reason to slow down. You can see that commitment reflected in the project's own health metrics on the Linux Foundation's Insights dashboard. What matters is the overall trajectory: having joined the CNCF Sandbox in January 2025, CloudNativePG is now under evaluation by the CNCF Technical Oversight Committee for Incubation.

As I always say, CloudNativePG is pushing Postgres into boundaries it has never had to defend before, exploring territory the rest of the ecosystem simply hadn't reached yet, continuously observing, learning, adapting and improving. Looked at that way, finding a bug that's been sitting there for thirteen years isn't all that surprising.

Thanks where it's due

Bugs like this don't get fixed by one person. Thanks to the CloudNativePG team and community for sticking with an intermittent, hard-to-pin-down failure until it became a reliable, reproducible test case. Thanks to Marco Nenciarini for taking that case upstream and turning it into a proper PostgreSQL fix. Thanks to Xuneng Zhou for the review. And thanks to Álvaro Herrera for supervising the fix through the community process and committing it across every supported branch.

The fix has been committed upstream and is queued for the next minor releases: PostgreSQL 18.5, 17.11, 16.15, 15.19 and 14.24. See the original report and the commit for the full technical details.