A Patroni cluster sitting idle is not the same thing as a Patroni cluster that works. The only way to verify failover behavior is to exercise it. The first opportunity came when I needed to install TimescaleDB and pgvector: both extensions require a shared_preload_libraries change, which means a restart, and a restart on a Patroni cluster means a controlled switchover. That is the forcing function that turned a theoretical HA setup into a tested one.
What followed was an after-action review of three problems, none of which were obvious before the restart began.
Finding 1: The pgvectorscale symlink was missing on both nodes
pgvectorscale ships as a versioned shared library. The file on disk was named vectorscale-0.9.0.so. PostgreSQL's extension loader resolves the library name from the extension's control file, which referenced vectorscale.so without the version suffix. The symlink connecting those two names did not exist on either node.
This would not have manifested during normal database operation. The extension was loaded on cluster startup, the primary came up fine, and nothing in the logs hinted at a problem. The symlink gap only matters when a node that has never been primary tries to load the extension for the first time, which is exactly what happens during a failover promotion. The replica would have crashed on startup.
The fix was straightforward: create the symlink, verify it resolves correctly, confirm both nodes match. But the lesson is that pre-flight for a failover drill needs to walk every extension dependency on the replica, not just the primary.
Finding 2: The replica's CPU model lacked AVX2 and FMA
pgvectorscale is compiled against specific instruction set extensions, AVX2 and FMA in particular. The replica was running a CPU model configuration that did not expose those instructions to the guest, even though the underlying hardware supported them. When the replica restarted and PostgreSQL tried to load the vectorscale shared library, the process received a SIGILL (illegal instruction signal) and terminated immediately.
Diagnosing this was not fast. The error message from PostgreSQL at that layer is not particularly descriptive, and the connection between "pgvectorscale fails to load" and "CPU model mismatch" is not obvious. The resolution was to change the virtual CPU model to expose the host's full instruction set to the guest, which eliminated the SIGILL.
The broader point: if you are running compiled extensions with hardware-specific optimizations, every member of a cluster needs to present an identical or compatible instruction set. A replica that shares a physical host type but differs in its CPU model configuration can silently fail to run code that the primary handles without issue.
Finding 3: A downstream app wrote to a read-only transaction for 35 minutes
After the controlled switchover completed, one downstream service continued writing to what had become the old primary. The old primary accepted the connection but rejected writes with a read-only transaction error. The service did not log these errors prominently, and nothing surfaced in an alert within the window when it would have been immediately obvious.
The environment variable pointing that service at its database host had not been updated to reflect the new primary's address. The service was talking to a host that was now a replica.
This is precisely the failure mode that a floating VIP endpoint solves. If every service connects through a stable DNS name that always resolves to the current primary (see the VIP anatomy note), there is no environment variable to update after a failover. The discovery here validated a decision that had already been made, but it would have been a real operational problem if the failover had been unplanned rather than controlled.
Finding 4: The auto-failback service had an HTTP 501 bug
After the controlled switchover, I ran the service responsible for re-promoting the original primary to primary status after the drill. It returned HTTP 501 against a Patroni API endpoint. The endpoint it was calling had been renamed in a recent Patroni release and the service had not been updated to match.
This was found here rather than in a real incident, which is the entire point of running drills. The fix was a one-line URL update, but if the drill had been triggered by an actual failure and the recovery path depended on that service, the 501 would have been a problem at the worst possible moment.
What the drill actually verified
The cluster failed over correctly. PostgreSQL on the replica promoted cleanly once the pre-flight issues were resolved, the timeline advanced as expected, and replication resumed when the old primary rejoined as a standby. None of the findings represent a fundamental design flaw in the HA setup.
What they represent is the gap between a cluster that is configured correctly and a cluster that has been exercised. The symlink issue, the CPU model mismatch, and the stale application endpoint were all invisible during normal operation. They only appeared under the specific conditions of a failover.
The same cluster later hosted the TimescaleDB extension incident described in the .so file post, which was a different class of extension problem but rooted in the same category of risk: compiled, non-standard extensions create failure modes that only materialize when PostgreSQL restarts or promotes and tries to load a binary that does not match what it expects.
A switchover drill is only as valuable as its pre-flight checklist. The checklist now includes: verify all extension symlinks on the replica, verify instruction set compatibility between all cluster members, confirm every client's connection target resolves through the stable endpoint, and exercise any automated recovery tooling before you need it.