The outage started with a maintenance restart. Patroni completed the switchover cleanly. PostgreSQL came back up. And then every application that depended on TimescaleDB stopped working at the same time.
The error was the same across all twelve databases: could not access file "timescaledb-2.27.0-dev". Not one database. All twelve, simultaneously, on both nodes.
How we got here
The cluster was running a source-built TimescaleDB development build rather than the apt-packaged release. This was intentional: a specific fix had been needed that was not yet in a stable release, so the extension was compiled from source at commit ac4d4f6 and installed directly into the PostgreSQL library directory. The version string embedded in that build was 2.27.0-dev.
That non-standard installation was not prominently documented anywhere in the failover runbook. It lived in someone's memory and in the binary on disk.
The apt package for TimescaleDB on the system was 2.26.4. When Patroni restarted PostgreSQL, the startup sequence loaded shared_preload_libraries. PostgreSQL looked for timescaledb, found the apt-managed version at 2.26.4, and started successfully. Then, on connection, any database that had the extension enabled tried to load the version it was installed with: 2.27.0-dev. That file no longer took priority. The extension metadata in the database catalog said one version. The file PostgreSQL had loaded said another. Every affected database refused connections.
The blast radius was immediate: a smarthome database, a health data pipeline, the MTA dashboard, the weather radar, and several others all went dark at once. Any application backed by those databases stopped responding.
Why rebuilding was not straightforward
The fix was to rebuild the two split shared-object modules that TimescaleDB installs: the core module and the TSL (TimescaleDB License) module. Source builds of TimescaleDB split into these two .so files rather than one.
The rebuild had to target the exact PostgreSQL minor version already running on the cluster. This is where things got precise: PostgreSQL's internal ABI is not stable across minor versions. Between PG 18.3 and PG 18.4, a palloc function signature changed (palloc0_mul). Compiling against the wrong minor version produces a binary that links cleanly but crashes at runtime. The running version was 18.3. Rebuilding against 18.4 headers produced an incompatible binary.
Once the correct minor version was confirmed and the build was done, the recovery used an additive approach: the rebuilt .so files were dropped into the library directory without stopping the running PostgreSQL process. On the next connection that triggered an extension load, PostgreSQL found the 2.27.0-dev file, loaded it, and the database came up. No restart. No window of additional downtime.
All twelve databases recovered without data loss.
What the runbook was missing
This incident exposed a gap that is easy to create and hard to notice until it fires: source-built extensions exist outside the package manager's knowledge. apt list --installed shows nothing about them. A Patroni health check does not validate extension binary versions. A node rebuild, a failover to a freshly provisioned replica, or a package update that happens to touch the library path can all silently remove or replace the file.
The fix to the runbook was straightforward: document the source build explicitly, record the exact commit, the PG minor version it was built against, and the library directory where the files live. Any procedure that touches those files (OS updates, PG minor version bumps, replica provisioning) now has a pre-flight step that verifies the extension binaries are present and match what the catalog expects.
The Homelab Control Plane and the Apple Health Pipeline were both backed by databases in this cluster. Neither has any logic to handle a database that refuses connections at the extension load stage rather than at the query stage. That is the real lesson: source-built extensions create a failure mode that looks nothing like ordinary database downtime. Twelve simultaneous connection failures with identical error messages is not a load problem or a network problem. It is an extension binary problem, and nothing in standard monitoring catches it before it fires.
If you are running any PostgreSQL extension that was compiled from source, check your failover runbook today. Document the build, the version, and the PG minor version it targets. The package manager will not remind you.