The monitoring stack went down during a maintenance window on June 26, and the error message was one of the worst you can see on a database volume: InnoDB: Log scan progressed past the checkpoint LSN. The database was not corrupt in the sense of missing data. It was corrupt in the sense that InnoDB could not figure out where the write-ahead log stood when it last flushed, and it refused to guess.
How the setup worked (and why it was wrong)
Uptime Kuma stores its state in an embedded MariaDB instance. In a Docker Swarm environment, you want persistent storage to survive container rescheduling, so the volume was backed by NFS shared storage. On paper this seems reasonable: the container can be scheduled on any Swarm node and its data follows it.
The problem is that NFS does not provide the fsync semantics that InnoDB requires for its write-ahead log. InnoDB's durability guarantees depend on the kernel actually flushing writes to stable storage when it says it did. NFS lies about this. The NFS server may acknowledge a write that has not yet made it to disk on the backing storage, and the NFS client has limited ability to force a true sync through the network layer. In practice this means the WAL and the data files can be in an inconsistent state after any non-graceful shutdown, because the ordering guarantees InnoDB assumes do not hold.
There is also a second failure mode specific to Swarm. If the container is rescheduled while a previous instance is still shutting down, both containers can have the volume mounted simultaneously for a brief window. InnoDB's crash recovery assumes exactly one writer. Two concurrent writers against the same WAL produce corruption that neither instance can recover from.
The ungraceful shutdown during the maintenance window hit the first mode: the NFS volume had an incomplete flush visible to InnoDB when it came back up, and the engine's log recovery could not establish a consistent checkpoint. It refused to start.
The interim fix
The path out was to move MariaDB off NFS. I pinned a new MariaDB service to a specific Swarm node using a placement constraint and gave it a local volume, not an NFS-backed one. A local Docker volume is a real directory on the host's disk with real fsync semantics. InnoDB is happy with local storage.
Monitoring came back up within the hour. The data itself was recoverable because the issue was a WAL inconsistency at shutdown, not physical disk corruption. With a clean local volume, MariaDB initialized cleanly, and a dump from a recent backup populated the existing monitors and heartbeat history.
But this fix introduced a different problem: the container was now node-tied. If the specific Swarm node it ran on went down, the container would be rescheduled onto a different node where the local volume did not exist, and monitoring would be down until someone intervened manually. I had replaced one failure mode with another.
The gap this exposed
The immediate incident was a failure of the storage layer. The deeper problem it exposed was a gap in the infrastructure design.
The homelab had been running Patroni-managed Postgres with etcd DCS, HAProxy health checks, and a stable VIP endpoint for over a month at this point. Postgres failover was automatic and transparent to applications. But every service backed by MariaDB had no equivalent protection. Uptime Kuma was the most visible example, but it was not the only one.
The honest description of the state after the interim fix: monitoring was back, but the monitoring service itself had a known single point of failure. A Swarm node losing power would take monitoring with it.
That is the architectural gap this incident forced to the surface: HA Postgres on one side, nothing on the MySQL side. Closing it properly meant building a real HA MySQL solution rather than continuing to work around an inherently fragile pattern.
The path from this incident to a proper resolution (a three-node Galera cluster with ProxySQL in single-writer mode and a Traefik VIP endpoint for the stable connection string) is covered in the post from July 2026.
What to take away
The lesson is not "never use NFS." NFS works fine for application data that is read frequently and written sequentially: logs, media, configuration files, export artifacts. It is wrong for any database that uses a write-ahead log with flush ordering guarantees, because NFS cannot honor those guarantees across a network boundary.
The rule is simple: embed storage with compute. If the database engine and the disk are not on the same machine (or at least the same local storage path with real fsync semantics), you are betting the database's durability guarantees on a network protocol that explicitly trades durability for throughput.
In a Docker Swarm context, "replicas:1 with an NFS volume" is not high availability. It is delayed failure. The only question is what triggers it first.