The work order was routine. Reboot a switch for a firmware update, wait for links to come back up, move on. Nothing about the plan implied that thirty production VMs would go dark thirty seconds later.
At 11:58 on a Tuesday, all three 10-gigabit NICs on every node in the cluster logged "Link is Down" at the same time. Sixty seconds after that, all three nodes executed a softdog reset simultaneously. The entire Proxmox cluster rebooted itself, without any operator input, in under two minutes.
What softdog actually is
Corosync ships with a watchdog integration. When a node loses quorum and cannot determine whether it is safe to continue operating, it uses the kernel's watchdog interface to reset itself. The logic is intentional: a node that cannot confirm quorum is a split-brain risk, and a hard reset is safer than allowing two cluster halves to diverge independently.
Softdog is the software implementation of that watchdog. It fires a reset when the userspace process stops feeding it keepalives, which happens when Corosync determines that quorum is gone and recovery is not possible.
The nodes did not malfunction. Softdog fired correctly. The cluster protection mechanism did exactly what it was configured to do.
The real problem was the ring topology
Corosync supports multiple communication rings for redundancy. The idea is that if one ring's network path fails, the other ring carries traffic and the cluster stays up. This redundancy is real, but it has an important dependency: the rings must traverse different physical paths.
Both rings in this cluster used 10G interfaces. Both of those interfaces were connected to the same upstream switch. That switch was connected to devices on the same PDU branch as the switch being rebooted. When the maintenance reboot cut power to that branch, both rings lost their upstream simultaneously. From Corosync's perspective, the cluster had lost all communication on all rings at once, which is exactly the condition that justifies a watchdog reset.
The software-layer redundancy was sound. The physical-layer assumption was not. Two rings on different interfaces do not provide fault isolation if both interfaces share the same failure domain upstream.
The second failure, found during recovery
Three hours into the recovery, the cluster was back up but storage was still dark. The backup service came up clean on its own internal checks but showed no connectivity to the nodes.
The cause was a DAC cable seated in the wrong port. The cable had been in the wrong port before the event, and it worked anyway because the path was not the primary one being tested at the time. The switch reboot changed the active path and exposed the misseating.
The fix was a software-side interface swap, not a physical recabling, because the cable was accessible from within the running OS by remapping which interface the storage service was bound to. That took minutes rather than requiring physical access during the recovery window.
After storage came back, 48 Swarm services needed rebalancing across the three nodes because the simultaneous reboot had broken scheduling state. That took another pass of manual verification.
What changed after the event
The upstream switches now sit on separate PDU branches. A future switch reboot or PDU trip can only take down one ring at a time. This is the condition the two-ring design always assumed, and it took a self-fence event to confirm it was not actually true.
The maintenance checklist now includes a step to verify which PDU branch each ring's upstream device is on before any planned network work. This is the kind of check that is easy to skip because it feels like an infrastructure concern rather than an application concern. The event made clear that the distinction does not matter when the outcome is the same.
The Homelab Control Plane that monitors this cluster logged the event across all nodes within seconds. Having that telemetry in a recoverable location outside the cluster itself is worth more during a full-cluster outage than it is during normal operation.
Lessons
Corosync ring redundancy is a software property. Whether that redundancy maps to physical fault isolation depends entirely on how the rings are cabled and powered. The software can be correctly configured and the hardware can still be a single point of failure. These two things coexist without contradiction.
Softdog firing is not a sign that something went wrong with Corosync. It is a sign that Corosync did its job. If the reset feels surprising, the question to ask is what assumption about the physical topology turned out to be wrong, not what configuration change would prevent the watchdog from firing.
The misseated cable is a separate lesson: a pre-existing fault can remain invisible until a different failure changes which path carries traffic. Recovery windows are when latent problems surface, because they are often the first time the full path is exercised under different conditions than normal operation.