I thought it rather ironic that as I was installing my HA firewall cluster that I hadn’t planned the whole hardware redundancy thing all the way through.
In order to install the new machine and the NICs, I had brought a screwdriver to mount rack rails and such. This particular screwdriver was one of the ratcheting kind, and it’s been a poorly functioning department fixture since before I arrived. Today, it decided to completely and catastrophically fail. One moment, I’m turning a screw, the next finds my hand holding about half a dozen pieces of the ratcheting mechanism, and the screwdriver shaft spinning freely and uselessly.
… and in my planning to build the HA cluster for the firewall, I’d neglected to brnig a spare screwdriver in case that hardware failed (which we’d expected it to do long ago). Luckily, one of our “neighbours” happened to have one with him and let me borrow it.
Moral of the story, make sure you have full hardware redundancy, including your screwdrivers.