I just came off my second consecutive overnight shift, after our SonicWall router/firewall/VPN appliance, on which we rely for all routing to our other sites and to our various clients, died of a blown power supply after an apparent electrical spike. The first overnight was spent patching together a bunch of crappy Linksys routers to do routing to the various links, and we were forced to route all internet traffic over to our sister site (over a horrifically slow link). As this is a work story, you’ll forgive me if details are extremely sparse.

Last night’s goal was to replace the SonicWall, and restore normality, as our bubble-gum-and-scotch-tape solution was not providing all the routing that we would have needed to be considered at 100% operating capacity, and a number of other divisions in the company were screaming for data that they weren’t getting without the routing that we had foregone in the kitbash setup we had to settle on. The IT guy from one of our other sites had driven up his backup SonicWall (mine is one of the only sites that didn’t already have a backup on hand… figures mine is the one that would blow). As I was at home, sleeping, from the previous night’s solo slog with support from a company higher-up over the phone from halfway across the country, I wasn’t there to receive the new appliance. The receptionist received it, and put it into the front office store room for security while she was gone.

And boy was it secure. That storage room is one of the two rooms in the building to which I don’t have a key (the other being the site manager’s office). Nor did any of the on-duty supervisors, nor is there one in the key box, though there had been at one point.

That said, I wasn’t about to let a little thing like my SonicWall being locked in a room stop me from doing what had to be done. ReformedYankee was there, and he’s built like a fridge, so I asked him if he could carry my weight — his response was to immediately pick me up by the knees and walked me around the room. So that was in no doubt whatsoever. Knowing I could trust him to carry my weight, I grabbed a ladder, and a 50ft coil of Cat5 cable, coiled the Cat5 so that it was quadrupled and tied around my ankle, and looped around my foot so it could carry my weight. Then, I went up into the ceiling above the drop tiles, over the rafters, over the top of the drywall, having to break a chunk of it that was fortunately above the drop ceiling line in order to fit between it and the ventilation ducts (I’m skinny, but even that was a bit small for me). ReformedYankee lowered me into the room from above at that point, with the Cat5 slung over the metal beams across the ceiling, and I made touchdown with hardly a scratch but covered in drywall chalk. I opened the door, high-fived him, grabbed the SonicWall and was on my way. So the lesson from that escapade is, secure rooms are not secure if you have a drop ceiling above them. (Alternately, another lesson is, four strands of Cat5 cable can hold 180lbs fairly easily, but that one doesn’t count toward my two lessons.)

Then came the fun part, trying to reconfigure the replacement SonicWall to connect to all our various setups without a recent settings backup. My first thought, from the night before, had been to check to see if the power supply was replaceable — it’s got custom screw holes, but it looked like it could be removed if I cracked the case. I didn’t have any spares, but I figured I could cannibalize the replacement to get my own up and running. It also had a flash cart inside to store the OS and settings, which I could theoretically have moved to the replacement as well, but my higher-up vetoed that idea.

So, two hours into the troubleshoot, we’re making almost no progress whatsoever, barely getting internet connectivity to my laptop via the regular internet pipe, which had been going disused for the day, without a proper router to configure it on. I sighed, went to get some coffee, and when I came back, I noticed the old SonicWall, which was disassembled and in the back of the LAN room where I had left it from the night before. Specifically, I noticed the spot where the power supply connected to the board. A standard molex connector. Just like on an ATX power supply.

I slammed back the remains of my coffee, ran to my office and grabbed a spare ATX power supply from a computer I had dismantled earlier in the week, connected it to the SonicWall, and held my breath as I plugged it in.

*whirrrrr BEEP*

I jumped back onto Skype and told my superior that I figured out how to get the old SonicWall to boot, and that I was going to download the settings. Without waiting for him to answer, I disconnected my laptop from the replacement and connected to the original, downloaded an export of its settings, and unplugged it all for fear of exploding the bloody thing when/if the power supply gave it too much juice. One quick import later, some reverting of routing on our Cisco Catalyst, a bit of fiddling with cords, and normality was restored. The lesson there is, SonicWalls can run on ATX power supplies in a pinch. And if I had known that on Tuesday afternoon when the old one had died, I could have saved myself a lot of trouble over the past two nights. (An alternate lesson: always have a recent backup for all your critical routing appliances. Or, alternately, always have an on-site physical replacement for these critical appliances. But these lessons were not learned last night, nor apparently were they properly learned previously; so I can’t rightly include them in my list.)

All in all, it’s been a fun couple of days. This was a huge distraction from my ongoing work-gone-overdrive project, but it will give me some breathing room on it as I’ve bought myself a bit of goodwill. And all of this is completely neglecting the home drama that Jodi’s embroiled in, and the attempt to get Jen and Opal settled and situated. I still might be a bit more absent than usual until normality is restored on the homefront, but I’ll try to get a few more linking posts and funny Youtube videos up here whenever I can.