Computers and Water Don't Mix.

Ever had a bad hair day, maybe too much moisture in the air? Well what happens when you happen to be a Dell Optiplex computer and have gallons and gallons of gross sludge water dumped on you? No need to imagine it, check out this video I took of just that exact situation playing out before my eyes:

The poor computer (that shorts out very quickly) you see in the back left on the counter was running one of my very first scripts. It’s job was to monitor and report toner & paper levels of all the printers I was responsible for across the campus. Instead of having to walk to each computer lab I simply had it check each printer’s levels remotely and report when I needed to fill it back up. Tragically it’s life was abruptly cut short by this flood which set off an incident response I’ll never forget.

The deluge of water you see was just the beginning of what ended up being several long months. This incident occurred on the 3rd floor out of 4, with several entire computer labs caught in the crossfire. Along with those computer labs were numerous classrooms, comms closets, elevator(s), a degausser, and so much more. I’m not sure the total damages but between the physical damage, water damage incident response contractors, classroom relocation, legal fees and more I’m sure the total was well into the multi-millions (never did find out the actual numbers). However, despite all this the college still had a service & duty to perform which includes technology- and lots of it.

Thankfully there was a plan, or rather resources set aside for something like this (because who can plan for an impromptu water park?). Immediately sent to our location and at our disposal were two separate sets of emergency-IT kits. Think of them of as entire IT-department in crates, everything you need to setup mobile operations and maintain business continuity. Each kit was rotated between NOVA’s campuses local IT departments on a fixed interval for maintenance and updates. This ensured that should they be needed the business could trust that lifeline to not be DOA at the worst possible moment. The two kits were always kept separate just in case one of them got hit, the other would still be available to be shipped in from another campus depending on where it was. Thankfully a few months prior we were able to pawn off one of the kits to another location after performing the needed maintenance, which meant that we had a full stock of equipment to distribute as needed. Included in the kits were laptops (tons!), peripherals, switches, phones, mobile hotspots (which turned out to be clutch!), cables, PDUs, and so much more. All of it updated and ready to hit the ground running, and hit the ground running we did.

First thing’s first after the initial shock and response was to get the kits. The incident occurred on a Friday, by Sunday-Monday we had both kits shipped in and business impact mitigation plans were under way. If I recall correctly the campus was closed for only 4 days (Saturday, Monday, Tuesday, Wednesday) before classes started back up. To get to that point the local IT team needed to do the following:

• Establish a temporary IT response center & workspace.
  • This took the form of a conference room, we certainly wouldn’t have complied with Covid social distancing!
  • Eventually once the building’s air was cleaned we were able to use a larger workspace so we each got our own table.
• Using the emergency kits, setup every impacted faculty & staff member with a working computer.
  • By then most had desktops that weren’t available so getting everyone access to their email, class portals, and business applications was critical. Most of this work was performed by the helpdesk once a machine had been assigned. Some of the older faculty had difficulty adjusting to a laptop but considering the alternative was dawning a bathing suit and going for a swim… they got over it :)
• Working with business units to establish new classrooms.
  • All of the classes in the impacted building had to be moved. It was essentially off limits for students. The business leaders were able to rent and borrow space from numerous locations in the local area. YMCA, churches, anywhere with space. I believe there were 4-5 locations in total that got utilized, each able to have a few classes running at a time.
  • This is where those mobile hotspots were huge. Having internet was critical for many of those classes (and professors) to function. While the relocated classes didn’t require a full computer lab, it still involved having a computer and whiteboards to teach with. Each kit had several hotspots from different vendors. ATT, Verizon, maybe even a TMobile one. Each and every one was used.
• Assess the damage to the building’s equipment.
  • Some computer labs got annihilated, while others got lucky depending on location. One of the big efforts we had to work on was saving/recovering those machines that were still operable and collecting those that bit the dust. Unfortunately the IT room was nearly ground zero for the flood so just about everything there got ruined. My computer was able to power on just long enough for me to recover some documents before getting thrown to the wolves.

There was plenty more that was done over those months, especially as the building remediation neared completion. Honestly though huge shout out to Rachel Overbey, she did a fantastic job leading the team and I’m sure there was tons of problems that the rest of the team didn’t have to deal with/worry about thanks to her leadership. Shout out to NOVA IT leadership too, having those kits was a great catch-all response to many possible situations and something I’ve taken inspiration from for my DR planning now. Later on the campus was able to get a hold of trailers to teach out of which curtailed the need for 3rd party locations, but prior to that having access to space was a massive part of the recovery efforts. I’d highly recommend pre-planning out those locations should your organization need it in the interest of stress/time.

Overall fun time that was, if only I had filmed in horizontal and looked on the outside of that door ;)

Stay safe out there,

~Marcus