The backup bridge question

This post is brought to you by the Department of Redundancy Department

In August of 2007, an eight lane bridge on Interstate 35W that spanned the Mississippi River collapsed. It killed 13 people and injured many more. It carried 140,000 vehicles every day. So critical was the link that the bridge provided that the replacement bridge was reopened just13 months later.

Design flaws and neglected maintenance were both blamed.

What no one blamed the Department of Transportation for was not having a "redundant" bridge. "Why," no one asked, "was a second bridge not in place so it could be used if the primary bridge failed?" 

__________________________________________________

I think about this bridge a lot as we start examining the results of a network audit we had done by an independent company this year. Knowing that analogies can be dangerous things, I still think about this bridge collapse when seeing recommendations from security gurus for high levels of redundancy.

The reason, of course, for building redundancy into networks is that if there is a failure in the system, downtime can be eliminated or minimized. Two servers, two switches, two network managers, two fiber networks, two ISPs are wonderful to have if a failure of the primary occurs. The secondary can either take over automatically or can quickly replace the defective unit.

In the best of all possible worlds, everything would have a ready replacement in the technology world including, I suppose, the technology director. The downside of course is that having this much additional equipment and other resources doubles (or more) the cost of operation. Fiber optic networks, for example, can't just be two strands - these strands need to be in different physical locations so a single swipe of the backhoe doesn't take them both out. Rack space needs to be leased if secondary service are to be safely located away from the primary data closet.

If there is one balancing act that those of us who work in educational technology must constantly perform, it is allocating technology dollars between the infrastructure/administrative and the student/teacher/classroom sides of budget. For each dollar spent on that second server, one dollar cannot be spent on student computers or online resources or professional development.

Much to the displeasure of the technology experts that recommend lots and lots of redundancy, I push back quite a little. I like to ask some questions:

1. What are the odds and consequences of this particular failure occurring? Can we prioritize which applications and resources are mission critical? 
2. How many services can be moved to the cloud where it is the SaaS providers' responsibility to back things up?
3. What is an acceptable recovery time for each application? A failed firewall that stops all Internet traffic has a much shorter acceptable recover time (none) than perhaps the library automation system.
4. Can we cross train technicians and have external support on call rather than hiring two network managers?

After spending the first half of my career urging educators to use technology, it is ironic that I am spending the last half trying to stay one step ahead of those educators in making sure the technology is reliable, adequate, and available. I believe in the professional parlance it's called "creating a monster."

Nobody wants or expects a bridge to collapse. Nobody expects or wants a network to fail. Sensible precautions need to taken to insure if a failure does happen, mission-critical services and resources are minimally disrupted .

But perhaps, "sensible" is the operative word.