Cascading Failures

Well, I promised not to discuss my life in here, but since I'm about to use it as an example of generalized system failure modes, I figure it's okay.

The goal: from Waterloo on Thursday morning, travel to Montreal by Friday evening at 8 pm. By car, this is a 7 hour drive. No problem, right?

Well, we were going to rent a car and drive on Thursday afternoon, but it starting snowing/raining/sleeting so we decided not to drive after all; instead, let's take the train, which is safer in bad weather. Since the night train leaves you kind of tired, we decided to take the Friday morning (9:30 am) train from Toronto.

Friday morning, the weather still sucked, but that's okay. We called a taxi at 6:20 am to take us to the bus station in Waterloo. At 7 am, it finally arrived - delayed by bad weather, of course. So we missed the 7am bus to Toronto. No problem, there's an 8 am bus that should still make it to Toronto in time. Unfortunately, the 8 am bus showed up at 8:30 (bad weather), departed shortly afterwards, and got to Toronto at 10:00 (extra late - bad weather). No problem, though; we rescheduled our train tickets from the 9:30 to the 11:30 train. We changed the reservation by cell phone from the bus, luckily, because by the time we arrived all the trains for the day were fully booked. Turns out all the airports were closed (bad weather) and the people taking flights had all switched to the train.

As we were picking up the tickets, they made an announcement that the 11:30 train would be leaving at 12:30 instead - bad weather. No problem: the 11:30 is supposed to get to Montreal at 4:45, so an hour later is 5:45, and even with additional weather delays I should certainly be in Montreal by 8 pm. So, we have some time, let's go for lunch.

At 12:20, we came back and found out that the train had left at 12:07, having been re-re-scheduled while we were gone. In fact, they had made the new announcement before we left the station, but because of a ridiculously loud random music performance (something about the Juno awards) in the middle of the station at the time, all the public announcements were inaudible.

Feeling guilty, they asked us to wait while they figured out what they'd do to get us to Montreal. The result: at 1pm or so, we found out that they could squeeze us on the 3:30 train (arrives around 9:30; useless) or a special 2:30 shuttle bus (could arrive at 8:30 in good weather; useless). So Via Rail wasn't going to be able to help.

Last chance: rent a car after all (there's a rental place at the train station) and drive it to Montreal. That takes at least 6 hours in good weather. By 1:30 we had almost finished filling out the rental forms, meaning that we could be in Montreal by 7:30 on a good day. Sadly, it wasn't a good day. (Interestingly, if we had known at 11:30 that we would miss the train, the rental would have saved us.)

I mentioned above that the airports were closed too (bad weather).

The Moral of the Story

Despite a metric tonne of backup plans (an extra day; an extra bus; an extra train; backup train should still arrive early; could rent a car if the train was cancelled) and slippage, we still didn't get to Montreal on time.

In management, we call this "slippage." In clustering, we call this "cascading failures."

The lesson to learn here is that if you're going to add redundancy (like the extra buses, trains, time, etc) you'd best make sure that the same root cause can't screw up all of your backup plans at the same time. That means don't put a five-station Oracle database cluster on the same power circuit, don't write software that shuts down and expects the cluster to take over if it gets confused (because what if all the nodes get confused by the same thing?), and don't plug all your backup servers into the same Internet connection. For that matter, don't store them all in the same nuclear bunker in the Swiss Alps. If exactly the wrong thing happens, you'll be in trouble.