On a cold winter night in downtown Toronto last year, Sean Shannon received the e-mail that most managing directors would consider a nightmare scenario.
“Your office is now closed, access is denied, please activate your continuity plan and continue your operations in emergency mode,” the manager of Expedia.ca read off his BlackBerry. It was a surprise drill run by Expedia Inc.’s business continuity team, and it meant all employees would be unable to access the office that night and the entire next working day.
But Shannon knew exactly what to do.
“You just go to your playbook,” he says. “Off I go to the hotel, the room should have already been set up for us because our office administrator booked it.”
Expedia runs several of the Web’s most popular travel sites including Hotwire.com, Hotels.com, and Tripadvisor.com. If its business operations at any of its international office locations are disrupted by a disaster and employees can’t get work done, it could mean a serious hit to the company’s bottom line. That’s why it executes these live-fire drills on a surprise basis at all of its locations — to be prepared for the worst case scenario.
Toronto’s drill is titled “Oh S%#t Canada.”
Disaster exercises are the best way to be prepared for disruptive situations, says Howard Mannella, principal resiliency strategist at Expedia. He presented at Toronto’s World Conference on Disaster Management June 7.
“You don’t get competency just sitting around a conference table and reading a plan,” he says. “You don’t find technology challenges unless you either run disruptively or do the real thing.”
During the Toronto drill, all employees were denied access from the main office building. That meant they were either working from home, or in a hotel business suite that Expedia had reserved as a backup site for these situations. The firm also has an office based in nearby Mississauga.
“People could work from a Starbucks if they wanted to,” Shannon says. “But for a one day thing, having a physical office isn’t that critical.”
Workers connected to the office using a VPN that gave access to corporate data on a Sharepoint server, e-mail access, and the live Web site’s content management system. By using a different PC and working remotely, Expedia discovered technology glitches that prevented work from being done, or slowed it down.
“You don’t discover issues like bandwidth and latency by doing table top exercises,” Manella says. “You discover them by forcing people out of the office and getting them to use tools under stress.”
Related Story
Want senior management support? Stop asking for it
Workers learned to make a practice of moving their files off their local hard drives to the Sharepoint server for remote access, he says. Expedia.ca’s backup servers also proved to be effective during the drill.
Expedia didn’t run this live-fire drill without weeks of advance planning. It may have been unannounced, but staff knew it was coming.
Program preparation started six weeks ahead of time, Manella says. It involved working sessions with the IT department, training managers how to respond, and identifying a recovery team that would mobilize. It’s important to think of everyone who would be involved in such a scenario — both inside and outside of the company — including authorities, such as the police and fire department.
“It’s a matter of being well documented and well talked through,” Shannon says. “You want to find the problems in advance instead of working through them on that day.”
Local Expedia management backed the plan, Manella says. But there is inevitably some push-back among employees.
“A bit of diplomacy and acceptance will help your cause,” he says. “Doing it not quite right, but still doing it will help you learn a lot.”
Every office that goes through the drill should expect a small hit to productivity that day, Mannella says. It’s the price paid to keep a business resilient and ready to respond to a unique situation.
Toronto’s office had very good productivity the day of its live exercise, Shannon says. Some in-person meetings were either shifted to the phone or rescheduled for another day, but otherwise it was business as usual.
From Toronto to Paris, from Munich to San Franciso, the staff reaction to these live fire exercises is the same, Mannella says.
“The universal feedback from the people who go through these exercises is: it’s annoying, it’s inconvenient, we learn a ton and please come back and surprise us again because we bet you we can improve,” he says. “They actually love it.”
Well, maybe love is a strong word.
“We’ll do these things once a year,” Shannon says. “It’s sort of like going to the dentist, I don’t like doing it but it’s the right thing to do.”
At least he knows what to do the next time that unpleasant e-mail shows up in his Inbox.
Follow Brian Jackson on Twitter and check out the IT Business Facebook Page.
