Data aggregation vendor QL2 Software Inc. already follows best practices to prepare for the worst. It has redundant systems and regularly reviews its plans for both disaster recovery and business continuity. But when it comes to the ongoingH1N1 pandemic, company executives admit, they were a bit worried.
To disarm their concerns, they organized and conducted a “dress rehearsal” to see how the company could handle significant staff absences and to find out what they could do better.
The goal: Ensuring that they could still operate if more than 10 per cent of the 42 employees in its Seattle headquarters were infected with swine flu and forced to stay home.
Related stories:
A telework program can help you cope with a ‘Swine Flu’ outbreak
Six smart steps to prepare for a potential swine flu pandemic
“Some contracts with our customers require us to have certain uptime, so we decided to do this to be sure that we could react within the terms of the contracts,” says Rosie Hausler, QL2’s vice president of marketing. “We decided to be proactive and try a work-at-home day. We decided we should practice.”
So on Sept. 24, that’s what QL2 did. All 42 employees in the headquarters building had been told that they had to work from home so the firm’s off-site work processes could be tested and fine-tuned in the event of a major swine flu outbreak locally.
The employees worked from home all day and dialed in for conference calls at 8 a.m., noon and at 4 p.m. with QL2’s business-continuity team.
The 411
Name: QL2 Software Inc.
Headquarters: Seattle
Number of employees: About 163 worldwide, including 42 in Seattle, 113 in India and 8 in Atlanta
Founded: 2003
Business: Scours the Internet for information for each client and turns the information into databases that clients can use for analysis — aggregating real-time fare and routing data, for instance
Client roster: More than 100 airlines, three of the top five global pharmaceutical companies and market leaders in retail, consumer products and life sciences
Any issues that arose that day were logged into a project-management and bug-tracking application called JIRA so the problems could be systematically fixed later. QL2 was already using JIRA as part of its normal IT tool arsenal and used it to help that day, Hausler says.
QL2’s business focuses on the travel, hospitality and retail markets. QL2 scours the Internet for information for each client from disparate Web sites and turns the information into databases that clients can use for analysis. For companies in the airline industry, QL2 aggregates fare and route data so the airlines can compare their prices and routes with competitors, in real time.
This rehearsal was for the Seattle-based workers only. The firm has 8 remote workers in Atlanta and 113 in India but they did not participate in the exercise. Hausler says the company was most concerned about headquarters employees being able to operate remotely; the other employees already do so. “No one was allowed to come into the office or that would have invalidated the test,” Hausler says.
The company’s 14-member business continuity planning team held several meetings to plan the processes for the day, and all workers were provided with the laptops, VPN connections and other IT tools that would let them work from home instead of from the office. Five IT staffers were on the planning team, and these were the people who helped identify potential problems and resolve actual issues that arose the day of the trial.
The company surveyed all the affected workers to be sure that they had used their VPN connections prior to rehearsal day. Employees were asked if there were any other family-related or IT issues that they needed help to resolve before the test, such as barking dogs, the presence of noisy children or a lack of wireless Internet access. Planners advised employees to try to arrange for day care ahead of time, to try to keep dogs in a different part of the house from where they were working and so on.
“We did the survey and we got the results so we knew who we might have to work with to make it a success,” Hausler says. For example, a few workers indicated they sometimes need physical access to paper files as part of their jobs. That issue wasn’t resolved for the day of the test, but in an actual pandemic situation one person would likely be tasked to go to the office each day, retrieve files and paper mail and make bank deposits, among other things.
Money well spent
These kinds of rehearsals are recommended by health agencies including the U.S. Centres for Disease Control in Atlanta. The CDC suggests practice runs as a way to help companies prepare to keep their operations running in the event of pandemics or epidemics that can keep employees home for extended periods.
Kim Elliott, deputy director for the Trust for America’s Health, a Washington-based non-profit disease prevention group, agrees. She calls QL2’s rehearsal a good business move that should be mimicked by other organizations.
“It’s not a practice run,” Elliott says of QL2’s exercise. “We are in a global pandemic. The fact that some businesses are taking it seriously is a good thing. I think it’s actually money well spent.”
Particularly with influenza, the result can be a sustained absentee rate inside a company, Elliott says. “And in a just-in-time economy, if other businesses are having issues with illnesses and their employees, this can affect suppliers, transportation” and many other aspects for partner companies.
QL2’s efforts revealed some important technical issues that might not have been uncovered had the practice exercise not been done, Hausler says. For example, one customer requires his data to be encrypted and backed up to a flash drive that has to remain securely in the QL2 office.
Because the data had to be accessed from offsite for the rehearsal, a specific backup plan had to be created for that customer so that backup could be done by a home-based worker while the data was stored in the manner prescribed by the customer’s contract with QL2, Hausler explains.
A nine-step “special customer file transfer process” was created with the client’s help to ensure that QL2 could do the client’s work while maintaining the required security, she says. The process would be used only if there were issues that came up regarding the data that QL2 delivers to this particular customer. The special process includes methods to decrypt and re-encrypt the data, and employees would access the QL2 server remotely and not transfer any files to a home-based computer.
“Now we have that work-around because we went through this process for this customer,” she explains. “That was another benefit of running this dress rehearsal.”
All told, the exercise cost QL2 under $3,000 — and the main costs were employee time to create the survey and resolve any issues ahead of time. Also included in this figure, Hausler explains, was the cost of employee time for the three planning meetings before the test, the 30-minute check-ins the day of the test and the one-hour post-mortem meeting.
Phone conferencing glitch uncovered
As the dry run continued, another technical glitch quickly surfaced. The IT staff found out that the phone conferencing system they were using had some unanticipated problems.
“A lot of people didn’t have headsets so there was a lot of background noise” that made it difficult for everyone to hear what was being said on the conference calls, Hausler says. “The inability to hear each other on phone conference was probably the biggest issue.”
That problem was easy to resolve. In preparation for a real emergency, all employees were given quality headsets to take home and use when needed in the future.
In general, the rehearsal provided helpful information for the QL2 IT team, Hausler says. One giant relief: The VPN system had never been previously load-tested for use by all Seattle-based workers at one time, but it performed well with all 42 concurrent users. “It absolutely passed the test,” Hausler says. “Everybody was able to, for the most part, do their work” and keep the operations going, she explains.
Only one person, a developer, couldn’t get in through the VPN, which meant that he couldn’t remotely log in to all of the applications he needed for his job that day. The problem was fixed later so he will now have VPN access in a real emergency situation. Following the exercise, the business continuity planning team sought comments from all the workers about their experiences with the drill.
“We came out of that feeling confident,” Hausler says. “We learned some things that we can tweak,” such as the need for telephone headsets. “And the fact that there were no negative customer impacts was really good, too.”
Analysts split on such ‘trial runs’
Results aside, industry analysts are split on whether such preparations are worthwhile for companies, especially with today’s tight budgets and spending restrictions.
Exec non-involvement in disaster recovery a “sign of complacency”
It is a “good idea” for companies, especially large firms, “to come up with any excuse to practice their business continuity plans,” says Richard Jones of the Midvale, Utah-based Burton Group. The swine flu can keep employees out of work for a week, typically, and it’s important to know how to keep going.
If such an extended office closure would happen, Hausler says, it wouldn’t be a significant problem. “For one week, I think our work would pretty much go on.”
Not everyone, however, thinks that QL2’s work-at-home exercise was worthwhile.
Ken McGee, an analyst with Stamford, Conn.-based Gartner Inc., says it’s great to have intact and formalized business continuity plans in place, but called the company’s rehearsal a poor use of scarce money today because budgets are so tight. Further, he points out, the mortality rate from H1N1 is low enough that such an exercise is not worth the expense.
“In this case there is simply no evidence to suggest that on a wide-spread basis we will have large numbers of people afflicted by H1N1 so that we will see the need for this kind of scenario,” McGee explains. If the mortality rate doubled or tripled — and thus approach other pandemic mortality rates from the past — then “we’d suggest this” exercise.
Today though, we are not in that place, he says.
“We are also trying to recover from the worst recession in 80 years,” McGee says. “You’ve got to pick your fights with the money you have to spend. Don’t pull the fire alarm until there is a genuine fire.”
For her part, Hausler feels it was money well spent, even in tough economic times. “The data we provide our customers is mission-critical so anything we can do to ensure minimum data disruption is very important to our business,” Hausler says. “Some of our customers have compliance issues and require BCP plans to be in place.”
Todd R. Weiss is an award-winning technology journalist and freelance writer who worked as a staff reporter for Computerworld.com from 2000 to 2008. Follow him on Twitter @TechManTalking.