It wasn’t until the lights went out in August of 2003 that many companies were able to see the folly of their non-existent, inadequate or long-shelved and forgotten business continuity and disaster recovery plan.
When the blackout hit on Aug. 14, 50 million Canadians and Americans were left
without electricity. Canadian businesses lost some 18.9 million work hours and the country’s gross domestic product took a 0.7 per cent hit, according to a report by the U.S.-Canada Power System Outage Task Force. This is not surprising, considering a recent survey of business decision makers across Ontario found that 28 per cent of organizations do not have any business continuity plan. The survey was conducted by Leger Marketing on behalf of Fusepoint and Agility Recovery Solutions.
The University Health Network (UHN) was among those organizations that did have a plan in place — yet even it learned a lot of valuable lessons during last year’s blackout.
“”All of our systems were available during the blackout period. If it had been extended, we would have had issues,”” says Jennifer Willis, manager of technical services at UHN in Toronto.
The hospital’s business continuity and disaster recovery plans had never been tested to the extent they were during the blackout. Though the hospital’s medical equipment was well protected, the blackout highlighted the need to further protect its computer systems, since they are increasingly tied to the hospital’s ability to provide patient care.
The hospital had planned for downtime scenarios of 12 hour outages, and since the blackout has decided to increase that to 72 hours. Though the data centre, which is located at a separate site from the hospital, had an uninterruptible power supply (UPS) system and a diesel generator in place, it had trouble getting fuel deliveries. It had to tap into the hospital’s supplier, Willis says.
The data centre only had one generator. If it had gone down, the data centre would have had to shut down applications. Addressing the power supply was one of the first orders of business for the hospital after the blackout.
It was also something Dick Swadley, the executive vice-president at Royal Bank of Canada in Toronto sought to address in the wake of last year’s blackout. Though the bank’s computer centre kept running, the suppliers the Royal Bank counts on to replenish its diesel had problems as they relied on electrical power to run their pumps. The bank now makes sure its suppliers have UPS systems and generators. The bank also had an electrical engineering firm conduct a thorough power review of its computer support facilities after the blackout. This is something it does every few years, since it is always adding and removing equipment.
The hospital conducted an audit of its own. It did an extensive review of its disaster recovery processes. It is currently relocating its data centre to Hewlett-Packard Co. in order to increase its redundancy, Willis says.
The blackout also highlighted the need to make sure more computers are available to the staff if another disaster occurs.
“”In the past, our emergency power was simply used for what you’d call medial devices and some connectivity for computers and printers. But with the expansion of electronic health records, the reliance on PCs as part of the care continuum has increased.””
The Royal Bank decided some time ago that it wanted to have guaranteed workstations and office floor space for its staff in the event of a disaster in downtown Toronto, so it set up a business continuity planning facility about 60-kilometres west of Toronto. Had the blackout lasted longer, the bank would have moved its people there.
The hospital is also testing its staff in disaster recovery scenarios during the day as a result of the blackout. It had previously only tested the scenario at night, during low volumes. Testing disaster recovery programs is something businesses need to do often, says Paul Sullivan, the director of business resilience and continuity services for IBM in Markham, Ont.
“”All too often, organizations look at recovery as a project, make it, put it on the shelf and then forget about it.””
But when a catastrophe strikes, some two out of five companies fail, he says. Statistically speaking, it is small and medium businesses that fail, as larger organizations have contingency plans.
The blackout was a wake up call for some businesses. Since the power outage of last August, Q9 Networks, an Internet infrastructure and managed hosting services provider, has seen an increase in business, says Toronto-based CEO Osama Arafat.
Genesys Conferencing in Toronto had luckily just got itself prepared. The Web audio and video conferencing firm had implemented its business continuity plan just three weeks before the blackout hit. Its building had no power or backup generator — but Genesys was able to get an analyst over to Fusepoint, its managed hosting service, within 20 minutes so it could continue to monitor its global infrastructure, says Eric William, the network operations manager at Genesys.
