Terorrist attacks, ice storms and a number of lesser emergencies gave Canadian enterprises a good education in disaster recovery that prepared them for last month’s blackout, experts said recently.
As many as five days after a grid failure darkened most of Ontario, the provincial government
asked businesses and communities to continue conserving power in order to avoid rolling blackouts.
At press time, private sector firms were back in business and government agencies were getting back to normal after working with skeleton staffs for the previous week.
Despite the severity of the blackout, which experts have called the worst in history, companies responsible for recovering lost or corrupted data reported little change in their day-to-day business.
“”There were no servers that caught fire or tragic issues that way,”” said Bill Margeson, CEO of Toronto-based CBL Data Recovery Labs. “”Things got managed this time.””
The situation after the blackout is much different from that following the 1998 ice storm in Montréal, where server failures and major data loss temporarily crippled some businesses, Margeson said. This time, CBL saw a 50 per cent increase in the number of calls it received, but Margeson said they mostly concerned surge or power supply-related issues around a single drive in a user’s laptop.
Although there were some concerns during the week of Aug. 18 that a failure to conserve would make the lights go out again, Margeson said user awareness about disaster recovery has grown considerably.
“”We’ve been standing in the wilderness for 11 years — now people are approaching us; it’s on their mind,”” he said.
At Royal Bank of Canada, Canada’s largest bank, the Toronto data centre and a backup data centre at an undisclosed location outside the city were drawing only 10 per cent of their power from the electrical grid, according to RBC senior vice-president and chief information security officer Diana Burke. The remainder of its power came from diesel generators, which themselves have backup generators. Burke said the bank hasn’t experienced any backup and recovery issues and the data centre and backups can be run three to five days solely from the generators.
Burke admitted the terrorist attacks of Sept. 11, the ice storm and the Year 2000 problem showed how necessary disaster recovery strategies are to the bank’s business continuity strategy.
“”Every incident like that helps you review your process and plans,”” she said. “”It’s unfortunate . . . (but) there’s no question they helped us.””
Veritas Software tried to further that education process by releasing a five-point overview of disaster recovery basics. Fred Dimson, general manager and director of operations at Veritas Canada, said the blackout would likely see many enterprises reconsider where their data centre and backups reside.
“”The replication of data centres (used to be) about migrating to Ottawa or somewhere else in Ontario,”” he said. “”Now we’re looking at wide-area replication issues where you have to think hard about where you may want to put secondary sites. A few years ago, it was one building to the next.””
The government’s request to continue power conservation during that week may also expand the scope of some corporate disaster recovery plans, Dimson added.
“”Most (companies) already have diesel and backup generators,”” he said. “”In reality, the data centres weren’t in bad shape, but a lot of them hadn’t thought of the power going off for two or three days.””
While it may be difficult for some enterprises to conceive of the worst-case disasters, Dimson recommended they think of the financial impact of failing mission-critical applications.
“”People don’t really spend the time to look at their operation and say, ‘If I don’t have HR for a week, I can survive, but what about payroll,”” he said. “”Here they had a test.””
As for RBC, Burke said the bank continued to use diesel power until it was comfortable there would be no more blackouts.
On Aug. 24, Ontario Premier Ernie Eves announced the province’s power supply was back to 100 per cent.
— with files from Dave Webb