Managed Web hosting firms, Internet domain companies and cellular operators have put their emergency procedures to the test following an unprecdented power outage that darkened major parts of Canada and the United States last Thursday.
The cause of the sudden blackout is still under investigation.
Various levels of government declared a state of emergency and those living in the affected areas were asked to stay home from work and conserve as much power and water as possible.
Several banks, including CIBC and RBC, issued statements confirming that customer information had not been lost during the blackout. Many of Canada’s largest transportation companies, including Air Canada, had to suspend operations as critical
IT systems that control the booking of passenger information were affected by the blackout. Although power has been restored in parts of Ontario and the majority of Toronto, the effort to restore the power grid completely may lead providers to conduct “”rolling blackouts””, according to media reports.
Many cellphone users have found themselves unable to use their handsets as little more than flashlights since the blackout began. While Bell Canada claimed in a statement its network was 99 per cent operational, Telus Mobility admitted service levels in Ontario were reduced. Wade Oosterman, Telus Mobility’s executive vice-president of sales and marketing, said engineering staff were manually operating network switches in an effort to balance the traffic loads and redirect signals that go to affected sites. Special power had been directed to emergency service personnel like the Durham Regional Police, he said, which uses its Mike network.
Oosterman said that while networks are much more efficient today and more sites are available, the number of users have increased and applications have exploded, making user expectations much higher than they were a few years ago.
“”If you were only making a phone call a month, if wouldn’t be a big deal,”” he said. “”Naturally there is much more sensitivity.””
Afilias, which runs back end systems for the .Org and .Info domain names, has its technical support centre in Toronto and other systems in New Jersey, and was therefore among the most likely to be affected by the outages.
Ram Mohan, Afilias’ chief technical officer, was in the Toronto centre overseeing the operation. He said the company had sent home its R&D staff, which are also based in Toronto, soon after the power cut out, shortly after 4:00 p.m. EST last Thursday. The company then brought in its on call staff on rotating shifts and switched over to diesel-powered generators to ensure sites with its domains were not affected. Mohan said Afilias called three providers of diesel fuel and offered a bonus if they could bring fuel within an hour. One of them, based in Markham, did. The switchover was complete before 5:30 p.m. last Thursday.
Though its critical systems are running, however, Afilias’s network operations centre is still without air conditioning, and Mohan said the staff were asked to bring in water.
“”We live for the boring days,”” he said, “”but it’s wonderful to think in a time like this that core infrastructure can remain operational.””
At Q9 Networks in Toronto, similar procedures kicked in immediately, according to CEO Osama Arafat. Q9, which provides hosting services for scores of enterprises in a variety of vertical markets, has been proactively calling customers to ensure them connectivity has not been interrupted, Arafat said.
“”We’ve had to use (diesel generators) many times before – there have been occasional brownouts in the city before,”” Arafat said, “”but there’s never been anything of this severity.””
While the outages meant no one in affected areas was likely to be accessing the Internet, Arafat said Q9 had to think globally. “”We provide hosting for Indigo; if someone’s in Vancouver, they don’t care if there’s an outage in Toronto,”” he said. “”They just want to buy a book.””
Mohan said Afilias had already noticed a 10 per cent increase in traffic since the blackout began. Afilias can run for another 24 hours on its diesel generators, he said, and has made provisions to refuel if necessary. Arafat said Q9 could last for four days before refuelling.
