When the lights went out across the Greater Toronto Area at 4:10 p.m., August 14, 2003, ADP Canada — a company whose main job is providing payroll services for 30,000 clients — did not miss a beat.
“”I was working behind my desk,”” says Alvaro Orrantia, manager of security and business continuity
at Toronto-based ADP Canada. “”I saw the lights blink off, and then blink back on again. I knew something had happened, but I didn’t know what.””
Some 50 million people in the Northeast U.S. and Southern Ontario — a region encompassing Toronto, Detroit, Cleveland, Buffalo and New York — were plunged into the Stone Age.
ADP Canada’s head office in downtown Toronto was evacuated. But at its nerve centre in Mississauga, Ont., it was business as usual. The lights were on, the coffee pots were percolating and the vending machines were dispensing cans of Coke as if the worst power outage in North American history were a figment of someone’s imagination. More importantly, ADP Canada continued processing cheques, printing pay stubs and making electronic fund transfers.
How was the company — a subsidiary of Roseland, N.J.-based Automatic Data Processing Inc., a provider of computerized transaction processing with annual revenue of $7 billion (US) — able to sail through the blackout?
As soon as ADP Canada lost power from the electrical grid, its UPS (uninterruptible power supply) kicked in to keep mission-critical applications running. Imagine unplugging your laptop, and not losing any data because it seamlessly switches over to its battery pack, but on a much bigger scale. A split second later, the company’s diesel-powered generator automatically came on, supplying power to both the UPS and non-critical components.
“”When we figured out what was going on from local newscasts and the Internet, we went into a teleconferencing mode with our client-support staff and call centres,”” says Orrantia.
ADP Canada was prepared. It had a tested business-continuity plan in place that could handle virtually any scenario. Major corporations, including banks and insurance companies, were in a similar boat. But many companies, especially small and medium-sized businesses, were caught with their pants down.
The question is: What lessons should executives take from the Great Blackout of 2003?
The plain truth is that the state of Ontario’s power grid — and that of North America as a whole — is much shakier than most of us imagined. New Jersey Governor James E. McGreeve perhaps said it best: “”We have an Internet economy reliant on Edsel technology.””
Is the real culprit behind the Great Blackout of 2003 deregulation and privatization? The short answer is yes, in part. With energy prices tightly controlled by government agencies, it’s true the private sector does not have much of an incentive to modernize the power grid. As a result, most of the hardware is anywhere from 30 to 50 years old. In today’s world of instant access and real-time information, the idea of relying on Elvis Presley-era technology to keep our economic engine stoked is a bit like asking the Jetsons to drive a Flintstone car.
But the biggest problem may not be equipment related. It may just be that our model for power generation and distribution is fundamentally flawed.
“”The system’s design dates back to the turn of the last century,”” says Tom Adams, executive director of Toronto-based think tank Energy Probe.
“”It’s based around generating stations located a long distance from end users, and individual transmission lines carrying a large fraction of the entire grid’s load. The result is a system prone to ungainly collapses that are dramatic and unexpected.””
This system was the brainchild of George Westinghouse, an American inventor whose transformer technology is the backbone our electricity grid. Westinghouse’s biggest rival was Thomas Edison, who argued for local power generation and distribution. Edison’s model was inherently more reliable, but it was also more costly, in part due to its heavy reliance on copper cables. As a result, long-distance transmission of power became the accepted standard.
But as North Americans become increasingly dissatisfied with their power supply, some scientists and engineers are pushing to reform the industry with something known as distributed generation. The idea behind this Edison-inspired strategy is to persuade major energy consumers — including car companies, smelters, refineries, hospitals and universities — to generate their own power, and use the grid for backup.
This may not seem practical on the surface, but the reality is that self-sufficiency may be the only viable alternative for companies who want a reliable power supply at predictable prices.
“”The business sector in Ontario cannot trust the power grid,”” says Adams. “”Ontario’s capacity to supply electricity is fundamentally weak, and we are at risk of a repeat blackout. Companies should plan for an unreliable power supply and higher prices.””
While large companies have a choice, small and medium enterprises do not. The situation is made worse by the fact that the vast majority of SMEs are ill-prepared for any sort of power outage, says John Newton, a director of the Toronto chapter of the Disaster Recovery Information Exchange (DRIE).
“”Investing in batteries and diesel generators is not on their radar,”” says Newton, who is also a crisis and risk-management consultant.
“”They’re just too busy trying to keep their operations going. Many SMEs don’t even back up their data, and if they do, often the backups are not kept off site.””
Newton says the key question businesses should ask themselves is: Will my customers want to use my services in the event of a blackout? If the answer is yes, then even the smallest of companies can invest in an entry-level generator for as little as $200.
If you are an e-business, it is important to find out what sort of business-continuity plans are in place at your ISP. “”Find out how your Web presence will be maintained in the event of a blackout,”” says Newton.
TD Bank Financial Group’s Web portal was not affected by the August blackout. And while the financial services giant experienced temporary disruptions at its transaction and data-processing centres in the GTA, these hiccups had no material impact on its operations.
“”I’m now a consumer of IT services so I am far more critical than I used to be,”” says Michael Foulkes, a senior executive at TD Bank, who until recently was the bank’s CIO.
“”Everything worked flawlessly during the blackout. Discomfort from heat and a little bit of darkness was about the only negative.””
Meanwhile at ADP Canada, Orrantia is busy planning another one of his company’s business continuity simulations.
In this case, ADP Canada plans to re-create its payroll environment at a disaster recovery facility in Markham, Ont., complete with payroll agents and IT staff.
On the one hand, this elaborate exercise is a pain in the butt. On the other, it’s the only way ADP Canada can be sure it will never be caught with egg on its face the next time things go terribly wrong.
