Data in the dark

E-commerce experts say the market should rebound relatively unscathed following a massive electrical grid failure last Thursday and continuing power woes affecting Ontario and parts of the United States.

Jim Okamura, senior partner with retail analysts J. C. Williams Group, pegged the lost retail sales in the U.S. at roughly US$25-50 million a day, given a quarter of the population was offline. Four days later, users were coming back to their computers gingerly, worried about crashes from rolling blackouts and overloading the grid again.

“”No business can afford to take two to four business days out of their plans,”” Okamura said. “”It’s hard to fully recover those lost business days.””

That said, he doesn’t think there will be any long-term damage to consumer confidence in the medium. And Steve Mossop of Ipsos-Reid in Vancouver says those sales will be made up as soon as online activity returns to normal.

Based on Ipsos-Reid’s estimate of Canadian online retail sales at $4.8 billion a year, having a third of the population out of the buying loop would cost about $4 million a day. However, Mossop said he hasn’t seen a reliable model for making such a calculation.

And since the majority of purchases made on the Web aren’t time-sensitive — commodity items like books and CDs — Mossop doesn’t view the sales as lost.

“”I would suspect a lot of the spending will just be delayed,”” he said.

While blacked-out consumers weren’t thinking much about online shopping, some were at least able to make and monitor banking transactions.

Brad Taylor, Canadian Imperial Bank of Commerce’s senior vice-president of infrastructure services, feared the worst when he learned the power failure extended far beyond Toronto’s borders. “”Having lived through and participated in the 9/11 recovery, you get a chill that goes up your back,”” he said.

But senior CIBC executives were assembled on a conference call and recovery plans were implemented minutes after the lights went out, said Taylor. All of CIBC’s data centres have diesel-fueled back-up generators. The bank’s two main centres in Streetsville and Markham, Ont., remained operational. One acts as a backup for the other in the event of serious problems.

CIBC’s Web site continued to function during the power outage, but obviously not every CIBC customer had the means to get online. Mike Boluch, CIBC’s senior vice-president of retail markets technology, wouldn’t say how many potential transactions the bank lost during the power failure, but there were other avenues available to clients.

“”We also have access through IVRs (Interactive Voice Response) and access to agents through telephone banking,”” said Boluch. “”Those call volumes got re-routed to centres outside of Ontario. If you had power, there was certainly technology there to deal with your inquiries, to deal with your ability to transact.””

Bank transactions that may have been interrupted by the sudden power failure at approximately 4:10 p.m. Thursday weren’t lost, he added. “”No transactions would be lost in-flight. If anything, they would queue up and then there are a bunch of business rules around how those queues are handled,”” said Boluch. CIBC customers were able to check those transactions — and, if necessary, complete them — once power had been restored.

“”We’ve been through this a number of times — with 9/11 and the ice storms in Montreal and other situations — that we’ve continued to refine our plans,”” said Boluch. “”Unfortunately, we’re getting quite good at this stuff.””

Like other Web hosting providers affected by the blackout, Fusepoint Managed Services immediately switched over to its backup diesel generators at its Toronto data centre when the lights went out, according to president Robert Offley. While for most customers this meant business as usual, at least one customer in the IT/media sector took advantage of a FusePoint service introduced six months ago and moved in seven employees to Fusepoint headquarters, where they were provided “”hot desks”” and the ability to keep their business running.

At the client end, Offley said customers have a couple of options if they want to recover the “”orphan data”” from interrupted transactions, depending on how they built their application. Most firms cache all requests, Offley said, but more advanced systems will poll the client until their power is restored, remind them of their transaction and ask them if they want to complete it.

“”I think every business has a fiduciary responsibility and it’s becoming more broad-level governance view — of making sure they can maintain operations and business with their customers,”” he said. “”The critical nature of that type of plan becomes more important depending on the type of business and what it’s trying to achieve.””

Like denial of service attacks and other security problems, Fusepoint is developing a range of professional services to help customers grapple with the unexpected, Offley said.

“”Sometimes it’s takes a disaster like the one we saw last week — and I think we’re still in it — to see some of the real realities home to some businesses,”” he said. “”It was seen after 9/11, it was seen after other events where the spending on putting higher redundant solutions in place will become more critical.””

— with files from Dave Webb, Neil Sutton and Shane Schick

Comment: info@itbusiness.ca