When I woke up on Friday, Dec. 12, our house in southeastern New Hampshire was dark.
The previous evening an ice storm coated trees, roads and power lines with a layer of ice at least a quarter-inch thick. Tree branches—and whole trunks—snapped, taking vulnerable power lines with them.
By Friday morning, with the storm ongoing, blackouts had hit hundreds of thousands of households and businesses in Maine, Massachusetts, New Hampshire, and New York, including mine.
Power outages are fairly common in our part of the state. In the past year, we have had a dozen or more minor hiccups, resetting electronic clocks and flickering lights. Once a year, we might have an outage that lasts hours. Never before had I seen one that lasted for days.
In the end, the New England Ice Storm of 2008 knocked out power to an estimated 1.25 million people. The impact on businesses has not yet been fully tallied, but—while hotels with power were flooded by residents needing shelter and hardware stores did a brisk business in generator supplies—other companies found themselves shutdown for days to weeks and unable to contact remote workers in the area.
“Manufacturers were especially hard hit with shift after shift for days of idle machines followed by unreliable, on and off, power for additional days after area power was restored,” says Nancy Jackson, director of the North Central Massachusetts Economic Development Council (NCMEDC). “Their customers complained that these manufacturers, their suppliers, were ‘not even answering the phone’ when they called about late or absent shipments. They didn’t understand that there were no land lines for email and telephone communication!”
Nano-materials startup Nanocomp, based near Concord, NH, lost power for six days. To the company’s customers, the firm essentially disappeared, says Peter Antoinette, president and CEO.
“We were completely out of business for that period of time,” he says. “No phones. No Internet. Basically, to the outside world, we disappeared.”
Here’s five lessons any IT pro or telecommuter can take away from the storm.
1. Scope out back-up power for necessary systems
Businesses need to ask themselves a simple question, says Steve Hilton, VPof enterprise and SMB research for the Yankee Group, a business intelligence firm.
“How much will an hour, a day, or a week without connectivity or power cost their businesses?” he says. “This simple question puts some boundaries around the need for and scale of business continuity and disaster recovery (BCDR) planning.”
For some, the answer will be a complex mix of off-site data centers, quarterly catastrophe simulations and duplication of critical business applications, Hilton says. For others, a few extra cell phones and a power generator from Home Depot will do the trick.
For a telecommuter, a generator suits just fine. However, for a manufacturer, the equation looks much different. At Nanocomp, management is still mulling whether the risk of another storm is worth spending needed cash reserves on backup systems.
“It becomes a very difficult cost tradeoff situation,” Antoinette says. “Do you go out and spend (hundreds of thousands of dollars) on that type of systems, when you are a small company trying to wring out the investment out of every dollar that you have?”
2. Remember: The Internet is everything
For most businesses, the Internet represents the primary line of communication with the outside world. Yet, even with the power on, Internet connectivity that relies on cable and phone lines may no longer work.
On Tuesday, Dec. 16 — Day five of the power outage — I had a generator powering the essential heating and water-pump systems in my house as well as my office. However, our cable provider had not yet repaired the damage to their own system, so we still had no Internet connectivity.
Companies with a large portion of telecommuters will increasingly have to deal with the infrastructure problems of their workers. The Yankee Group estimates that 15 percent to 25 percent of all workers (depending on the size of the company) telecommute.
While there has been some backlash against work-at-home employees, small- and medium-sized businesses will continue to allow more employees to work remotely, says Yankee Group’s Hilton.
“Telecommuting is headed in one direction and that’s up,” he says.
3. VoIP and PBX: No power means no phone
While both voice-over-Internet-protocol (VoIP) and private branch exchange (PBX) telephone systems save companies money during normal business times, during an outage both systems exhibit a glaring weakness: No power means no phone.
In addition, VoIP also relies on having Internet connectivity, and a company’s back-up line may not have enough bandwidth to carry voice signals reliably.
My cell phone replaced my VoIP phone line for the six days that we had no power. For most telecommuters and mobile workers, cell devices allow reliable communications, even during an outage. In addition, my voicemail is kept on my VoIP provider’s servers, so I could retrieve messages from anyone who tried to contact me.
That was a problem for NH-based Nanocomp. The outage downed the company’s PBX system, and it had no easy way to notify customers of its situation.
“It took us a while to jury-rig an announcement to let people know that we were not gone,” Nanocomp’s Antoinette says.
4. Fault-tolerant drives are not backups
Perhaps the most long-lasting impact of the outage for me was that, when the power came back on, I found that my redundant back-up disk had died.
While I backup my writing and business files to an online service, I also have to archive a lot of large media files. I had settled on a four-drive RAID system as a way to keep a copy of the 200GB of media files I had collected. While none of the hard drives crashed, the drive hardware had. Using various archives, I have been able to recover about 95 percent of the data to date.
While many companies have strict back-up policies in place, their telecommuting workers likely do not. Now might be a good time to revisit the issue.
5. Brace yourself for outage stress
A final lesson from the outage: Don’t underestimate the impact of the events on workers. Telecommuters, for whom the Internet and phone is a lifeline to their co-workers, may feel completely cut off. Managers will have to endure the additional stress of trying to make the business work during extraordinarily hard times.
Outside of work, everyone will have to deal with stressful situations at home, such as flooded basements and families tired of the difficulties of life without power.
What did this one ice storm add up to in the Northeast?
“Lost revenues for most businesses, lost wages for thousands of hourly workers, unexpected additional expenses for businesses, including home-based, and households alike, coupled with the frustration of communications difficulty or failure, (and) all caused by a massive ‘poles and wire’ system failure,” laments NCMEDC’s Jackson. “Sounds like a tale your grandpa would tell from the old days, but this one event will have a dampening effect on the region’s economy for some time to come.”