Hard Rock Hotel & Casino in Biloxi, Miss., was two days shy of its grand opening when company executives realized that their first guest would be an unwelcome one.
Hurricane Katrina was barreling toward the Gulf Coast town and would pound the casino before the doors had even opened to the public.
The killer storm, which made landfall in Louisiana and Mississippi on Monday, Aug. 29, 2005, caused US$148 million in damage to the Hard Rock. Hit hard were the entertainment behemoth’s brand-new gaming facilities, which floated fully exposed on a pair of barges in order to satisfy offshore gambling laws.
While some IT assets bobbed alongside banks of slot machines on the barges, the main server room was in an office building on land. Yet that equipment proved just as vulnerable in the face of a 30-foot storm surge that caused the Hard Rock to suffer a huge loss.
Equipment located in the casino structure and on the first two floors of the building was lost. And although the main server room was spared, the floor was covered with six inches of standing water.
Looking back on the devastation with no shortage of “Katrina fatigue,” Hard Rock executives stand by the plan that kicked into action as the entire area braced for the hurricane.
“When we received word that we were in the path of the storm, we immediately began our disaster recovery preparations while continuing to make arrangements to open in a few days,” recalls John Murphy, vice president and CIO. “It was quite a weekend.”
It was a weekend that entailed some final touches on a pristine architecture, but it was also a weekend marked by updates to the company’s “hurricane hotline” and redoubled efforts to make sure sufficient backup tapes and other disaster recovery provisions were on hand, Murphy says.
“We recovered very quickly” from an IT perspective, says Rob Weir, director of technology. “Within two days, we had critical servers back online and were able to process payroll.”
“If a similar event were to happen today, we would react in much the same way. We did a lot of things right, but some things we just were not prepared for,” says Weir.
Among other things, the Hard Rock staffers didn’t expect the storm to cause as much damage as it did — especially to the local communications infrastructure.
“Cell towers had been heavily damaged, so communicating was accomplished strictly through text messaging and only after several days of total outages,” says Weir.
As a result, the Hard Rock has changed its plans for deploying personnel if it faces another disaster, says Murphy. “We plan to have an assessment team meet immediately at a specified time and place to determine the extent of the damage and the best course of action.
Additionally, we all carry car chargers for our cell phones,” he says, noting that employees faced a lack of batteries and power.
Preparing for the next disaster has also entailed setting new target recovery times.
“Our primary goal was to reduce backup windows by performing disk-to-disk backups followed by copies to tape,” says Weir. “We now use an off-site storage facility that picks up our tapes daily for secure storage.
“In making these changes, we drastically reduced our backup window and can now back up approximately 50 servers with over 2TB of data in a few hours,” he added. That’s a vast improvement over the 68 hours it once took the Hard Rock to complete backups.
The Hard Rock achieved those goals with the help of storage-area networking software from EqualLogic Inc., which was recently acquired by Dell Inc. The company now uses EqualLogic’s PS Series SAN array, as well as data management software from CommVault Systems Inc.
Those systems host applications like Oracle Corp.’s People??Soft financial software.
“Our disaster recovery objectives are well defined,” says Murphy. “Our primary systems that must be recovered include financial, payroll and human resources. Recovery times for these systems are critical. If we were to have a complete loss, we now anticipate recovery times to be in the 10-to-12-day range. This would include the time it would take to order new equipment and have it shipped.”
Although Katrina prompted the Hard Rock to take an even deeper look at disaster recovery, Murphy emphasizes the company’s determination to look forward.
“Simply put, it is just time to move on,” he says. “However, it is important that we remember the lessons learned during the storm, so that we may be better prepared next time.”