The Ontario Ministry of Labour Monday said it is reviewing its backup and recovery policies following a server failure late last week that wiped out countless e-mail records.
Any e-mail sent to the ministry from 9:00
p.m. Monday, Aug. 16 to 6:00 a.m. Wednesday, Aug. 18 was affected, according to information released by the ministry last Friday.
Ron Brittain, head of IT services for the Economics and Business Cluster, wasn’t available for interviews but explained in an e-mail that “”the failure was caused by two hard drives failing simultaneously in a RAID5 configuration which was fault tolerant for a single drive failure.
“”This occurred during the regular nightly backup window. The failure affected both the operating system and data partition, which meant loss of the entire server,”” he wrote.
The ministry’s e-mail infrastructure uses a Microsoft Exchange server using dual processors. Brittain added that a replacement server had to be built in order to expedite data recovery.
The failure affected every Ministry of Labour office in Ontario as well as the Ontario Labour Relations Board, the Office of the Employer Adviser, the Office of the Worker Advisor, the Human Rights Tribunal of Ontario, the Grievance Settlement Board, the Pay Equity Commission, the Pay Equity Hearings Tribunal and the Jobs Protection Office.
The security of sensitive or confidential data that was e-mailed to the affected bodies has not been compromised, according to the ministry. However, some of those e-mails were sent in response to job postings. As a result job application deadlines have been extended to at least Sept. 1.
Ontario Ministry of Labour spokesman Patrick O’Gorman said that the ministry plans to take out advertising in daily newspapers across the province to alert the public of the e-mail meltdown. The ads should appear some time this week. O’Gorman said that, to his knowledge, no data other than e-mail was lost due to the problem.
Bill Margeson, president of CBL Data Recovery Technologies Inc. in Markham, Ont., said that e-mail is perhaps the most sensitive issue in any enterprise environment.
“”We have recovered some Exchange databases in the past,”” said Margeson. “”Because e-mail has such a short life span, they just panic. The corporation freezes right up and heads start to roll. E-mail is probably the most emotive project for us.””
According to Brittain, the ministry performs a full e-mail back-up nightly starting at 9:00 p.m. and that this process will be fully reviewed as a result of last week’s problems. The server rebuild was accomplished in 16 hours with another four hours to restore data to it. “”We have not yet been able to assess how many messages were lost,”” Brittain said.
This is the second technology problem for the Province of Ontario in as many months. In July, the Ministry of Community and Welfare said it was unable to deliver a three per cent across-the-board increase for welfare recipients. The failure was blamed on the ministry’s IT system which had been supplied by Accenture through a 1997 contract with the previous Tory government.
Comment: [email protected]
