After discovering that a serious system failure could result in two weeks of downtime, Grey County recognized it had to modernize its backup and recovery solutions to ensure continuous availability for critical services like ambulances and long-term care life safety systems during a disaster that couldn’t afford more than 24 hours of downtime.
The decision to modernize was made before a global pandemic swept the nation and the rest of the world last March, forcing organizations to reach similar conclusions about cloud.
“The idea that cloud is a scary place is no longer a thing. But, I think, especially in government, a lot of people are still afraid to go to the cloud. We’re by no means a cloud-first organization but something that we definitely look at – when we need to implement something – is if there is a cloud solution available for us,” Evan Davis, technology and infrastructure manager for Grey County, told ITBusiness.ca.
After evaluating several cloud platforms and disaster recovery solutions with the help of solution provider CDW, Grey County chose Microsoft Azure as its cloud platform along with Zerto for backup and recovery in January 2020.
“Zerto and Azure meet all our varied, demanding requirements and definitely make me sleep better than before, it’s nice knowing that these solutions are there,” Davis said. “Of course, I can say that you just can’t buy a solution and put it in place, you have to do the work at the time, you have to do the work after, it has to be tested, but that’s a great solution and I’m very happy with it. In terms of costs, if we get hit once it’ll pay for itself. But in terms of other solutions, it is more affordable at least for us compared to say, running a second data centre.”
Setting the stage
Grey County is an upper-tier municipal government agency, which covers nine municipalities in southeastern Ontario and offers services to more than 96,000 residents, including long-term care, roads and transportation, paramedic services, social services, government housing and many others.
The biggest threats facing Grey County’s technology infrastructure today, according to Davis, are ransomware, accidental or intentional misuse of data, and environmental threats such as tornadoes and ice storms.
“I’m very much a proponent of being prepared and being proactive instead of reacting to something, so that’s part of why we went with the solution we did so that we could be proactive should something happen to us. When it comes to ransomware and viruses it’s honestly and unfortunately, a case of when and not if it’s going to happen,” Davis said.
The county previously used traditional backup and disaster recovery solutions that were often slow and inefficient.
“If we were hit with a ransomware attack that encrypted all of our systems or had a natural disaster that knocked over our data centre, it would likely take weeks to recover from that with our previous backup and disaster recovery solutions, which is an unacceptable time frame especially for the mission-critical services that we offer,” Davis explained. “This is because the backup would previously be done to disc on site. So, for example, if we had a server issue or if someone deleted something or if we had a hardware failure, we could recover from that however we could have lost up to a day’s worth of data. And if we didn’t have all the hardware available, we would have to buy it which would often consume a lot more time.”
A 2019 survey of the Grey County IT landscape sheds light on some of those inefficiencies. The county backed up more than 100 diverse systems nightly to an on-premises server at the Owen Sound headquarters. Long-term retention data was confined to a single server.
Sure, the redundancy between offices enhanced backup safety, but restoration couldn’t meet the county’s performance standards, Davis explained. Grey County requires a recovery time objective (RTO) of 24 hours or less for some business systems, and it can’t afford downtime at all for others. His team estimated that in the event of a ransomware attack or other disaster, restoring systems could take much longer.
With Zerto and Microsoft Azure solutions in place now, the county can have most systems up and running within a couple of days following an attack, says Davis.
The Zerto solution delivers the continuous data replication that the county wanted. Traditional backup software schedules begin at a designated time and run for several hours during a non-peak usage period (usually at night), meaning that there could be significant gaps between the time of a disaster and the point where the restored data began. “Because the Zerto solution replicates data continuously, we can retrieve data from a recovery point that’s six seconds prior,” Davis said.
In addition to much better performance and peace of mind, Davis says the Grey County IT team has experienced many benefits since implementing in early January 2020 including:
- Projected savings over the next five years of more than 25 per cent.
- Reduction of the amount of people required to manage.
backup/recovery from 13 to 1. - Robust reporting of what’s being replicated and backed up, actual RTOs and RPOs, and numerous other data for monitoring, compliance reporting, and providing information to council executives.