It’s not that keeping up with software patches is getting harder. If anything, patch management has become a bit easier in recent years, thanks to software vendors taking the issue a little more seriously and listening to their customers’ concerns. Thanks also to a proliferation of third-party tools and services designed to help IT departments handle the problem.
But many IT shops still struggle with patching. And while doing so may not be any more difficult than in the past, it is increasingly important.
While some software patches are simply meant to clear up bugs that are annoying but don’t present serious threats, a good number deal with security vulnerabilities that could lead to real trouble if left unpatched, and the urgency of fixing those vulnerabilities continues to increase.
“In today’s world, there’s very little time to go through a process and to evaluate the fix and test it out and get it applied,” warns George Kerns, president of hosting and security services provider Fusepoint Managed Services Inc. of Mississauga, Ont.
Fail to apply a critical patch and your systems may quickly be vulnerable to attack. But apply the patch without testing its compatibility with your systems and you risk creating other problems. It’s no wonder that an ad hoc approach to patch management just doesn’t work for any but the smallest organizations.
Businesses that have got patch management under control have done so by thinking through their patch management policies, setting priorities intelligently, using automated tools and services to help manage the task and recognizing the importance of never-ending vigilance.
Sirius Canada Inc. is in an emerging and competitive business: satellite radio. To compete effectively with its primary rival, Canadian Satellite Radio, Sirius requires the best employee productivity bang for its information technology buck, according to Andre Allen, the Toronto-based company’s senior director of IT. Sirius is also concerned about making sure that both corporate information and data on its customers is kept secure. Proper patching, Allen says, is the first line of defence.
For software that is used widely on its roughly 100 personal computers, Sirius has an automated patch application. Tools from Microsoft Corp. allow software updates to be applied when an employee logs in.
But Sirius also does periodic checks to make sure patches have been applied to all its machines. If a laptop has not connected to the network for a while, for instance, it might not have received an important patch.
For software running on only a few machines, manual patching makes sense. It’s not worth putting an automated process in place for something installed on only 10 or 12 desktops, Allen says.
It can be important to test patches before distributing them throughout the organization. “Often software developers, in their haste to fix things, will be narrowly focused on something,” Kerns says. A patch may fix the original problem but also create a new one.
Nor is this simply a matter of sloppy programming – software can be installed in so many different environments, with virtually unlimited combinations of hardware and other software, that software vendors can’t anticipate every possibility. The impact of a patch is really only clear when it’s installed in the environment where it will run.
“Gone are the days of just blindly applying patches as soon as they’re available,” says Darin Stahl, research lead at Info-Tech Research Group Inc. in London, Ont. “I mean, that just inevitably breaks things.”
Yet testing takes time, so there is a perpetual balancing act pitting the need for testing against the need to apply the patches in a timely manner.
The answers very much depend on the potential impact of problems. Allen came to Sirius from a bank, where patches were tested extensively before deployment. That made sense there, he says, because of the number of machines being patched and the complexity of the environment.
He doesn’t think it is as necessary in the smaller and simpler Sirius organization, where the policy for many patches is to install them on the IT department’s machines first, see if any issues appear there and, if not, release them for automatic updating to the rest of the company’s PCs. “Our strategy really has been to get the patch out there fast,” Allen says.
There are exceptions, though – notably applications where a glitch could cause major trouble. “We’re not going to
automate any patches related to our payroll management system,” Allen says, because in the event of trouble, backing out of failed transactions could be tricky. Also, this system runs on a limited number of PCs, so manual updating – with careful attention to making sure the patches don’t introduce problems – isn’t too onerous a task.
McCain Foods Ltd. of Florenceville, N.B., tests patches for a day or so before deploying them widely, according to Ken Brown, McCain’s director of computing services. Using automated distribution tools, “we can get the patch out now en masse fairly quickly,” Brown says. “We still have to go through our due diligence, though, to ensure that the patch itself isn’t going to cause problems.”
Scheduling downtime
Another problem that increasingly afflicts IT organizations today is the issue of when to patch. “A lot of our systems now are online 24/7,” Brown says. Depending on the amount of disruption a patch will cause, he says, sometimes it’s necessary to schedule downtime – during the night if necessary – to get the updates done.
Brown is pleased with automated patch distribution, which McCain has used for about a year in most parts of its operations. While he admits he has no hard numbers on the amount of time saved, he believes the savings are substantial.
Knowing what to patch can be a significant challenge. Software vendors have adopted varying approaches to helping customers with this, such as Microsoft’s “Patch Tuesday” plan, in which software updates are issued on the second Tuesday of each month and ranked by criticality to help customers decide which ones need the most immediate attention.
Other sources of patch information include anti-virus companies, which warn of significant security vulnerabilities, and the CERT Co-ordination Center at Carnegie-Mellon University in Pittsburgh, which tracks security issues. Allen reports turning to both these sources for information.
Just knowing what vulnerabilities and patches exist is half the battle. But it’s also necessary to know which machines in the organization will be affected. Third-party companies such as San Francisco-based nCircle Network Security Inc. offer help to organizations on this front.
Each time a major software vendor issues a patch, nCircle provides customers with an electronic signature they can use to identify all the machines that require that particular patch. For Microsoft patches, nCircle guarantees this signature will be available within 24 hours of the patch itself being released.
“You need something to provide you the awareness of what’s on your network and you need to be able to determine in real time where to focus your efforts,” says Minoo Hamilton, senior security researcher at nCircle.
“The interesting problem becomes selecting the right third-party tool for the unique situation that a business finds itself in,” Stahl says.
That decision depends on two major factors, he says: the complexity of the IT environment, which determines what capabilities are needed, and the cost of the tool, which can vary from free to substantial.