When a new global vulnerability threatens to take centre stage and capture the media’s attention, the public naturally contemplates the individual and coordinated efforts of mitigating the risk. It’s all hands on deck for days or even weeks – as it was with Heartbleed earlier this year – until the situation is resolved and corporate IT can resume normal operations.
During the initial media frenzy, we consider the impact, fret about the damage and sympathize with IT professionals on the front lines. But what about the companies that are part of the security fabric of today’s business world? What about the anti-malware firms, firewall and IDS vendors? What about the companies that have to have almost precognitive capabilities to release zero-day solutions for zero-day problems?
How does a company respond when Fortune 100 and 500 clients instantly expect them to have a workable solution? The controlled chaos of learning about a vulnerability, ensuring that exploits do not victimize them and finally, managing the process of producing, testing, and releasing a product update to deal with the issue is something we don’t often think about.
When the Shellshock/BASH Bug hit prime time and again when POODLE reared its ugly mug, I took the opportunity to speak with Qualys about their tools and processes. Amol Sarwate, Qualys’ director of engineering, took my questions.
Claudiu Popa (CP): What did you do differently/the same way to address POODLE versus ShellShock?
Amol Sarwate (AS): As you may know, Qualys has different offerings including  VM, WAS, WAF, PCI, SSL Labs and FreeScan. When a vulnerability like POODLE or Shellshock comes along, we update all relevant offerings. POODLE was related to SSL and therefore we updated our VM, PCI, SSLLabs and FreeScan offering with unauthenticated detection capability. ShellShock was related to the Unix shell and we updated VM, PCI, WAS, FreeScan with similar unauthenticated detection capability. In the case of POODLE, since the vulnerability could be mitigated by securing the browser, we developed checks to verify vulnerable browser configuration, in addition to the server side check. In case of ShellShock, to complement the original detection, we developed patch detection capability as each vendor released vendor-specific patches. Qualys WAF (Web Application Firewall) always had the ability to block ShellShock type of attacks, and that capability came in handy as no change was needed in WAF to block ShellShock.
CP: Were any of these steps incrementally improved (e.g. by having gone through the necessary response for Heartbleed)?
AS: I believe we incrementally improved on various outreach methods with users to enhance their knowledge, answer questions and make Heartbleed/ShellShock/POODLE much more manageable for them. This includes dedicated microsites which gave tips on the quickest way to detect vulnerable hosts, how to monopolize our recent continuous monitoring features, and other techniques. Each microsite has links to dedicated blogs and community articles, and users can exchange their experience or ask questions directly. We also improved our ready-made templates, which saves users a few clicks during crunch time.
CP: Has POODLE’s detection made it into your free scanning product available online?
AS: Yes, POODLE is in FreeScan as well as in SSL Labs.
CP: As with any company in this space, Qualys has a responsibility for quality and thoroughness when it comes to detection and reporting of vulnerabilities. How is the process similar or dissimilar from the way you responded to BashBug vs. Heartbleed?
AS: The process by which we responded to Shellshock, Heartbleed or any other vulnerability is, for all intents and purposes, the same. Generally, the process includes the following steps – first, our vulnerability signatures team researches the technical details of the vulnerability, and second, it develops a signature that identifies the vulnerability on the impacted systems. Third, it performs quality assurance testing on the signature to ensure it works as expected with six sigma accuracy and fourth, then pushes the new signature into our scanning platform.
In addition to the technical side, we also strive to provide thorough documentation for our customers on how to scan and remediate vulnerabilities as serious and widespread as Shellshock and Heartbleed. In cybersecurity, timeliness is essential and we strive to produce new signatures within 24 hours of learning about the vulnerability.
I’d also like to point at that at Qualys, we feel that the security of the Internet is a shared responsibility, and oftentimes we provide free tools for Internet users to identify and remediate the most serious vulnerabilities like Shellshock and Heartbleed.
CP: In addition to scanning for the vulnerability itself, is there any other type of probing that would indicate the exploitation of Shellshock, even in the absence of a vulnerability (e.g. assuming that an exploited system was patched by bad actors to avoid detection)?
AS: Another common means of detecting a Shellshock attack or exploitation is through network signatures, whether this is automated by a firewall, intrusion prevention system or manual analysis of capture network traffic. In order to manipulate the Shellshock vulnerability, the attacker sends a specially crafted HTTP request to the compromised application that then executes the malicious code (Bash shell commands). This specially crafted HTTP request can be uniquely identified and blocked or identified for further investigation.
Once a machine is compromised, however, the job of the security team becomes much more difficult. Oftentimes, the attacker will patch the vulnerability they exploited in order to prevent another attacker from gaining access to the same machine. As such, the vulnerability may not appear to be active but the security team would want to look for other signs of a compromise such as connections to known command and control servers, new and odd ports opened and services and/or processes started, firewall and security product services turned on and off, non-IT initiated re-boots, etc.
CP: Is Qualys today able to detect all six known Bash Bug vulnerabilities? What was the process you followed to incorporate all six vulnerabilities into your testing capabilities as they were discovered and reported?
AS: Yes, Qualys can detect all six known Bash vulnerabilities that were recently released. We used several mechanisms for detecting the vulnerabilities, some of which included active probing of common CGI directories, authenticated tests of Bash and more generally identification of the Bash versions running within our customers’ networks.
In addition to identifying a specific vulnerability, we also identify vulnerabilities by patch release, as oftentimes one patch addresses multiple vulnerabilities, as is the case with the Bash bugs. This reporting and grouping process is done to make it easier for the IT operations teams to patch the vulnerabilities, as they only need to worry about one patch as opposed to dealing with 6 independent vulnerabilities.
CP: Given that much of the vulnerable infrastructure, is there a more precise role the user must play to ensure that Qualys scanning can penetrate network defences?
AS:Â The most accurate scanning results for the Bash vulnerabilities are done through what we refer to as authenticated scanning. This allows our scanners to gain access to the machine being tested and pull better and more accurate information. This is especially important with the Bash vulnerabilities, as it is extremely difficult to remotely identify all possible URLs on a website that might make calls back to Bash. There are also several cases where the exploit vector is more subtle and does not directly go after websites, but focuses more on impacted endpoints. As such, I would highly recommend authenticated scanning.
CP: What have you learned as an organization or as an incident response team, following this year’s Heartbleed/Shellshock discoveries? How has your process for developing intelligent testing product improved as a result, or has it been business as usual?
AS: Internally, as a security team, we have definitely learned the hard lesson that what was once considered to be secure software can very quickly become a huge risk for your organization. For Qualys, and what I would recommend to others, is to continuously scan your systems using authenticated scanning, maintain an up-to-date inventory of your hardware and software and then use this information to plan a quick and comprehensive response to newly released vulnerabilities. From each vulnerability, we as a company learn and develop more intelligent systems and code that ultimately allow our customers to respond more quickly than those attacking them.
In this case, we learned to better leverage our existing data to more quickly develop high quality signatures and to provide real-time situational awareness to our customers of the risks poised by these vulnerabilities. We were also able to leverage our cloud computing technology to better refine our signatures and enable our customers to scan millions of systems geographically dispersed across the globe to provide real-time situational awareness of their environment.
You do compliance testing for a number of clients of all sizes. Have your scanning capabilities been seamlessly incorporated into tests such as PCI scanning, or do some of these have to be run according to a precise set of instructions to avoid false negatives?
All of our scanning services, whether for compliance related issues such as PCI or for general vulnerability management, go through an extensive quality assurance process. One of the key things that any CISO will tell you is that they lack sufficient energy and/or staff time to accomplish everything they need to get done in a day. We don’t want to add to the depletion of this most precious resource by having security staff chasing down false positives – basically a vulnerability that doesn’t actually exist.
On the opposite side of that coin, we don’t want false negatives – this is where an actual vulnerability is missed. So, to avoid either one of these scenarios, all of our services go through extensive testing. On a positive note, however, our shared cloud platform enables us to share vulnerability and configuration checks across all of our services. Therefore, once a signature is tested and approved, it can be immediately applied to our vulnerability management, PCI compliance, web application scanning, web application firewall and other services.
As an aside, the author would also like to thank Jonathan Trull, the CISO of Qualys, who also participated in this interview.
