Welcome to Cyber Security Today. It’s Monday, December 27th. This is the Year in Review edition of the podcast. I’m Howard Solomon, contributing writer on cybersecurity for ITWorldCanada.com.
 |
 |
 |
With me for the first time together are commentators Dinah Davis, vice-president of research and development at managed service provider Arctic Wolf, and Terry Cutler, who heads Montreal’s Cyology Labs, a training and incident response firm. Hello to you both.
[The following transcript has been edited for clarity. To hear the full discussion play the podcast.]
Howard: Terry, you’re the guy who does a lot of incident response. What’s the year been like?
Terry: It’s been really, really busy … I’m starting to get more requests for proactive audits as companies are starting to get the impression that they should be taking their cybersecurity more seriously. But I’m also seeing a lot of victims of ransomware. I’ll give you an example of one of the more recent ones: One organization’s department wasn’t keeping their software up to date so they got hit with the Fortinet vulnerability that came out that twice this year, where there was a problem with a vulnerability and then there were credentials that leaked. Attackers launched a [Windows] Bitlocker attack and locked 400 computers. They demanded $10,000 per computer to get the decryption keys back. They used legitimate WIndows tools against the system and locked everybody out.
I’ll give you another example. We had an MSP [managed service provider] that got hacked, and they [the attackers] ransomed 40 of their customers. They got into their customers via TeamViewer [remote access software]. One was in the medical industry and they only had 40 computers. But when you see the cost of rebuilding the environment getting lawyers involved and such you know you’re looking at bills of close to $400,000 to recover.
Dinah: Even if you’re not going to pay ransom now, you need to have cyber insurance because of the cost of recovering even if you’ve done everything. Even if you’ve got the backups and you know you’ve done all the things, it’s totally possible for one person to make that click [on a malicious link] and you get ransomed. That insurance is key for you to be able to recover.
Terry: The insurance company actually told him next year, forget about it. We’re not insuring you.
Dinah: Of course, that’s what’s going to happen. But at least you’re good one time. It’s like accent forgiveness on your car insurance.
Howard: But that’s that’s great discipline for other organizations. You know you may have cyber insurance now, but if you get you the insurance will be canceled next year and that hopefully will be one way in which companies will take cyber security more seriously. The other problem, though, is I hear companies that don’t have cyber insurance are having trouble getting it, and if they can get it they may not have ransomware coverage.
Terry: I’m actually dealing with a customer right now. He just got hit with a ransom and they stole his data and it just so happens to be a [physical] security firm so he didn’t have the insurance. The insurance wouldn’t cover him because he doesn’t have things like EDR [endpoint detection and remediation] in place. He doesn’t train his employees. The basics of cyber security are not even in there.
Howard: How were they hacked?
Terry: Through a Microsoft Exchange vulnerability.
Howard: We’ll talk about that a little later. You mentioned a managed service provider that was hacked. How how did the intruders get in there?
Terry: We’re not sure how but we believe it was because of the TeamViewer exploit. One thing we’re noticing across multiple customers that we’re dealing with is the IT guys are getting firms in trouble. Because the IT guys have Security 101 level education, and they don’t know about the latest exploits that are available or all these flaws so they’re not setting up the customer correctly. With the MSP, when they got breached they were using really terrible passwords, like really garbage passwords. And they [the attackers] were able to gain access to the passwords and they logged into all 40 customers of the provider.
Howard: And, of course, probably no two-factor authentication among the IT employees.
Terry: No. In fact one of the IT guys said, ‘Oh, you can turn on 2FA?’
Howard: It’s been a huge year for ransomware and three of the biggest were an insurance company called CNA, which reportedly paid $40 million to get access back to its network, Colonial Pipeline in the U.S. and JBS Foods, which had to shut down some of their production in Australia, Canada and the U.S. Can you talk about those attacks?
Terry: In a lot of cases we’re seeing is that a lot of these organizations had flat networks, which means that they had no segmentation of their systems. If a computer gets compromised everybody can access everything and next you know the administrator password is stolen and the ransomware is logging into all the machines and encrypting it and they have no endpoint detection or response technology that’s programmed to stop these threats. And they’re wondering why this is happening.
Howard: One of the things that the Colonial Pipeline attack did was spur the Biden administration in the U.S. to take cyberattacks more seriously. And coincidence or not, not long after that attack the FBI and law enforcement agencies began going after ransomware groups more aggressively.
Terry: The firepower of law enforcement exists and I suspect that we’re going to start seeing a bit more of this because there’s not too much that a firm can do. Cybersecurity firms don’t have the power to go after these gangs. You have to get the telecom firms involved, you have to get warrants, there are jurisdictional issues.
Howard: Dinah what’s the year been like for you at Arctic Wolf?
Dinah: One of the most interesting things happened just weeks ago is the Log4j vulnerability. It’s a Java-based logging library that the Apache software foundation gives away for free. And it’s pretty good at what it does. It’s used to log information like when you log into a website and you put your username in. The other thing to remember is this thing’s been around for like 10 years so lots of people are using it all over the globe. So on December 9th we find out that Log4j or Log4Shell has a zero-day. An adversary can exploit it by submitting a very specially crafted request to a vulnerable system, and then that system will arbitrarily run that code and the adversary can take full control over the system.
There’s an analogy I read on Twitter: Imagine there is a screw and it’s used in all types of door locks. The reason this screw is being used is because it’s free and it does the job really, really well. But it turns out that it can be easily broken. Log4 is the screw. The problem is that it’s not known which locks actually use the screw because it’s embedded inside and you can’t see it from the outside. But as time progresses we’re finding out it not only locks doors but it’s also locking cars and vaults and other things. Some of the locks might not be easily accessible because they’re in a house or a building, and it has other locks and security around them. Some locks can be replaced by the owner, just like you can update Log4j yourself. But sometimes you’re dependent on the manufacturer so you got to wait for it — just like you’ve got to wait for a software fix. The difference between the real and the digital world is in that the digital world attacks on the lock can happen from anywhere, not just at the door itself. The flip side of that is that the screws can also be replaced from almost anywhere, and very quickly if we can get there.
Right now attackers are trying to do as much reconnaissance as possible so that they like embed stuff in there to launch later. I think this is a little bit of a lull before the storm. Lots of people are getting fixes out but there’s always going be the people that don’t get it out quick enough.
Terry: I’m dealing with large environments, scanning tens of thousands of machines. And you know, Log4j and these other components might not be running but when you run a vulnerability scanner you detect that the JAR [Java archive] files are actually on the system but may not be active. So there might be something where in the future [threat actors] they can just launch an attack against the sleeping JAR file.
Dinah: At Arctic Wolf we woke up Friday morning on the 10th going, ‘Wow. This sucks.’ We started right away looking in all of our client environments: What are they running, doing scans and everything, and by noon we had scanned all of our clients. By the end of the day we had contacted anyone we thought had an issue. We sent out a larger notification just to all of the clients with recommendations on what they can do. One of the interesting things that we saw in scans was that in some instances we were seeing threat actors dropping [cryptocurrency] coin miners upon compromise. You don’t always think about it, you always they’re going to go for an attack, but another way for them to use your electricity for coin mining. We wrote a vulnerability checking tool that our clients could run in their networks to find things. And by last Friday we published that to everyone — you don’t have to be a [Arctic Wolf] client anymore. We put it on Github. It’s open-source. That was just for our clients. Internally we also had to go through all of our stuff and see what are we using in our software and how are we going to mitigate all of this.
Howard: The cyber groups from Canada, the U.S., the U.K., Australia and New Zealand issued a very helpful compilation of advice for IT people who are struggling with this … They say if you detect that you have Log4j in your systems, patch but assume that you’ve been compromised and start looking for suspicious activity. Dinah, how long do you think that we’re going to see threat actors exploiting Log4J vulnerabilities?
Dinah: Years. Think about how people use Windows 7 for ages and ages and ages [and not upgrade].
Howard: There’s been efforts to get software developers to create a software bill of materials to go with their applications So IT departments can more easily track down problematic components. Would that have helped track down Log4j use? Would it help in any issue?
Dinah: You could try but you’re going to be searching through hundreds and hundreds of pages of documentation for every piece of software. You use one [software] package that package pulls in maybe 50 different things and then that’s only one of the 50 packages you’re pulling in. It’s just so embedded, and it’s even hard for the developer to always know everything that’s in there because sometimes things are pulled in just via a binary compile and so they’re not easily listed. I don’t think that’s going to be the solution. It’s like trying to name all the pieces of sand on a beach. It’s not going to work.
Terry: It’s so embedded in other software that developers don’t know what the other guy has coded, and you don’t have the source code anymore. So some of this has to be tracked down manually.
Howard: I want to talk about supply chain attacks seen this year: Instead of directly attacking companies one by one the threat actor compromises a platform used by many so that a number of customers can be exploited. In December 2020 the world was stunned to learn that the security update mechanism of SolarWind’s Orion platform was hacked. This is important because Orion is a network monitoring suite used by many major companies and governments. Compromised updates were downloaded by 18,000 thousand organizations, and, luckily, perhaps only 100 of them were hacked — but some included government departments. So much of the first half of this year was spent by companies scrambling around to see if they’d been victimized in some way by this attack. The attack was so serious that the U.S. government sent out an alert saying that the threat actor behind this was likely Russian in origin. And this was such a complex attack that it could have dated back to the fall of 2019. The infected updates didn’t get distributed until March of 2020, and then was discovered in December of this year. Also, vulnerabilities were found in two enterprise file transfer utilities. Coincidentally one of them was from SolarWinds, the other from a company called Accellion. Victims of the Accellion FTA vulnerability included the Shell Oil company, Bombardier and the City of Toronto. According to one report I saw out of about 300 users of the software a hundred had some measure of impact from the attack, with another 25 of those experiencing significant data loss. And then there were vulnerabilities in the on-premise version of Kaseya VSA, which is a remote monitoring and IT management platform used by enterprises and by managed service providers. The REvil gang used this opening to deliver ransomware to a number of organizations. Gang, what do you make of these and other supply chain attacks?
Terry: I think it’s really devastating. You rely vendor software to manage your environment, but then you’re at the mercy of them securing their code. Meanwhile attackers have got access to hundreds of environments … It’s very hard to detect, especially if they’re using tools like Cobalt Strike, where it’s programmed to hide its tracks in your environment and the only way to spot these tools is to have proper network monitoring technology that looks for beaconings.
Dinah: I also found Kaseya to be really bad. The vulnerability that was found in their code was kind of a no-brainer. It just wasn’t that difficult to actually exploit, and they knew about it for some time and didn’t get fixes out fast enough. But that’s not really why I think it’s one of the worst. It’s one of the worst because they had MSPs using their stuff. So it was like a double-level supply chain attack — the attackers went through Kaseya to get to the customer but in many cases the customer was an MSP also using Kaseya to talk to their customers. So Kaseya can say they only had a certain number of customers affected, but it’s not including that cascading effect [of customers].
Howard: There are two questions here: What does it say about the cyber security of application developers because in SolarWinds the attacker had to get into the SolarWinds Orion update mechanism to begin with. The other question is what does it say about application development and their ability to find vulnerabilities before the code goes out to customers?
Terry: We’re dealing with that already. We work with it with a bunch of companies that deal with in-house development and we’re finding that the developers don’t necessarily have the proper training to develop with security in mind. So they’re just putting out products fast because they want to get to market as quickly as possible. But then when we come in and start doing a software development lifecycle on their platform we’re noticing a lot of flaws that should have been fixed from the start. Now it’s costing them more money to redevelop that code and get it and get done right. And when they do it breaks compatibility with the past software they’ve already released.
Dinah: There’s a lot to say about Security By Design and Privacy By Design. If you are starting a company or you are starting a new package right now, don’t ever think about adding security later. It’s always going to cost you more. It’s always going to be more of a pain in the butt and you’ll never be able to secure it in the same way than if you started from scratch with security and privacy in mind. Blackberry is a great example of that. Full disclosure: I used to work there on the cyber security team. We did really think security into the mobile phones first, and then added features on top.
Howard: Another big news story this year was the attack on Microsoft Exchange by a group Microsoft calls Hafnium. Terry, tell us about that.
Terry: Back in March there were tens of thousands of organizations in the world running on-premise Microsoft Exchange servers that were exposed to these zero-day vulnerabilities. In fact I think there were three or four [vulnerabilities] that came out back to back. And once the attacker gained access to the server they were able to launch web shells that would allow them to steal data from the company and even launch a ransomware attack … We’ve seen IT guys tell the customer, ‘You don’t need to run EDR, or endpoint detection or response technology, on these servers because it slows it down.’ So we’ve had cases where just this one Exchange Server wasn’t protected. They [attackers] gained access to it and they were able to steal data because some of the computers didn’t have EDR on it. So I find the IT guys are getting folks in trouble. They’re not patching properly.
For all listeners [using Microsoft Internet Information Services (IIS)], look inside your InetPub Folder to see if there’s a bunch of aspx files in there. Those are the malicious ones, like web.aspx if you see these files. There’s a good chance you may be compromised.
Howard: I want to talk about three Canadian cyber incidents. First is from the biggest incident of this year, the hack of Newfoundland and Labrador’s healthcare IT system. It’s probably the biggest health network hack so far in Canada. The Newfoundland government is being very sparse on its information. It’s still calling this a cyber attack. One of my sources says it’s ransomware. Newfoundland has not said how this attack started. The other thing, though, that we do know is there were years of data involving some patients and employees. That attackers were able to access raises questions about data retention — and never mind that this data wasn’t encrypted — but it certainly raises issues about data retention. Are you keeping data longer than you really need it?
Terry: In my experience with healthcare, it’s a mess because there’s tons of legacy technology that’s lying around in there and they’re forced to keep it because maybe the software that they’re running on it doesn’t run on newer versions. For example, software that controls the door locks for security. Some software can’t be updated, so they’re forced to have this legacy technology — and, of course, it’s end of life. There’s no more updates for Windows XP or Windows 7. Maybe newer hardware can’t support the software. In order for them to replace it would cost them maybe $200,000. Not many people [in healthcare] have a proper inventory of what’s running in their environments. That’s the biggest piece I see right now is is they don’t know what’s there. They have no visibility.
Dinah: The thing is, it’s healthcare. So you have to keep the health data of people around for a while because the doctors need it. In the health industry it’s a little harder to purge your data.
Howard: Another big incident was the Canada Revenue Agency had to force 800,000 taxpayers to reset their online passwords after discovering that an unauthorized person was accessing accounts. There was no explanation of how that person had gotten hold of other people’s passwords and it certainly leads to the suspicion that the attacker assembled a list of stolen and reused passwords. This is an old and continuing topic.
Dinah: There was no evidence that were security holes in the CRA software. So it seems to me it was probably a dictionary attack or password spraying. People, do not reuse passwords. Especially do not reuse passwords for your government website logins, like the ones that deal with your taxes, or your bank account.
Terry: That’s where we’re seeing. A lot of people are using the same crappy password everywhere like ‘123’ or ‘12345678.’ …. It all comes down to security and education awareness, and people don’t care about it until it’s too late.
Howard: The last thing I want to touch on was the attack on the Governor General’s office. The Governor General of Canada is the head of state and doesn’t sit in Parliament. The person who holds the job may or may not have high security clearance but the government isn’t saying if documents were accessed or if email was accessed. It could be serious stuff.
I want to close today with my favorite news story of the year. It was a report of a student who inadvertently caused a ransomware attack at European medical research institute where I think he was an intern. As a student he wasn’t entitled to a full copy of one of the applications that the institute was using, so he went hunting on the internet for a free copy and he found an illegal free copy that was infected. So when he connected his computer to the internet to the institute the malware got downloaded. This is why IT managers have gray hair.
Dinah: My favourite story was about the AnoM app. The FBI and the Australian federal police had seized a secure phone that was being used [by a crook] and cracked it. The police were sitting having beers after they’ve broken this case, thinking ‘Wouldn’t it be great if we had our own app that we sold to criminals as a privacy app and we could just pull everything they were saying.’ and then they actually did it and made hundreds and hundreds of arrests. And they funded the software by making criminals subscribe to it. I really think this is going to be a movie.
Howard: I want to close by asking you for advice to CISOs — chief information security officers — about what you learned this year
Terry: The biggest thing I learned this year is that the IT guys are getting folks in trouble because they only have Security 101 [training]. When you work with outsourced cyber security folks, we’re at level 401 or 501. So we’re there to compliment the IT department because we’re going see stuff that they’ve never seen before, and can help them implement our changes as quickly as possible. So we need to work closer together. Get some auditing, get your inventory in there. Work together.
Dinah: I think it’s evident that point solutions are not going to help you very much. There’s way too many vectors for hackers to come in. It’s really about managing your security operations and looking at everything holistically together so that you can paint a bigger picture of what’s actually happening in your environment. That is one big step. But if you don’t do awareness training and make sure your team and your people know how to handle things coming in you’re going to be just as open [to attack].
Howard: My prediction is not enough IT and cyber security managers will have disciplined cybersecurity programs, and that’s what you need: You need discipline in order to stop threats and so unfortunately we’re going to see more breaches next year.
Terry: I think we’re still going see organizations that don’t have a proper inventory of what’s running in their environment. They don’t have enough logging in their systems turned on. The biggest problem is they have an undervalued security budget. The biggest problem I see is companies say, ‘Ransomware will never happen to me. We’re too small.’
Dinah: I think we’re going to start to see a little bit more Linux-based vulnerabilities. Pretty much anything a live website is on today is running in the cloud. A lot of that stuff’s running on Linux. So if attackers are going to want to start getting into those types of things we’re going to start to see a lot more Linux vulnerabilities. Also, the insider threat is coming. Insider threats are huge problems right now. Companies need to move to more of a zero-trust approach.
Howard: Finally this is the last podcast with Dinah Davis. She has been with us since our first Week in Review podcast on October 16, 2020. Thank you for being patient with me and thank you for your opinions and advice.
Dinah: I have loved my time here. I think I am more up-to-date on the world of security than I would have been without it. Some of the stories I wouldn’t have read about or seen. So my knowledge is much greater because I was doing the show, and I had a lot of fun.