mnemonic security podcast
The mnemonic security podcast is a place where IT Security professionals can go to obtain insight into what their peers are working with and thinking about.
mnemonic security podcast
n-days
In this episode of the mnemonic security podcast, Robby is joined by Dustin Childs, Head of Threat Awareness at Trend Micro’s Zero Day Initiative (ZDI). Dustin explains the ZDI’s role in purchasing and analysing vulnerabilities to provide early protection for customers and how zero days – previously unknown vulnerabilities – become "n-days" once disclosed or patched.
The conversation highlights the critical importance of timely patching, the risks posed by bad patches, and the concept of virtual patching as a defence strategy. Dustin also delves into attack surface monitoring, the evolving threat landscape, and the ongoing challenges of balancing security and usability in modern networks.
From our headquarters in Oslo, norway, and on behalf of our host, robbie Perelta, welcome to the Mnemonic Security Podcast. So, as I record this message to you, my beloved podcast friends, it is Thursday, the 30th of January 2025. And my iPhone is currently upgrading itself to iOS version 18.3. 20 security patches in this firmware update, apparently, and I can't tell you how many apps of mine that needed updating because it's unclassified.
Speaker 1:On the topic of embarrassing, though, I was going to make a joke about Vendor of the Week X and their critical CVS score 10 vulnerabilities, but before I did, I consulted my LLM, who actually changed my perspective on things a bit. As it's said, so philosophically, having vulnerabilities isn't the real problem. How vendors handle them is what truly matters. Why it's not necessarily embarrassing. It says All software has bugs. Even the most well-tested code will have flaws in a complex system.
Speaker 1:Security research is evolving, hashtag AI and many vulnerabilities are reported responsibly and patched before they are actively exploited. Fair points. When it is embarrassing, however, if a vendor keeps having the same type of vulnerabilities, like authentication bypasses, in every other update, slow patching, putting your customers at risk, and when a vendor downplays or tries to hide their mistakes instead of fixing them. I have a few vendors in mind, especially on the OT side of the house. Anyways, what do I know? At the end of the day, I'm only responsible for my iPhone, mac and browsers, I guess. But our guest for the day has a more nuanced way of looking at the issue. Dustin Childs, welcome to the podcast.
Speaker 2:Thank you. Thank you for having me.
Speaker 1:Happy to be here. So where have you been in the past couple weeks or months?
Speaker 2:Well, I've been in Ireland for a couple weeks for our latest Pwn2Own, which is a contest that we hold where we invite some of the best hackers around the world to come in and hack the latest devices or software and see what is what. What is the security of what we consider to be our daily use?
Speaker 1:Cool. Does anybody get in trouble at these events?
Speaker 2:No, Fortunately, we don't get in any trouble At least we haven't in quite a few years. So it's all white hat hackers who are coming in and they're doing this, you know, with permission. So they're doing this on closed networks, they're doing it in a controlled environment, but yeah, it's a lot of fun. I mean, we had one hacker come in from the NCC group who hacked the Samsung Galaxy, so that was quite interesting. He was very excited when he was able to do that. That was worth $50,000. Oh nice. So not only does he get the device, but we pay them as well. Yeah, cool.
Speaker 1:So what is your relationship with the Pounda owned?
Speaker 2:Well, I'm the head of threat awareness for Trend Micro's Zero Day Initiative, so part of my job is to understand exactly what's going on, what the threat is, so I can kind of reach across all of Trend Micro's threat research teams and see what's going on where and understand what the threat is to our customers and then tell that story to as many people as I can.
Speaker 1:Cool. Well, it looks like you're in the right place. I will do my best, Dustin Cool. So the Zero Day Initiative how long have you been there since beginning, or when do you get into that?
Speaker 2:So I actually joined the Zero Day Initiative in December of 2014. So I've been here almost 10 years now and doing this for quite a while. The Zero Day Initiative has been around since 2005. So we're actually approaching our 20th anniversary next year, but I've been around for about half of that, and prior to that, I was actually at Microsoft and I was at the Microsoft Security Response Center they're the people who do Patch Tuesday so I was receiving bugs from the Zero Day Initiative to be patched by Microsoft. So I was receiving bugs from the zero-day initiative to be patched by Microsoft.
Speaker 1:So I've been involved with ZDI since about 2008.
Speaker 2:And I'll tell you, it's a lot easier on this side. It's a lot more fun on this side, to be sure, too.
Speaker 1:Cool, Just to go back in time travel. What was like when Microsoft released Patch Tuesday? What was that like back then?
Speaker 2:Well, back in 2008,. I mean, it was a big deal, you know, we'd get there early, we'd have breakfast, and then at 10am Pacific time, you know, we check everything and make sure it was good to go, and then we'd release it, and then we'd hit a song, like they would play a song in the hallway, like really loud. It was usually like classic rock, I think one time it was Klingon, so it was, you know, a celebratory thing that, like once a month, we were trying to make the world a better place. So it was. It was a big deal back then, you know Cool.
Speaker 1:Wow, and and fast forward to today. It's still on Tuesdays, though.
Speaker 2:It's still second Tuesday, still patch Tuesday. I don't know how they do it if they still play music, but yeah, definitely a lot more bugs they have to deal with now. I mean, they're patching LLM bugs, they're patching open source bugs, which I never would expect Back in 2008,. Steve Ballmer was like no, we'll never do Linux, and now they're patching Linux bugs. So it's definitely changed. It's definitely a wild time.
Speaker 1:Yeah, and I guess you of all people can confirm that Microsoft does actually care about these things A lot of people just throw shit at them, but they have a lot of good people working there.
Speaker 2:Yes, they mess up a lot, but they do have a lot of good people working there. They have a very tough problem and I always root for them, because Microsoft doing well is better for the world but I'm often disappointed in them as well, so it's kind of a you know that's where we're at.
Speaker 1:Yeah, don't want to use love-hate, but you know respect. So I mean, I've had the pleasure of speaking with a lot of your colleagues, even those that have come from that work with the Zero Day Initiative. But for those that haven't heard about it, what is the Zero Day Initiative and Trend Micro for that? Even though if you haven't heard of Trend Micro by now, I'm very surprised. You're listening to this podcast and haven't heard of Trend Micro. Yeah, yeah.
Speaker 2:So the Zero Day Initiative is the world's largest vendor-agnostic bug bounty program, and vendor-agnostic are the key words there. So, like Microsoft is a bug bounty program, they buy Microsoft bugs. Hackerone runs bug bounty programs for companies. They buy the bugs that contract. You know the companies contract them. They buy the bugs that the companies contract them. We buy bugs in just about everything. So we buy Microsoft bugs, adobe bugs, google bugs, apple bugs, dell, cisco, ibm you kind of get the point. So we buy across everything and what we do with that is we turn those bugs into threat intelligence and filters for our products. So that's where the zero-day initiative part comes in. So if you're a trend micro customer, you're getting protections on average about 90 days ahead of everyone else because we're buying these bugs ahead of time.
Speaker 2:And people will say, well, zero days are like so rare and so expensive. I don't care about zero days. But what people forget is zero days become in days, so it's something has been patched in number of days. Those are incredibly common. So once like patch Tuesday happens, it's exploit Wednesday and that's just the common thing. So those zero day protections go in and are still there when you're rolling out your patches, when you're testing and we know people do not patch anywhere near fast enough, and it's kind of like saying, oh yeah, mr Dentist, I floss, no, you don't. You don't patch fast enough, you don't floss. You're like these are lies. We know these are lies.
Speaker 2:The average that we're seeing for critical is 90 plus days. However, for critical bugs, they're being exploited within 48 hours. So that's one of the things where we see this stuff and we know you're not patching fast enough. But those protections, if you're a Trend Micro customer, then you get those ahead of time. So that's really what ZDI does, and we enhance our own research too. About half the bugs that we talk about so that we get patched are from ZDI researchers. So we have our own group of researchers who are finding and reporting bugs all the time. So, yeah, it's a lot of fun. So it's really interesting to see the information that comes through the program, to see what type of bugs there are, what gets patched, what doesn't get patched, because one of the things we do is we hold people accountable. So after 120 days, if you don't patch it, we release a certain amount of information to make sure that vendors are held accountable.
Speaker 1:Very, very, very cool. I have a lot of questions I just wrote down, but that last one there you said you release a little bit of information, right, I've always thought why do security researchers release exploits? I get that you have to hold the account like the vendor accountable, I I really understand that. But then again it's like what do you think about like those researchers out there? Obviously I would assume that you don't do, you don't release everything that that they can make like an exploit, but what do you think about those that do like what is? Yeah, I know there's an interesting conversation there. Yeah, yeah the.
Speaker 2:The disclosure debate has been held for years and will probably be held for years to come. Obviously, when it comes to full disclosure, like what you're talking about releasing exploit code and everything, I don't agree with that personally. But I also understand that because there's a lot of frustration behind that, where you have vendors who are not being honest, brokers who are ignoring, researchers who are doing whatever and the researcher feels that their only recourse is to release everything. That's unfortunate for a lot of reasons, and the main reason is because it's putting customers at risk. So when we expose something, what we do is we release a certain amount of information that we think is useful to defenders so that they know what to do to protect themselves from this unpatched bug, but we don't release like proof of concept code, we don't release exploits. You know we don't do that sort of thing.
Speaker 2:But for those who do, obviously I disagree with that. But I also understand their perspective, where there's a lot of frustration behind that, a lot of distrust behind that and a lot of hurt, and that's why I hope that the Zero Day Initiative and other programs like ours can be kind of a safe harbor for those people. So if you're an individual and you go to a huge company whether it be Microsoft or Apple or Adobe they might not take you seriously because you're an unknown quantity. But if you sell something to the Zero Day Initiative, then the ZDI is representing you and then you're being represented by a known quantity and you're taken seriously more often. So that's one of the things we hope to do with our program is give people the opportunity to be financially rewarded Because I mean this is hard work, I mean this is a big deal finding these bugs but also be represented in a way where they're taken seriously and their research is handled.
Speaker 1:How many security researchers do you have under the ZDI umbrella?
Speaker 2:Well, over the course of the program over the last 20 years, we've had over 10,000 researchers participate. At any given time, it's, you know, three to 500 who are really active.
Speaker 1:And are those like similar people that would be on, like what, HackerOne and those sort of platforms as well?
Speaker 2:Yes, very similar to that. I mean, there are people who are researching, there are people who are disclosing, sometimes directly to the vendors and sometimes us, or maybe exclusively through us. One of the things that we do is we kind of have a frequent flyer program, so like the more you submit, the higher you can earn. So our top level gets 25% bonus, so if we pay you a thousand then it becomes 1250, that sort of thing. So we encourage you to report everything through us. But, yeah, it would be the same type of people who are doing research, usually as a side gig, but we do have a couple of people who do it-time.
Speaker 1:And I mean, I don't I'm in deep water here, but like when you're going to be doing this type of research? I know that, like I was just at an industrial security conference and you have guys there that are, you know they're like I want appliances, I want your stuff so I can take it apart, break it open, what are some of? Like the? Yeah, I feel I was trying to remember a conversation I had there, because sometimes that's illegal, right, taking somebody's appliance, breaking it apart, understanding how it works, even if it's in the name of security research, right? So how do? Yeah?
Speaker 2:Well, there's illegal and then there's against the terms of service. So I would say it's against the terms of service in a lot of cases. Legal is a legal question. That I'm not a lawyer, I'm not an attorney, I'm not going to pretend that I know this stuff. So the legality of it is one thing. But, yeah, you definitely can violate the terms of service. In, especially in the industrial control systems world. They hide behind that a lot and I intentionally use the word hide because they're like oh, you can't test our stuff because you're not licensed for it. It's like we've had that happen where we've tried to report bugs to companies and they're like well, we're not accepting your bugs because you're not a licensed customer. Therefore you cannot have reported this bug or you could not have found this bug otherwise. And it's like it's in your trial software. We found it that way and in those cases those.
Speaker 2:Yeah, those are one of those cases where we have to disclose publicly, because they will not acknowledge the bug without some public pressure.
Speaker 1:Right, and that is unfortunately the world we live in, isn't it? Mm-hmm? Is the term virtual patching? Is that a Trend Micro thing, or is that a branch standard?
Speaker 2:term. Virtual patching is definitely a trend micro thing, but it goes beyond just trend micro and it's one of the things that the zdi does is we take these bugs that we purchase and we turn them into filters that become virtual patches, specifically for the tipping point ips device. But yeah, it's, it's a very that's what I'm referring to when I say you get these protections ahead of time, you get these virtual patches ahead of time, and that's really what it comes down to. It's not exclusive to Trend Micro, but we definitely, I think, do it better than just about anyone. Of course, that's my very, very biased opinion.
Speaker 1:I've only heard of the term virtual patching through Trend Micro and I think that it's kind of it's just surprising, right, if, like, if you know that, like, for example, end days, right, and just for those listening, is it correct so that if nobody's ever used it before, if it's never been observed, it's a zero day, but as soon as it's released, then it's an end day because it's out there?
Speaker 2:and you just have to patch it.
Speaker 1:Yes, as soon as it's patched, or it they. Is it just me that's not up to date on vendors that do this virtual patching? But just don't call it virtual patching, they'd probably call it something else.
Speaker 2:Well, there are a couple other vendors. Zero Patch, I think, does it. They release micro patches independent of the vendors. But for most the IPS system itself is not a very common thing. The a lot of people don't do that anymore, not anymore, uh. So tread micro with the tipping point device really does that very well. It does it very, very fast too. I mean there's the line speeds that they can uh handle. Is is really really crazy uh. But most people don't do that so they don't have something sitting in line that can be a virtual patch device. Then it becomes too difficult for them so they just don't try.
Speaker 1:Yeah, and people don't want stuff in line just because they're scared that it's going to like stop the network, though, right?
Speaker 2:That's right. A lot of people don't. But with the tipping point you can either have it in just observe mode or prevent mode. So yeah, but it's. It's pretty interesting how fast it goes and I know the team down in Austin. They're doing some amazing work with that device and it's. It's a really really cool. I actually like even before I was at Microsoft, I was a defense contractor for the U S air force and we had tipping point back in 2002. And it was the fastest thing we'd ever seen. It was still. It was really good even back then and it's only gotten better. I can't remember the top speed now, but it's. You know multiples of gigs that it can go at fine speed, so it's really crazy.
Speaker 1:The concept of like attack surface monitoring. I've heard that so many times. I don't even know what it means anymore. It depends what vendor is pushing it. What does that mean in your mind, right? I don't even know what it means anymore.
Speaker 2:It depends what vendor is pushing it. What does that mean in your mind, right? So in my mind it means understanding what you're trying to defend. So that's one of the biggest challenges that defenders have. The network these days is so complex, there's so many devices on it, there's so many targets on it.
Speaker 2:Understanding your attack surface is one of the hardest jobs that you have as a defender to make sure that you're actually defending it, Because there's more than defending, than just patches. Right, you have a bunch of different things that you can use to defend your network. So attack surface monitoring is really understanding not only what you have to defend but what the risk is from it. So, for example, the risk to all your Windows devices goes up on the second Wednesday of every month because you just had the second Tuesday where all the bugs were released. Patch Tuesday, exploit Wednesday so understanding that.
Speaker 2:But what if there's something that happens, like with your Cisco switches, like they put out a patch, and you need to understand that this is now a higher risk and you need to look at these more carefully, even if you can't patch them, even if you know that, okay, our next patch is going to be this Saturday, because you know we can't do the network outage that it takes to update this. So just understanding what is really the risk in your network at any given time, especially understanding what is on your network at any given time, and that is really difficult to understand. And I've talked to people going back to industrial control systems, talked to people with OT devices. It's like we can't find it. We know it's on the network, we can ping it, we can talk to it. We have no idea physically where it's at yeah, physically yeah, yeah.
Speaker 2:So it's like we don't know where this thing is and we're trying to patch it, we're trying to do these things and you have to physically hit a reset button or something and we can't find it. So it can be incredibly difficult and I've talked to multiple people in multiple environments going yeah, that's a real thing that we just can't find this device. So how do you defend something that you physically can't even put your hands on, and that's an attack surface and that is really, when it comes down to the ASMR, the management of it. Where is your risk? How can you defend it? What can you do and where do you prioritize? I always say that everyone in this industry is three unders You're understaffed, you're underfunded, you're under pressure and you have to do resource management. Because you don't have just resources, you can throw at the wall and just see what sticks you've got to be. You got to be mindful of what you're doing.
Speaker 1:So that, to me, is attack surface management and I mean, obviously you're might be a little uh little uh bias towards one side here, but like what, like, what does that look like in like a daily perspective, if you're under on three areas, right, so how would? Yeah, what's your sort of insight into that?
Speaker 2:Well, I think it comes down to a lot of what tools you're using and what techniques and understanding what you have on the defensive side. And obviously patching is one thing, but you have network segmentation, you have your firewalls, you have your EDR, xdr, you have your IES, ips, you have whatever you have and I'm going to keep this vendor neutral.
Speaker 1:Thank you.
Speaker 2:Whatever you have, how are you observing that and how can you best utilize those devices? How are you keeping up to date with information? Where are you getting your news from, like in, you know, in those five minutes where you're not getting alerts where do you read about the latest threats to your system? So that's the thing to me is like just understanding what tools you have available, that you've paid for, that you use, and how do you get those to their maximum value and are you keeping those updated. So for a lot of people, like I always said, an antivirus with out-of-date signatures is worse than no antivirus at all.
Speaker 2:So are you sure that you're update on all of your signatures and your filters and your patches and that sort of thing? So that, to me, is what the day-to-day looks like, is coming in, understanding my tool sets and using it to the best of its ability, whatever that tool set may be, whether it's something simple like Snort as an IDS or your EDR, or just even network segmentation, so that if you do have some sort of outbreak, you can take down that segment quickly and not have it spread to your entire enterprise.
Speaker 1:Right, and one thing I want to touch on there. You like the concept of staying up with the latest threats, right?
Speaker 2:It's difficult.
Speaker 1:What does that look like? I mean, you have a research team dedicated to that, so I guess that your point of view would be that we have people making sure that is. But yeah, just what does that look like from their?
Speaker 2:point of view. From my point of view, it's an RSS feed with like way too many things popping off, you know, every hour, but it can be a lot of things. Obviously, subscribe to whatever you can, so, whether that be newsletters, an email list from vendors or RSS feeds. So subscribe to those news sources that are going to be reliable, but also where the signal to noise ratio isn't crazy, so you're not getting thousands of alerts for really minor stuff. There's also social media that you should follow people on, whether it be Twitter or Mastodon or now Blue Sky. There's a lot of good folks on Blue Sky these days, but it's overwhelming, jake, and from this fire hose.
Speaker 2:So you almost have to take and figure out which taps can I turn off and figure out what's really important, especially to your network or to your enterprise or whatever.
Speaker 2:So, for example, if I have an entirely Windows enterprise, I'm going to turn off all the Linux stuff because at that point I don't have the resources to handle it.
Speaker 2:And conversely, if you're a Linux shop, turn off a bunch of that Windows stuff because you don't really need to care. So definitely focus on and this is another case of like if you're running Apache web servers, you want to make sure that you're getting all the Apache news and you can ignore these other web servers, the news for that, so it's not as important to you. But it's again a case of understanding what you're trying to defend and then focusing your news sources on the places that are reporting things in those things, reporting bugs in those areas. I should say so, yeah, and it's difficult because there's so much news, there's so many bugs, there's so much stuff. And it is easier for me because I have a whole research team telling me what's important. But I also have a few trusted reporters and a few trusted news sources that I follow that are going to tell me the most important things that I need to know and then work backwards from there.
Speaker 1:Do you agree that, like you know, when you hear about a patch these days, it's not because there's a patch available, it's because it's being exploited, because there's like so much noise about, like the only thing that's pop up on LinkedIn or Blue Sky or whatever is the stuff that's like actively being exploited? Is that sort of the case?
Speaker 2:Yes, yeah, I mean, the squeaky wheel gets the grease, the squeaky patch gets exploited. So yeah, there's usually 50 from Adobe every month. There's usually around a hundred from Microsoft every month. There is 20 plus from Cisco. There is all of these things that are coming out every single month and it's hard to pay attention to them because Only about 5% of those actually get exploited.
Speaker 2:So you have to really focus on which ones are the most exploitable, and it's really really hard to figure that out. Microsoft tries to do it, but they don't do it very well. They haven't done it very well for years. Nowhere else even tries. So kudos to them for at least trying. So yeah, so a lot of times the ones you hear about are the ones getting exploited. So it becomes very difficult to weed through all of this noise, to prioritize on what is being exploited and what will be exploited. Because it's easy Microsoft will tell you every Patch Tuesday. It's like these two bugs are currently under active attack. These three are publicly known. Here's the other 97. But out of those 97, some of those are going to get exploited, and which ones? Microsoft gives you a little hint. It's like we think this is more exploitable, we think it's less exploitable. They're wrong all the time. Again, kudos to them for at least trying, but it's really hard to guess exploitability.
Speaker 2:All you can do is look at this and go, wow, this is a CVSS 9.8. It affects all of the servers I have on my network and it's wormable. I need to pay attention to this one, even if it never gets exploited and there was a couple of bugs like that in this last patch Tuesday that are incredibly severe, incredibly applicable. Will they get exploited? I don't know, probably not just running the numbers and playing percentages, but is that a risk that you're willing to take as an enterprise defender? I'm not. I'm not willing to take that risk. I'm willing to work a few extra hours to make sure those get patched. But yeah, that's the sort of thing.
Speaker 2:It's like you have to look at the severity of the bug. You have to look at the applicability to your network. I mean, maybe it's a super severe bug in WordPress. You go, hey, I don't use WordPress, okay, great, then you don't need to worry about it. But if it's something that you are using, hey, it's a super severe bug. In Apache. It's like, oh my God, I have like 14 Apache servers. Then, yeah, you need to take the time to test and deploy that update. But again, it comes down to knowing what you're trying to defend and that is a difficult scenario I understand. But that's why I say be ruthless in your asset discovery. And again, I use the word ruthless intentionally there, because you have got to get out and find everything that you've got.
Speaker 1:All the shiny, cool new stuff ends up boiling down to the bare bit the necessities. Uh, in the foundation. Yes, which is why it's called the foundation, I guess. Um, have you ever like some of the smarter clients who've seen doing this like that, just knowing what you have, all those sort of things like it? Have you ever heard of, like a real, a customer that had like a really novel way of going about that? Or is it just sort of boring foundational stuff that's not really fun over a podcast?
Speaker 2:It's. It's really boring foundational stuff that's not really fun. Over a podcast there I have seen some programs that do passive discovery for their assets. That's very interesting. But passive OS fingerprinting has been around for at least 25 years that I know of and oh wait, it's not called Rumble anymore HC Moore's new company.
Speaker 2:They do it very well from everything I've seen and I can't remember the name of it. I apologize, but their entire product is about asset discovery and apparently they're doing it so well that they can tell you the version of whatever you're using. If you've got EDR deployed, they can tell you the signature level of the version that you're running, even beyond just that. You're running EDR. So I think that's pretty novel.
Speaker 2:But of course that's a third-party product that you're going to have to pay to use, are incredibly restrictive on change management and incredibly disciplined on patch management, which can get in the way of user interaction, and it's always that balance right Of. The users want to do everything, they want to have all of these access, they want to do this, they want to do that. Security says no. There's got to be a balance in there someplace of letting your users do what they want to do and actually be able to do their job. Whatever that function is, it's security saying no, you can't do that, you can't have the latest shiny thing because we haven't tested it yet, you know, and so on and so forth.
Speaker 1:It's hard for them. Poor IT team, it's difficult.
Speaker 2:I mean yeah, I mean, security is difficult. I mean that's why we're employed, that's why I've been employed for 30 years in the IT security industry, because it's hard and it's you know, I don't know when we're going to ever get to a point where it's easy. I hope we could get to that point. But yeah, security and enterprise is a difficult challenge.
Speaker 2:There's a lot of things you have to balance and the users always want more. They always want more access, they always want more freedoms. They always want to bring in this new software or this new device, or you know, I just got this thing at Best Buy and I want to hook it up to the corporate network and they don't like being told no. So there's a balance of that and there's some logic to that as well. It's like this is a really cool device that will help me in my job. Why shouldn't I use it? And then security has to step in and go. Well, because it does everything in clear text, that it shows all of our secrets to anyone who might be listening. So there's a back and forth and there's a balance there and it is definitely difficult, but, if nothing else, it ensures that we're going to be employed.
Speaker 1:That's at least positive for some of us. So, speaking of patches and everything, like you know, palo alto, today it could be avanti, tomorrow it'll. It'll definitely be one of the two right, and even trend has some vulnerabilities absolutely.
Speaker 1:We're not unique in that area yeah, where do all these vulnerabilities come from? Why is there so many in all these like big, so successful companies? And I've heard like it, just like they bought company, that bought a company, that bought a company and all of a sudden, you're in a 2004 version of something that nobody knows what it is anymore Like. Is that really what it is or is there more to it?
Speaker 2:Well, that's one of the reasons. It is like it is Definitely the acquisitions. You look at things and it gets bought by this one and this one and this one and all of a sudden, like this incredibly important piece of software is running off code that some intern wrote in 2004. That's one of the reasons. There are other reasons is there's more people looking these days. So the number of researchers, the number of people out there, has just increased exponentially and they're looking at it because there's more to look at. Just the amount of software available today, I think back to 2008, 9, 10, what Microsoft was patching back then versus what they patched this last Patch Tuesday. I'm seeing stuff that never would have gotten patched, whether it be GitHub or LLMs or machine learning stuff or open source stuff or Xbox stuff, like all this stuff that's getting patched these days because there's so much more out there that could be exploited. So it's a combination of things. There's more to look at. There's more people looking for it. It's financially interesting to people right now, for people like the Zero Day Initiative or Bug Crowd or HackerOne, or even going straight to the vendor, the vendor-specific programs like Google or Microsoft or Adobe or whatever.
Speaker 2:But then software these days is not built with security at the beginning. Software these days is built to get to the market fast and then bolted on with security later. And what we're finding out, which we find out every single year and we never learned from, is that's not a very smart way to build software. If you want it secure, you have to build it security at the foundational part of it, and there used to be a thing called the secure development lifecycle that would go into this and show you how to build secure software, and it's just not followed very much. These days you have all of these things that are rushing to get to market and I'm not going to call out any vendor, because no vendor is perfect and better is very qualitative. But yeah, it's one of those things where there's so much software out these days. It's rushed to market as fast as possible Because if you don't publish, you perish, and there's just security bolted on at the end.
Speaker 2:I'm always fascinated by security researchers and how they are so inventive with coming up with various things to exploit systems. I'm hopeful that with LLMs and with AI, we will get better at code review prior to release. I'm not impressed with what they've done after release. Looking at a piece of released closed source software, I am impressed with some of the things that they can do looking at an entire code base. So I'm hopeful.
Speaker 2:If there's anything to be hopeful with for AI is that we'll get better at code review and we can find some of these things, especially the simple things stack-based buffer overflows, command injection, double freeze. These are relatively simple bugs to find when you're looking at the source code, but it astonishes me that they still exist in the wild. In our last Pwn2Own we had a format string bug in one of our products and like format string, but it was just 2002. I mean, it was incredible. I haven't seen a FormatString bug in years, but there was one there and it was just kind of astonishing that it made it through all that code review for a big product from a multinational corporation.
Speaker 2:It's like yeah, so it's a combination of things, it's not just one. You can point to a lot of different things on why there's so many bugs of things. It's not just one. You can point to a lot of different things on why there's so many bugs, but it really is a combination of a lot of different things.
Speaker 1:What do you think about that report that was saying that shift left is bullshit, like that IBM made up this whole. Have you heard about that one? Ibm and the Ponemon Institute came out like a long time ago and like they were the ones that I'm not sure if it was them, but the whole shift left right like that, if you. What they were saying in the report is like if you fix a security bug before it gets released, then it's going to be worth. It's going to cost you ten dollars, whereas if you fix it production so it costs ten thousand. That but they said that there's no evidence behind it. It's bullshit is what the new thing is uh, I've been there.
Speaker 2:Uh, so I know the price of how much Microsoft paid to fix bugs. Yeah, and I know so just with my personal experience and looking at budgets it's cheaper to fix it in production than to fix it after it ships period. So I mean because I know shipping it, I mean fixing a bug before pre-release is going to cost you however much dollars it's going to cost in, you know, personnel hours or whatever. Retesting they were bugs, they were multimillion dollar bugs at Microsoft and you can't tell me fixing something in pre-production is multimillion dollars. So, yeah, don't.
Speaker 1:I'm sorry.
Speaker 2:I think there's no evidence of that. Because no one wants to talk about how much it costs. Like I still don't feel right like saying actual numbers from Microsoft, even though I haven't worked there in years. But yeah, no one wants to talk about how much it actually costs. Because if your customers figure out how much it actually costs for them to ship a patch, they're going to be furious.
Speaker 2:It's like no wonder the stuff is so expensive. Because it costs so much to fix a patch and to and it at least to fix it correctly. Because because all the patches one of the interesting thing about patches is it shines a gigantic spot bite on whatever's being patched. So if there's a variant of that bug, researchers are going to find it. So you have to find it first. So you have to do variant testing on the thing that you're patching to make sure that any closely related things you're going to find before the attackers do so just in that testing alone it's going to be more money than just oh, here's a stack-based buffer overflow, let me fix that real quick. Or you know, here's command injection bug, let me clean up the input on that. It's going to be more expensive and that's just the reality, and as someone who's been around patching since 2008,. The reality is it's cheaper to fix it before you ship it, period.
Speaker 1:So, I guess, is the future moving forward with all these vulnerabilities in virtual patching? Is that just like something that everybody needs to just figure out a process for, because it's just the sheer numbers? I don't know how they're doing it today. I feel bad for them, yeah.
Speaker 2:I feel bad for the responders as well, because I know, I mean, I managed networks in like the late 90s and early 2000s and it's so much harder these days than it was when I did it and it was hard that. So, yeah, I really feel for folks. I think what people are really going to have to figure out best is looking at their network defense as a team, whether it be like, okay, I've got this, I've got that, I've got this, and then understanding that, okay, my running back is going to be this product, that it, you know it does this thing. Or you know, this is my wide receiver, I'm sorry, I was watching football last night, but you know, but you have to like, patching is incredibly important, but we also know that patching should be tested before you roll it out. I think what's going to happen is so they have to look at things holistically and I hate using that word because it's such a buzzword or whatever but they have to look at everything available for them as far as their defense goes, whether it be something like virtual patching again, network segmentation or different IDSs, ipss, edr, xdr, all the letters, whatever you have, whatever you can afford. You're going to have to figure out how that works best together and then lower the number of panes of glass that you look at to have to monitor all that stuff. But I think what it's really going to come down to is, sooner or later the vendors are going to have to step up and stop making garbage patches. Patches break stuff all the time and that's unacceptable, especially for certain ways. It's like how in the world did you miss this in testing those really things?
Speaker 2:There's going to come down to liability somewhere on that, whether it's people getting sued, vendors getting sued by customers. We saw this recently with CrowdStrike. Delta Airlines is now suing CrowdStrike and Microsoft for a bad update. So that's kind of the first really big salvo in that, and I'm not going to try and comment on whether or not the case has any merits. It's far too complex for me, but that's happening. Now. There's going to be a point where the government gets involved too. The FAA was already looking at that because it shut down so many airlines.
Speaker 2:It's like should this be legislated? And I think it's going to come to a point where it has to be. If they don't clean up, they have to be legislated, and no one wants that. No one wants legislation around this. But if you look at the continued poor quality of security patches, sooner or later some senators are going to step up and go. This affected my constituency. This has to be dealt with. It's been a problem for 30 years. We have to make it stop and I think that's really where it's going to head.
Speaker 2:If the vendors really don't clean up their stuff and I think there are going to be a lot of network defenders who will be absolutely thrilled with a better quality of patch, because right now, I don't tell people to patch, I tell people to test and patch, because rolling patches straight off to production, that's risky. And people I don't say this, but there are people who say turn off automatic updates on the second Monday of every month so you don't automatically get patched and then automatically get broken. And I've been on the side where I've broken people. Like I broke Norway twice. Yeah, really, yes, I sent out a bad update. Norway's offline. I'm like, oh, my goodness, how did I manage that? Then did it again three months later.
Speaker 2:So, yeah, you're forgiven, so it happens. But yeah, there's gotta be some improvement in quality. There's got to be some outcry from the people who are really being affected by this the most, and we saw this with the airline industry in the CrowdStrike update, but that's just the most egregious example. There've been a lot of other examples over the years with a lot of different vendors, and I'm not trying to pick on just one vendor here, because there's a lot of blame to spread when it comes to that. So, yeah, I think it's going to be very interesting over the next decade to see where it goes legislatively, either in the courts for being sued or from the government EU perspective, and I really think the EU is probably going to lead the way on this, even over the US government.
Speaker 1:Well, it's because you have 50 states, yes, 29, or something like that. But yeah, I mean it's because you have 50 states yes, we have 29, or something like that. But yeah, I mean, it's interesting, I never really thought about that. Like you have a patch comes out, security's like patch, and then there must be another side of the company that's like ow, do you think it's been more incidents caused by the security incident due to not patching or patching automatically and dealing with consequences?
Speaker 2:Really, if you look at the numbers, there've been more incidents and more downtime from bad security patches than there have been from attackers no way.
Speaker 2:And it's terrible. But people are afraid of security patches. I mean, beyond just being inconvenient, because all of a sudden you have this little pop-up says you need to restart, you need to apply patches, you know whatever, because it never pops up at a time where you're just watching a youtube video. It always pops up when you're doing something important, right? But yeah, beyond just being inconvenience, they're they often break things. I always say the fastest way to get fired as a sysadmin is to break email, and the fastest way to break email is to patch exchange. And it happened with this last security patch too for Exchange, where people applied it and, sure enough, it broke a bunch of Exchange servers. So Microsoft has withdrawn that patch right now. So you have a bug that is publicly known that you cannot patch because the security patch broke the Exchange server and you got to roll back to a vulnerable version.
Speaker 1:You have to roll back to a vulnerable version exactly.
Speaker 2:And that's part of the problem is like these patches are of such poor quality that people are afraid of them. And that's why I say test and deploy, because you have to figure out what's going to break in my environment, because there's so many variances in different environments. That's one of the big advantages of Apple is they control so much of their own enterprise and everything.
Speaker 2:Yeah, their own supply chain, so they can get away with more of that. But that's still a very interesting area. Apple's had bad patches too, so how do you have a bad Apple patch? I mean, what scenario are you not taking into account for there? But it happens. A bad Apple so yeah, there's been more downtime from bad patches than from actual attacks.
Speaker 1:Wow, I've actually never thought about that. I guess we should end there then End on a strong note. Yeah.
Speaker 2:Yeah, I'm very passionate about patches and that's one of the reasons why is because they affect so much stuff, and we've been saying it for over 20 years is patch is the best thing you can do to protect your networks and yet we're still so bad at it and people don't do it. And one of the reasons people don't do it is because patches break things. There was a huge printer, a huge bunch of printer bugs that came out a couple of years ago print nightmare, what's the name of it? Yeah, I remember that those bugs, those patches, broke so much printing. It's like, well, yeah, if I turn off printing, I'm no longer to this bug, but I need to print stuff and it was terrible. It took months and months and months to get it straight so that the printers were safe and working. And, yeah, you have the big incidents like the cloud strike, but you have all of these smaller incidents that add up over time and, yeah, the state of patching is really abysmal and we have to be better.
Speaker 1:Yeah, I guess the thing that has in common on both sides is the speed right, like bad software because you got to go to market, and then bad patches because, fuck, there's a problem, we got to fix it.
Speaker 2:Yep, you got to fix it. Yeah, but especially with people saying you have 120 days or whatever. Yeah, you know those jerks. Yeah, you know those jerks. Yeah, always hold me accountable Right.
Speaker 1:Well, Mr Childs, thank you so much for coming on and sharing your expertise and your opinions.
Speaker 2:No problem, rob, I'm happy to be here. I've enjoyed it Cool.
Speaker 1:Well, keep up the good work at the Zero Day Initiative and keep having fun at Pondone and we'll see you around.
Speaker 2:Absolutely Thank, you See you Ciao.
Speaker 1:Well, that's all for today, folks. Thank you for tuning in to the Mnemonic Security Podcast. If you have any concepts or ideas that you'd like us to discuss on future episodes, please feel free to hit me up on LinkedIn or to send us a mail to podcast at mnemonicno. Thank you for listening and
