The Asymmetric Future

With or without agents, the cost of failure is asymmetric for attackers and defenders.

An attacking agent that fails risks getting caught. A defending agent that fails risks taking down its own business.

In this episode of the mnemonic security podcast, Robby is joined by Matteo Strada, a cybersecurity engineer and researcher specialising in AI security.

Drawing on his blog post, The Asymmetric Future of AI and Cybersecurity, they explore the guardrail paradox exposed by the recent OpenAI–Hugging Face incident, the growing case for local and open-source models in the enterprise, and how much control companies are giving up when they build critical workflows on top of frontier AI models.

https://mstrada.me/posts/aicybersec

Transcript

Speaker 0:03

From our headquarters in Oslo Norway, and on behalf of our host Robby Peralta, welcome to the mnemonic security podcast.

Robby Peralta 0:15

An attacking agent that fails might get caught. A defending agent that fails might take down its own business. Attackers get retries, defenders get consequences. And the recent Hugging Face incident exposed another problem. OpenAI's agent took more than 17,000 autonomous actions over a weekend, escaping its sandbox and ultimately breaking into Hugging Face. And when Hugging Face tried to investigate what happened, they said Claude Opus and Fable refused much of the reverse engineering work because their guardrails struggled to distinguish analyzing an exploit from actually carrying one out. Lovely. So what do defenders do? Do we keep relying on frontier models that can refuse us when things get uncomfortable? Or do we bring the models in-house, run them locally, and take control of the guardrails ourselves? One thing is for sure, that notion of attackers moving at machine speed is finally coming true.

Robby Peralta 1:18

Matteo Strada, welcome to the podcast.

Matteo Strada 1:21

Thank you very much for inviting me. It's a pleasure.

Robby Peralta 1:25

Ciao Bello. You're in Milan, right?

Matteo Strada 1:27

Yeah, right now I'm in Milan.

Robby Peralta 1:29

Lovely. Lucky man. So I came across to you in your blog, uh, which I'll put in the show notes. I think you called it the asymmetric feature of AI and cybersecurity. What was running through your head when you wrote that?

Matteo Strada 1:42

So I think that there is an symmetry, mainly because attackers can use these tools to perform cybersecurity attacks. And the worst scenario that they get is that they get detected. The vulnerability that they're trying to exploit gets fixed. And so this is their worst scenario, detection. On the other side, if we think about defenders in companies try to automate with AI agents all the uh cybersecurity processes, we have uh a different scenario. And here um we have the asymmetry. Because if the AI agent makes some mistakes, then you have some production issues and some business uh issue, and you're going to lose some money. So the the freedom that you have on one side as an attacker is much broader than if you are a defender. So I think that this is the the biggest asymmetry that we're going to see uh even more in the future. And that we also started to see uh with the aging face cybersecurity incident. Um so if you want, I can try to make a summary of what happened. Yeah, that'd be great. Yeah, so from from what we know uh right now, OpenAI was testing different models on different uh cybersecurity benchmarks. And during one of these executions, one model had a goal of solving one of these tasks of the benchmark. And the model to solve this task was able to find an exploit to go outside, to break the sandbox, which is like an isolated environment where these tests are usually run. And uh so it was able to break this sandbox, to go outside the safe network, to search for a library that the model was thinking that would allow him to solve the test, uh, to find a vulnerability inside that library and break into uh Hagging Face. So an extremely sophisticated attack. And I think that here is where the asymmetry came in. Because Hugging Face detected this breach, and as they wrote inside their incident post, they were not able to use like public models, like closed source model to investigate this issue because GPT was blocking their requests, thinking that Hugging Face was trying to use this this uh information in a malicious way. So they they had to use open source model, they use a GLM uh model to help them investigate this issue. So it's definitely clear that we have an asymmetry. And this topic is also linked on how the current models are are measuring whether we can do some actions and whether we cannot. And there we have a threshold, and we are moving that threshold one side, we we allow the model to release information, and if we move it on the other side, we are going to say to the model that uh it should not release too many information. And that threshold is basically one of the most important things in this in this asymmetry discussion. Because if you move it too much on one side, you have too many refusals from the model, and if you move it too much on the other side, it's going to release too many information. So I think that this is all this would be one of the um central point of of uh future discussions.

Robby Peralta 5:30

Do you think it's normal for companies to use models in the incident response situation? Do you think that's like the the business as usual moving forward? Because usually you would call like a company if you didn't have the resources, but maybe now people just throw everything at a model instead of calling calling incident response firms. Do you have any thoughts about that?

Matteo Strada 5:48

Yeah, I think that LLM models can can help uh this type of processes because they may see some some connections, uh some relations between logs that a human uh didn't see. Um at the same time, I also think that the the the human part still plays a central role in all the cybersecurity processes.

Robby Peralta 6:11

So yeah, when they when they did use the model for incident response purposes, it just said we can't help you with this because it looks like you're trying to do something that we're not gonna allow you to do. At the same time, uh this was going on maybe a couple days before Kidmy was released. Uh would you explain why that's relevant here?

Matteo Strada 6:29

Yeah. So right now we have closed source model that usually have better capabilities, like Fable, Mythos, GPT 5.6 SOL, and then we have open source model, GPT OSS, which is a bit old, but Deep Seek, uh V4, GLM, and uh and much more. So as we saw with the Alligame Phase incident, uh local models can have an extremely important role. For I think for three different reasons. So first reason is that you have some um some expected costs, right? So when you're using a closed source model like GPT, OpenAI may just change pricing tomorrow and you have to accept it, right? Uh if you create a pipeline for local models, you know what you're going to spend as an upfront cost for the hardware and how much you're going to spend for the usage of the model. The second element I think is about availability. So a few weeks back, we saw how the US government blocked the release of uh of Fable and the GPT 5.6. So with uh if you have a local model, you don't have these problems. You are you're owning the model, you're owning the inference, so you will not have any availability problem. If tomorrow a jailbreak uh comes out for GPT 5.6 and the government is going to block it again, you you will not have access to the model, right? But if you if you have local inference, this is not an issue. And uh and finally you have more control. So as we saw uh with the aggregate incident, with local models, you usually can bypass the guard delays easier. Uh, you can perform uh steering with the model, so you have more control also on cybersecurity topics. And uh as a last point, you have a full data control, right? So I think that we should remember that when we are using um like a closed source AI model, we are sending information to a third party, right? So we we may, of course, accept this risk because there are contracts and we are trusting that party, but for some extremely confidential information, we may not want to send information data to a third party. And so I think that this is another important element to keep in mind.

Robby Peralta 9:16

It just sounds like to me, there must be so few use cases where you actually need a frontier model. Like I get it if you're doing exploit development, because that's where you actually see the differences between the the frontier labs and the on-prem models. But it sounds like there are so many downsides to using one of these closed source models. Does everybody agree with that? Or is that sort of like a new thought?

Matteo Strada 9:36

I think that the the most popular thought is that you want to use the best tool out there, right? So you're not trying other things, right? You're just using the best tool and that's uh that that works pretty good. Or do the job, yeah. Yeah. And if you're using it just for coding, for personal use, I think it's it's extremely fine. But if we talk about LLM as being an asset, like a strategic asset in an enterprise environment, I think that you should also think about the the long-term cost, the availability, and all the other uh points that we discussed before.

Robby Peralta 10:17

How should one be architecting the usage of LLMs for normal employees? I'm thinking of uh the concept of like token maxing. You shouldn't be using the best models for what I use it for, like, hey, sum up this uh blog post that I read.

Matteo Strada 10:32

Yeah, so the first option is uh awareness, right? If you explain to people that for some tasks, using some models may be better than using other models, they are going just to choose the right model for a specific task. In the browser, right?

Robby Peralta 10:50

You just click the yeah.

Matteo Strada 10:52

Um at the same time, I I understand that this may be a bit complicated. You just want to write something and uh uh you don't want to make the choice for every task. So um there are different orchestrators that based on the request, they're going to use like a router that is going to select the best model. So if for easy tasks, they're going to select a very cheap model. And if a task requires more reasoning, they're going to select a bigger and more advanced model. The model selector could also um could also see if you uploaded any sensitive information. And in that case, it's going to throw you to the local models. That that may be another uh interesting use case. But again, right now we don't uh we don't have the like the cost issue, right? Everybody just spending tokens and tokens, but I don't think that that's sustainable over time, right? We should start to think about these um these optimizations.

Robby Peralta 11:57

One thing I want to touch on before we uh go away from um open source versus frontier lab stuff. Did you see the interview with Palantir CEO?

Matteo Strada 12:05

Which one?

Robby Peralta 12:06

Basically, he came out and said that he thinks it's crazy that enterprises are building their business, their processes inside of the frontier labs because they're basically giving all their intellectual property to the frontier labs and giving them all they need to be able to copy their business process, right? They can literally go in and see how exactly my business works. And if it was novel enough and they saw an opportunity, they could just steal it. Do you see his point? Or what do you what do you think about that?

Matteo Strada 12:33

I don't know specifically the open AI uh policy. Uh I assume that they're not able to to create a a profile of of Robbie and make decisions based on that. Uh, but if you are a company and you're using these tools to reason over uh secret projects or IPs, then I think you should it should be like a center of discussion, the fact that you're you're sending data to a third party.

Robby Peralta 13:06

What are the downsides of using open source models?

Matteo Strada 13:10

So the first downside is upfront cost. Hardware. Yeah. And uh right now hardware costs um a lot. Nvidia is selling the GPU that we're using for inference, so that we're using when we use a model, right? For an enterprise use, of course, you are expecting to buy some some very powerful GPUs because then you have to distribute the model across different people. If you're talking about just personal use, they're starting to set some um some workstation, they're usually pretty small. For example, um NVIDIA, I think it's called NVIDIA Spark, uh, which is three or four thousand dollars. Of course, um Apple MacBook. Um, since they have a unified memory you can use to for the model inference. And um we are starting to see different uh local inference engine projects. A famous one is uh is Dwalf Stars from AntiReds, the creator of uh Redis, where with MacBook Pro, you can you can basically run DeepSec uh Flash with a quite high speed. And um and also they they're starting to implement also SSD streaming, where you can use the SSD to um as a as a storage for for the model. Uh so even if you don't have high uh memory, you can you can use it. Of course, uh uh it would be a bit slower, uh, but you can you can still use it.

Robby Peralta 14:48

That means you can be running a model locally and asking it whatever you want without the oversight of anybody because it's running locally, but you're still being able to use the fancy model because it's some you're pulling it from somewhere else.

Matteo Strada 15:02

Yeah, yeah, exactly. Of course, the model will be uh smaller, so you will not have the same reasoning capabilities of uh using closed source model. Um but as we discussed before, you may not need uh such a bigger model for for every task and for the most confidential ones. You that that that may be enough.

Robby Peralta 15:23

When you have an on-prem model that's running locally, nobody can control what you're doing anymore or see what you're doing. Would you mind elaborating that a little more? Or did I understand that correctly?

Matteo Strada 15:33

Yeah, no, that that's an another open point. Mainly because with local model, it's easier to bypass all the guard rates that the model the model have. So when we are using like GPT, you have different protection layers, different guard lays, of course, you have the alignment phase, but also other types of guard rates that they put before. If you use a local model, you have access to the model itself. So you can you can try to brute force the model. So you can use maybe hundreds of different prompt and try to jailbreak the model. While if you use GPT or anthropic models, they're going to detect it uh like pretty fast. So while local model on one side allows you the freedom on the defense side, so on all the benign usage, on the other side, it may be an interesting tool for attackers, where you if you can get a powerful local model and you're able to unlock it and through the usage of some jailbreak, you can use it to perform attacks. And while this is just like a forecasting game, I don't know uh what the future will be, but definitely one of the possible use cases where you have maybe tens or hundreds of these models that are going to analyze the system that you want to attack, they're going to coordinate one with the other, and they're going to perform the attack. So what one one note on this possible uh scenario is that right now we don't have any model except for like GPT 5.6 Sol and the Mythos that are able to make cybersecurity attacks in a consistent ways, right? So one interesting study from the AI security research team from the UK, I don't know if if the name is exactly that, but they tested with a benchmark where a model was asked to bridge into a system that had like 30 different steps to do to reach the um like the final uh step. And uh a lot of models were able to do the first like five or six steps as an average result, while uh bigger model like Mythos in 5.6 were able in the best run to achieve the latest uh step. And we're talking about like 30 different steps to to reach the final one. And this is the this was the best attempt. So it was not able to make it for every run. So right now we don't have like the consistency, but we don't need it probably. It is enough if you if you try and try it again, it is enough that one of these times you you're able to to breach because the model finds the right pattern, and and basically you're you're breaching a system. So when you're using tens of of these agents, you're you're increasing the likelihood of of finding like the right combination, the right path to to breach a system, basically.

Robby Peralta 18:56

So alert triage, reverse engineering, vulnerability prioritization, threat intelligence summarization, detection engineering. You mentioned all these things in your your blog posts. How do you look at those aspects these days?

Matteo Strada 19:08

Yeah, I think that all these areas of cybersecurity basically share two properties uh in common. So the first one is that there is a human who can cheaply check the output of a model. And uh the second one is that the the model uh if you get the answer wrong, it's not touching like any live system. And this is linked to the asymmetry of damage that a defender can do. In software development, we saw that these models are much faster than human developers when it comes to writing uh functional code. There is uh like a bottleneck where uh software developers who are reviewing the code basically are slowing down the entire process, right? Because if you're using the a state-of-the-art model to write code, you are uh writing code much faster than a human could do. But if you're using a human to review the entire code, basically the the speed is the same, right? Um so you you you can try to to fix this bottleneck by using very large test suite where you can test the software and uh you see the results and you can fix and if you find any bug. I think that in cybersecurity it's uh it's a bit different. So you you want that human bottleneck for for the decision, right? What you can use the LMs for is to reduce the information, to reduce the number of alerts, to reduce the number of tickets that you get inside of the bottleneck, right? So you're reducing the sources uh that comes inside the bottleneck, but you still want a human bottleneck that is going to perform the last decisions, right? So I think the LLMs can help to uh prioritize vulnerabilities, to prioritize alerts, but at the at the end of the pipeline, you still want a human that perform like the the highest values decision that an agent may miss due to the context, or may may miss because it doesn't have some some information that a human uh may have.

Robby Peralta 21:38

The implementations you've been doing, is it has it been building new things using LLMs, or have you been using a lot of time to in like you just kind of pointed to increase the efficiency and accuracy and improve existing processes?

Matteo Strada 21:53

I think it was an approach from from Google where you use uh like 70% of your time. To run daily activities, then you use 20% of the time to optimize things, and then you use 10% of the time to experiment, right? You want to to make high risk testing on uh idea that you have that may work or may not, but you want to also use more time to optimize process that you already have, right? And then of course you have to uh do your your day job. Your day job. Yeah. At the end of the day, you do you have to do it. And linked to that, I think that it should not be just something that an internal AI team uh can do, right? Each team is made of the best person that can find the right idea to optimize their workflow, right? If I'm part of the AI team or the security team, I have no idea what's the best tool for a marketing team, right? So I think that the marketing team or whatever team it is should have the ability of creating with uh with these new tools, of course, in protected environment without creating a risk for the company. But I think they should have these this opportunity to build tools, to build internal software, something that they think can bring additional value and can optimize their workflow. Then these these types of tools should be reviewed by some other teams that are going to say, I mean, it's something that we can keep or it's something that it's not possible to do from the security team who will have to check what kind of information are involved. So you you have something uh very useful for that team, and that you know that it exists because you have a process in place.

Robby Peralta 23:55

Yeah, I do feel that a lot of people these days are doing the they're maybe doing 70, 30, and then 50% of their uh evening time on the innovation part. But uh that's the life that we live in these days, trying not to fall behind. So there's security companies out there that you know help you with pen testing, GRC, security operations. There there must be a shift in the economy these days for security partners because everybody was used to doing what they're doing before. Now here comes large language models, and they are kind of eating security companies lunch, right? Because you could just ask a model today to do all these things. Where do you see the future of security vendors? Like when I started with cybersecurity, you would you would call people that knew what they're talking about, and you'd say, Hey, I have this project. What do you recommend? And maybe there'd be some software from some popular vendors, or you need an architect. But now I could I could totally see those people just going to an LLM and asking them and then just going with whatever it says.

Matteo Strada 24:59

I think that it is maybe a risk, right? Overconfidence towards the LLM. I think that in the next years we are going to to see, I would call bubbles, uh, like internet bubbles and so on. And as part of this process, I think we should use LLMs on on our own to try to understand which limits LLMs have, right? Because if you if you know the limits of LLM, you can understand whether someone is selling you something that doesn't have any additional value that you can do on your own, or if it's something that can can bring some interesting feature uh that can simplify your your your working day, or find new vulnerabilities or solve some security issues. So at the end of the day, I think that as every other tool, we should use it, we should study it, we should uh understand it, and and when we have made more practice with it, we can make a more aware decision uh on whether to choose or not choose, for example, a vendor.

Robby Peralta 26:17

70 2010, Mateo. I'm gonna go for that. I'm gonna make some calls to make sure that that's how we're doing it mnemonic as well. Thank you so much for your time today. We'll talk to you soon. Take care.

Matteo Strada 26:28

Thank you very much, Robby.

Speaker 26:30

Bye. It was a pleasure. Thank you. Bye. It's always ciao.

Robby Peralta 26:34

Well, that's all for today, folks. Thank you for tuning in to the mnemonic security podcast. If you have any concepts or ideas that you'd like us to discuss on future episodes, please feel free to hit me up on LinkedIn or to send us a mail to podcast @mneonic.no. Thank you for listening. We will see you next time.