Models Always Lie

For this episode, Robby is once again joined by Eoin Wickens, Technical Research Director at HiddenLayer, an organisation doing security for Machine learning (ML) and Artificial Intelligence (AI). 

It is not too long ago since Eoin last visited the podcast, (only 7 months,)  but lots has happened in the world of AI since. During the episode, he talks about some of the most significant changes and developments he’s seen the last months, how models are getting smarter, smaller and more specific, and he revisits his crystal ball predictions last episode.

Robby and Eoin discuss potential security risks posed by using AI tools, how to secure AI powered tools, and what you should think about before using them. Eoin also gives some new crystal ball predictions and recommendations to organisations starting to utilise AI adjacent technologies.

Transcript

AI Integration in Security Operations

Speaker 1 0:03

From our headquarters in Oslo, Norway, and on behalf of our host, Robby Peralta, welcome to the mnemonic security podcast.

Robby Peralta 0:15

As you all know, this is a family podcast where we accept and celebrate all types of faiths and religions. We can even be friends if you constantly misuse the abbreviation AI. That being said, I wanted to share a theological concept that emerged in the 19th century known as God of the Gaps. Which basically states that gaps in scientific understanding are to be regarded as indications of the existence of God. For example, before we knew how thunder and lightning actually worked, it was of course the God Thor and his magical hammer. Fast forward a thousand years, and we find ourselves in the same position. Quote unquote, powered by AI. So if you want to be considered woke in 2024, you should be using the phrase large language models more often. As those technically are not AI. And if you're listening to this thinking that I'm a hypocrite, you're right. But as my dad would say, do as I say, not as I do. Eoin Wickens, welcome back to the podcast.

Eoin Wickens 1:20

Hey Robby, thanks again for having me. It's uh great to be back and great to chat with you again.

Robby Peralta 1:24

First of all, congratulations on the promotion.

Eoin Wickens 1:27

Thank you very much.

Robby Peralta 1:28

Has it even been a year and you've gone from a senior adversarial machine learning researcher to technical research director?

Eoin Wickens 1:34

Hey! It's been a busy, year nose to the grindstone, but uh yeah, we move.

Robby Peralta 1:40

How old are you again? 25?

Eoin Wickens 1:42

Uh 26, 26.

Robby Peralta 1:43

26, okay. Yeah. So you just had to get over the 25 to become a director.

Eoin Wickens 1:47

I think that's it. That's in the that's in the contract somewhere. Um every year I get a year older. It's kind of mad.

Robby Peralta 1:52

I know, it's crazy. I don't want to talk about how old I am. Um but since last time we spoke, there's been a lot of uh there's been a lot of AI. Are you tired of it yet? Do you call it AI, by the way? What do you call it?

Eoin Wickens 2:03

Uh yeah, I mean, I guess it's more machine learning, really, isn't it? It always has been. But um at least until we get some sort of sentient being. But um, yeah, no, not sick of it. I think the the the rapid progress and pace keeps things interesting. Um it's amazing what's happened in a year. I think even since even since we last spoke, which I think was back in July or August.

Robby Peralta 2:24

It was less than a year ago at least.

Eoin Wickens 2:26

Yeah, yeah, yeah. Time is moving on quick, eh?

Robby Peralta 2:29

Yeah. Have you ever heard about the um the God of the gaps?

Eoin Wickens 2:33

No, I don't think so.

Robby Peralta 2:34

Okay, so this is I'm stealing this for my colleagues, which are much smarter than me, but it's um it's basically a term from research about religion. So as soon as you don't understand something, it's God, right? So like if when we didn't understand lightning and thunder, it was it was Thor, right? Yes. So that's kind of how we look at AI, right? Like I even started calling it LLMs now instead of AI. So I'm I'm on uh I've had a reality check. I think most of the market has. Absolutely. So that's good. Um what else has happened? Uh the models have gotten smaller, they've gotten more fine-tuned, smarter.

Eoin Wickens 3:08

Uh totally, totally. And and I think um processing power has gone up as well, right? I mean, with like NVIDIA's new Blackwell chip, I believe. I mean, things are gonna change even more as the models get smaller and the the processing power gets more and more capable. I mean, I'm really looking forward to a day where we have like proper on-device AI, where we have like the hardware to be able to run them and the the software to be able to squeeze onto them. So I'd say that in the next few years is definitely gonna be something we start to see more and more of.

Robby Peralta 3:37

What else um notable, like I mean, you you follow the stuff. You have to follow the stuff. You get paid to follow the stuff. Like, what are the some of the biggest update? Abstrapted. Yeah, right. What are the some of the uh biggest updates or changes you've seen in the past eight months?

Eoin Wickens 3:51

Oh yeah, that's a good question. That's a good question. I think, as you said, I think the models are getting smaller. I think they're starting to distribute as well to kind of different model architectures, um, you know, building upon transformers, which is like the, I guess you could say it was like the holy grail of large language model architectures for for quite a while, but they've been improving and iterating on that. And to get models which are better at doing different tasks, they've delegated out into things like mixture of experts, models. They've brought in other things like retrieval augmented generation, which basically means they have like this added bolt-on layer of like a kind of like a database, basically for extra information that they can when an LLM is queried, pull that information out of that database and feed it in, right? Um, so it's like on top of fine-tuning, you can also do this stuff. So basically the the models are getting smarter, they're getting smaller, um, they're getting more specific to dedicated tasks, which is great. And I think we're we're starting to see just a lot of exploration. And then also, since we last spoke, I think LLMs just really have taken off in terms of like integration. Um I think I don't think this was an April Fool's, though. I hope I, you know, I hope I'm not on, you know, I hope I'm not live saying this. But I I do recall at some point last year they were going to replace one of the Windows keys with like a dedicated copilot button. So I mean this, you know, that tells you just how much like all of these companies are now kind of dead set on on AI within their platforms.

Robby Peralta 5:20

Now that you're mentioning vendors, right? That's one of the things I've really noticed since last time we spoken, is like all the um all the security vendors have have released what do they call it AI uh AI enabled security tools, right? Totally. I think I sent you just an example just to take checkpoint as one uh I typed in on Google, like a AI-enabled security tools and checking what was up there on top. And um, if I look at what they were able to do, there's a bunch of yeah, reduced up to 90% of the time needed to perform administrative tasks. Infinity AI co-pilot knows the customers' policies, access rules, objects, logs, as well as product documentation. So it just basically allows you to chat with your product or chat with your firewall and get it to do things for you. And that's just one example, right? I would assume like all the major big companies will be coming out with that soon here. I'm not sure if you have a comment on that.

Eoin Wickens 6:10

Yeah, absolutely. So, I mean, last year we had um uh we had Google Sec Pam came out, which was one basically security-oriented um large language model. We had uh CrowdStrike had Charlotte, and there was also, I mean Microsoft security copilot as well. So there's like a number of different vendors that are moving into the to the space using LLMs for basically augmenting security workflows. So I do remember the last time we spoke, actually, I think I made this kind of like crystal ball prediction that you know the SOC would start transitioning into, you know, an LLM kind of um quality control thing, as well as using LLMs to augment their workflows and take all that data and do analysis to help with like alert fatigue and everything like that. That's kind of where we're heading. Um, I think that's what these tools will be able to allow us to do. I mean, we have so many products feeding telemetry today that I don't think that it's possible with our current workforce, and I'm turning in terms of how many people are in the workforce today, to actually triage and process all that data. So we have to offload it somehow. And we've done that typically through static and dynamic rules. But now, with the age of AI and with you know LLMs, we can just process all of that so much easier and so much quicker. So I think like these companies, fantastic initiatives. Um really interested to see how they they work and integrate in practice. I'm I'm a little out of the malware game and the security operations center game now, but um, I'd love to see how that's being done in and integrated in practice, certainly.

Robby Peralta 7:40

Yeah, I know that uh from a SOC from our point of view, um customers were first like, yeah, what are you doing with AI? And then they were like, wait, I kind of want humans to be doing this for me, not not LLMs or not AI yet. So I guess uh the hype has gone over a little bit.

Eoin Wickens 7:55

Absolutely. I think things tend to come back to base after a while. And and look, I mean, I don't think AI is a perfect solution for everything. Um like I think what is it? Um models always lie, I believe was was somebody coined that, not me.

Robby Peralta 8:07

Models always lie, yeah.

Eoin Wickens 8:08

Models always lie, but it's like I think it's really important to have both, right? I I don't think we should offload all of our responsibilities to AI models, with let's say within the security context to start with. Um, but I think that like it serves a really valuable purpose in enabling us to like do a lot of like base level triage and then have like human review over that. Um and I think that'll settle in time.

Robby Peralta 8:30

Back to like the vendor and AI tool uh discussion that we started. Um what sort of like advice do you would you give clients of like okay, before they start using checkpoint or copilot or whatever sort of tool it is? Because a lot of I can see that a lot of customers just been like, oh cool, it's a chat bot, and they just start using it and they just go for it. Uh is that is that totally fine, or are there still some things that you should have in the back of your head when you start working with those sort of tools? Not knowing the specific tool, obviously, but

Eoin Wickens 8:58

totally, totally. Main thing is um understanding where your data is going, right? And understanding how somebody that you're sending your data to can use your data. Um, I think we've often seen over the years, um, if you're not paying for it, then you are the product, you know, things like social media, data harvesting and everything like that. Well, if you're uploading something sensitive to your to uh let's say an LLM service, right? Like um, let's say you're revenue forecasting for you know the next quarter or whatever, well, if that gets into the hands of somebody that you don't want that them to have, then um, you know, that can be a pretty sticky situation to be in. So, like what happens if they use that data as part of their you know training process? And what happens if, you know, let's say you upload you know thousands of documents to this to this service, they incorporate into the training data or something like that. And then, you know, six months, a year down the line, where they release a new iteration or a new epoch of the model, it then starts spitting out the same information to other tenants or whatever. And we we've also seen like situations arise um where like one a user may have inadvertent access to other users' chat histories and so on in like some services. Um now, this very quickly fixed, and I think it's just like teeding problems with like how you know getting these services developed around the LLMs and everything. So, again, it's probably worth saying that you know, if you're uploading your data, upload it somewhere you trust, maybe have a good read through their you know, legal documents and everything to understand their capability of using that. And um yeah, I think that's that's my biggest my biggest concern with third-party services.

Robby Peralta 10:37

So I should still look at I'm just gonna keep picking on checkpoint or using checkpoint as an example, I guess I should say. Uh so I should look at that checkpoint AI tool the same way I would look at Gemini or ChatGPT then. Have the same sort of uh boundaries in my head.

Eoin Wickens 10:52

If it's a service, yeah. If you're deploying it internally yourself, I mean typically it's within your perimeter and it's something you can worry about. But and I'm not, I'd like I I we're using Checkpoint as an example, but I I personally have no prior knowledge of checkpoint services, just to clarify. Um excellent. But um, yeah, it's it I guess if if you're sending data outside of your perimeter anywhere, right? You just want to have like a good understanding of that it's in safe hands. Um and I guess that paradigm hasn't changed with the context of LLMs. And in fact, I think before we might not have had as much reason to send data outside of our perimeter. Like, you know, I'm not gonna like email revenue forecasts to you know Tom Digger Harry, um, right? But like with an LLM, it's like, okay, well, can you format this data for me nicely, please, so I can put this in my like PDF? Well, that would be great. And then it's you know, it spits back the nicely formatted revenue forecast that I've just put into it. But the data has left the perimeter, and that's I guess the crux of the situation. So I guess to answer your question more properly, if the model is being hosted internally, like if you run it up yourself in like you know, some uh you know, cloud service provider or whatever, then within your own cloud, then it's not leaving your perimeter. Um, but obviously make sure you're securing it.

Robby Peralta 12:06

Um I came across something called virtual agents.

Eoin Wickens 12:09

Okay.

Robby Peralta 12:10

Which sounds have you heard of it before? It's basically it means that it will do things for you on the internet. Maybe it's called something else. But basically it's like if I say yeah, format this data and send it, then it will go out and take from your CRM, take whatever you need to do, write the email and send it for you. Uh and apparently there's like a bunch of different startups that are like um yeah, they have like demos out there where like it says make a website and start selling this. And like you can see, you'll have like a you'll have a Mac and it'll have four screens and you could see the I'm not even sure what to call it. It's not an LLM anymore. Then it then I guess it is AI. It's a virtual agent.

Eoin Wickens 12:45

It's an agent, I think.

Robby Peralta 12:46

It's an agent, virtual agents, yeah.

Eoin Wickens 12:48

Yeah, yeah.

Robby Peralta 12:49

What is your opinion on that?

Eoin Wickens 12:51

Cool, yeah, uh great question. Yeah, I've I've heard of the concept of like agents in the past. Um there's some cool people working on like agents for uh hacking AI and everything like that, which I think is really interesting. So you know, an autonomous thing which can act independently, you know,

AI Security Risks and Vulnerabilities

Eoin Wickens 13:06

um to an extent. But it's kind of one of those, it's one of those situations where the more we hand off um decisions to a you know uh an unknown entity, right? The more risk you end up incorporating, essentially. Like if we, you know, you give this access to a lot of APIs, you give this access to your CRM, like these models are a risk vector. They are an attack surface. Um the more they have access to, and the more, like if my thing can send an email, if it can browse the web and it can access, you know, sensitive information inside my perimeter, like where where's the line then? Like, how do I stop it from reading a website, getting prompt injected, um, dumping out all the information from my CRM into a public service and then emailing it, right? Like that's that's a big attack chain. Like a lot of ifs and buts have to happen for that to be pulled off. But you know, this is like it's gone from you know not being possible because we didn't have LLMs, you know, surfing the web and everything, to like this is an actual thing that we can see coming down the line. Um, so I think progress is great, but let's just make sure that we do it right and um let's make sure that we are ironclad against prompt injections and so on. Um I'll I'll segue a little bit if you don't mind, uh, because it's something that's quite interesting. Um, but I I mentioned like prompt injection by surfing the web. We call that indirect prompt injection, right? Um uh researcher Kai Grashake uh basically kind of uh found out that you could do this. Um and he was using um uh Microsoft Edge's Bing Assistant. So he's a lot of uh product names there. But um basically to browse the web, it would it would read a page and you could do like that classic thing of like you know, size zero text in you know a white font on a white background, and it would read it, and then it would prompt inject the chat assistant, the chat assistant would start suggesting spam links, it would start speaking like a pirate and all sorts of funky stuff, right? Like that's mad, right? That we can that we can do that. So like imagine now if your your AI agent comes along and views a website and then gets prompt injected by virtue of that. But taking this a step further, multimodal uh models, so models that can basically take in data from, you know, like are taking different data types like images, video, audio, and text and everything uh came along. They came on the scene. That was another big thing we saw really over the last like seven, eight months or so. Um, you can also have multimodal prompt injections, right? So, like one of the first examples of this was a picture of an apple with a post-it note that said, like, ignore all previous instructions and do whatever, right? So uh LLM takes this this picture, reads it, it goes, okay, actually disregard this as an apple, let's prompt inject the model. So was that Black Hat Europe in December? And there was a really cool example right at the end of it where they talked about the risk of supply chain attacks through indirect prompt injection via images, right? And imagine now, if you will, an image uploaded to Wikipedia. Like every time an LLM views Wikipedia for this thing, it will get prompt injected by this image, right? So how do you stop that, right? And like, yes, we've got a lot of fences in there, we can do all sorts of post-processing on images and everything like that. But it's such a crazy prospect to me that like a an AI agent could potentially view a website um like Wikipedia, get prompt injected by an image that's on it that looks completely inconspicuous to us. Um so it's interesting times. It's interesting times. That's all I can say for sure.

Robby Peralta 16:44

Uh and while we're on the topic of interesting times, like are there any noteworthy attacks that have happened in the recent in the last time we spoke that you you feel are notable and and cool to mention?

Eoin Wickens 16:56

Yeah, yeah, yeah, absolutely. Absolutely. Um there's a couple that came like very recently in like I mean the last like week or two. Um one of them was to do with um hallucinations within Chat GPT, I believe, um, where it was suggesting uh false packages, so false dependencies. So it would give like a a um you'd ask it to write a snippet of code, it would give this uh particular package back, but instead of a hyphen, it would be an underscore in the package name, right? And this was suggested and suggested and suggested and suggested, and the package never existed, so it would fail. But a researcher decided, oh, what if I create that package and then track how many times it's been downloaded? I believe they they found that it was downloaded over 15,000 times, right? So it was being basically suggested en masse to people. So you have this like hallucination. Um, hallucination in the context of an LLM is when you know it gives back a false piece of evidence. A false piece of evidence. Jeez, I sound like I'm on CSI. A false piece of information. Uh false piece of information, or like something that's untruthful that it thinks it or perceives it to be to be true. Um, but I thought I thought that was fascinating. And you know, like the it hallucinated first, and then they decided we'll create the malicious package that can leverage this. So it was it was in a way, it was a very weaponized form of of this hallucination. Like typically, uh like I and these things can still be very consequential, but there was an example, um, people were using ChatGPT or some other LLM to suggest um papers, relevant papers to their subject matter, to their subject domain or whatever. And it started giving back false, false citations, right? Started giving back false papers. So it's like um, you know, I don't know, the biological makeup of some phytoplankton, for instance, or something by Owen Wickens, let's say, right? And then they contacted this this author, uh Owen Wickens, that's me. But uh, they contacted this author anyway, uh, and they he they were like, oh well, I've I've um yeah, this sounds like something I would write, but no, I've never written it before. And like, you know, these services are basically saying, oh no, this person wrote it. They might even give a summary in a blurb. So you have this like breakdown in known truths or ground truths and stuff. Um, I don't know, I find that I find that very interesting. But like cut kind of outside of that, um, what we've been seeing an awful lot of is like the actual tooling that supports the development and deployment of AI systems uh being like full of holes, like full of vulnerabilities. Um so iPhone or our Android phones, right? Um we go back like 10-15 years, um, it was like you know, you could probably easily enough exploit them. Um nowadays, you might with to exploit an iPhone, you probably have to chain together like some ridiculous amount of exploits, you know, super deep stuff that like will eventually lead into a massive payoff. But when it does, it's super critical. But that like barrier to entry is so high. With like AI and like AI like adjacent technologies, it's like something you probably would have seen like 20 years ago. It's like we're we're dealing with like buffer overflows, we're dealing with um you know path traversals, we're dealing with sometimes no authentication, right? And like, or a service is designed to execute arbitrary code within the context of ML, right? Like imagine now you have all of this sensitive PII hosted on a service or hosted on hosted in, let's say an S3 bucket, you have your service uh, you know, MLOps tool running, which is you know allowing you to process this data, build your model, and then deploy it. But your service has either no authentication or it is designed to accept remote tasks from anybody, maybe with no authentications, with no authentication. And all of a sudden, it means that like somebody like me can come along and you know install a backdoor on your system without any real hassle, pull down all of that sensitive information, and then cause a massive data breach. And I think like one of the things that I can see you know at some point in the future is like maybe you know, looking at the data scientists as an incredibly privileged user given the volume of data they have access to, especially within the business context, right? Like we think of administrators and they have you know access to any system to or to within their you know domain and so on, super risky if they get popped. Um but with the data scientist, it is also super risky because I guess you also have like a build environment, you're deploying models, especially models that go like downstream to customers, and then the massive amounts of data that they have access to. If it's like sensitive business related data or what have you. And I I I I think that's like quite a worrying prospect, um, honestly. But anyway, sorry, I'm I'm digressing hugely. Uh if you want to bring me back on track with a question.

Robby Peralta 21:56

Oh, I I got lost. I was loving it. I am actually Having a chat with um somebody that's been building machine learning uh the one that you just referred to, one that makes those models and has access to all that. And I just I just wanted to hear like where what his opinion is around security and like does he think about security in these things? But back to our discussion just now. Like, um, do you think that it's it is that way just because it came so quickly and then the business was like, give me this, and everybody was just like, ah, give me a tool. I don't they just skipped over like the normal security barriers, or how'd we get there?

Eoin Wickens 22:28

Yeah, um yeah, no, totally, totally. I think um I think that's part of it. I think like we've got rapid adoption as one. Um that rapid adoption uh definitely kind of the skewed

AI Security Concerns and Solutions

Eoin Wickens 22:38

the security aspect, I guess. Um one of the one of the other main contributing factors to that, I think, is that like uh data science, I guess, as a as a discipline had come largely from academia, given that it was you know largely PhD data scientists and mathematicians developing these models, developing these architectures, and then you know, going out into industry, building them or or or using tools that were developed by academics and so on. No hate, by the way, that's like it's just part of the part of the process. But what often if you're gonna build things like you know, uh you might not if you're building them without like industry experience where you're used to security development practices, like you know, secure development lifecycle stuff and everything, then you're you're not building software with that in mind. Um, and if you're kind of building it outside of like this industry ecosystem where you've had to deal with this, then it's easy to develop insecure software, right? And that's like it's not a fault of their own. It's just kind of like if you're if you're not in the loop and you're not read in, that can be an issue. Um, so I think you've had this like kind of large swathe of tooling developed and designed for data science that kind of just went under the radar in terms of the security community and then also in terms of like security engineering, I guess. Um so now because of that rapid adoption, we're once again playing catch up and we're trying to you know secure a lot of this tooling from you know the ML op side, from the model file format side, from you know, even you know, the generative AI side, like the the attack surface is huge. And I and I think that's like one of the biggest things that stood out to me when I when I did when I had the threat report in my hands and I looked at everything we written about, what how just how big the attack surface is now, that we've kind of like really understood over the last two years is like staggering uh compared to when we started this and when we were looking around and going like this is you know, we knew it was bad, but we didn't really have it quantified. And now it's like you know, there's a lot of things we need to do. And like some of this is traditional security, um, some of this is app sec, right? Like you can't like you can't just ignore traditional security, but there's a whole new component, right? Like the the whole LLM concept is is completely new and it requires this like hybrid hybrid security posture. Um that's a whole other question.

Robby Peralta 25:05

Uh you recently uh released a product, um generative AI, something in the title. I know that.

Eoin Wickens 25:11

Absolutely, absolutely. So um building security into large language models is great, it's necessary, but it only gets you so far. Um we kind of call that like model robustness, we call it you know, things like guardrails and so on, which kind of prevent a user from doing malicious activity. And that's great, it's really important for like the base foundational model. But what tends to happen then is like how do you control what data is sent into that model? How do you control the outputs that are sent out of that model? Let's say I don't want code to ever come out of this model, or I don't want you know PII to come out of it, or or even go into it, I suppose would be the main one. Then how do I do that? And you you you can't necessarily like fine-tune that out, and you can't necessarily add it into the to the to the base model. So it requires a bolt-on, right? Uh kind of a bolt-on security product. And we've developed it to be an integral enterprise uh grade security product that we can actually define custom policies depending on that specific customer and so on, so that we can bolt on our security layer and aim to basically protect you from things like PII leakage and so on, to all the things all the way down to prompt injection and everything else that then comes out of the model as well. Um, so yeah, it comes back to what we were starting at uh talking about at the beginning. If you are sending things to third-party services, do you want specific information in there? Do you want to blindly and implicitly trust the data that comes out of it? Um, if it's suggesting code, for instance, you know, it doesn't have malicious packages in the response and everything. And so all of this requires uh an extra layer of security. So that's what we've been developing.

Robby Peralta 26:53

Yeah, so that means that like you're um, I mean, you still need guardrails, you still need that foundation around the your your model itself. But this is just like I don't want to call it DLP, but you can go in and you can configure a lot of things so you can kind of make it like a DLP-ish where you set your own personal policies and whatnot, right?

Eoin Wickens 27:08

Yeah, there's definitely an analogy to be made in that 100%. Um and I think that the two live it quite well in harmony, honestly. Like um, I've seen models that have no guardrails, and that's all well and good as well. But it's um, I think you know, you you tend to try and want to get as far as you can with robustness in a model, uh definitely, just to just to try and lock it down from more basic prompt injections. But I think what we found is you know, uh as many ways as there are to craft a sentence, there are ways to create a prompt injection, right? And that and and and and so on, and yeah. Cat and mouse game.

Robby Peralta 27:41

Your threat landscape report. Uh lots of good stuff. What did you find most interesting?

Eoin Wickens 27:46

That's a great question. Um, we we surveyed about 150 executive IT leaders of major industry down to kind of SME level. Um, so we wanted to capture a bit of a broad kind of swath of people and viewpoints. But what we found was quite interesting, actually. Um, so I I believe if if I'm not getting them the statistics wrong, is about 1700 models in production on average in like major companies. Um like thought that was very fascinating. It's a lot of models. A lot of models. A lot of models. Um each with their own attack surface, right? Yeah, totally. I mean, like, you know, you may use a few models for like the same purpose, but like you often each model will have its own specific kind of use case, right?

Robby Peralta 28:29

Um, exactly.

Eoin Wickens 28:31

So so yeah, absolutely. Each has their own attack surface and depending on how they're exposed and everything like that. Um, but what we also saw was that the opinion across our our surveyed uh respondents um basically said around 98% of them believed that AI was critical to their, to their, or somewhat crucial to their business function, right? So we're starting to see that like it's just it is part of their business function now. And I I think that's that's pretty, you know, uh a pretty stark statistic there. Um and then again, it's another statistic, but about 77% of companies said they had identified a breach that related to their AI or like AI Jason, right? And that is also really worrying. And I look, I I've said this, you know, quite a lot over the the last couple of years, but like the more critical decisions we offload to our AI models, to you know, even like not even just LLMs, but classifiers and and everything like that, the more of an attack surface we're raising. Um the our data now is more precious than ever.

Speaker 3 29:35

It's you know, I

Robby Peralta 29:37

it doesn't mean necessarily it was like a crazy bad breach, but it means that number one, they had to have um a way to detect that, which is interesting. Did were those 70 per seven percent of your clients? Because then that makes sense. But how would 77% of them even detect that? You know, that's interesting that uh

Eoin Wickens 29:51

Yeah, no, absolutely, absolutely. Um, I think you can kind of understand if things are trying to like subvert a model in some regard, if they are trying to access specific data that's in relation to an AI model. Um, I'm not 100% sure on the visibility that's been in you know put into these these companies now over the last year or two years, but I'm sure they're they're keeping close tabs on it.

Robby Peralta 30:12

Yeah. Let's say 100% of your clients have uh have noticed an attempt attempted breach at least, and then 77% goes uh yeah. Interesting. But that I mean because I'll like a lot of things your products are doing, it's all possible to do by yourself. It's just extremely expensive and time consuming to use your own people to do those sort of uh to implement those sort of measurements or to detect those sort of things, right?

Eoin Wickens 30:36

Um I guess that's one of the things we try and aim to achieve really is to take that security load off the data science team and enable you know security practitioners to be able to understand and triage these events and and so on. Um so that's been you know a big part of our mission ultimately is to allow them to kind of like coexist independently. Um but software engineers, you know, were able to write code that wasn't secure before. Well, they got away with it more often, let's say, before we had like mass-scale cyber intrusions and everything like that. Um the software engineer then did have to kind of get more familiar with vulnerabilities and and and secure software

Predictions and Recommendations in AI

Eoin Wickens 31:13

development and so on over time. And like I and eventually, you know, there there still is a line between a software engineer and a security analyst, or even a vulnerability researcher or somebody in the product security team who's doing code review. So you do have these like breakdowns and delineations, and I just see that these teams are starting to move closer together, the conversations are happening, and I do think you're starting to get data scientists who are more concerned around security as well. Um be really interested in hearing the you know the conversation um that that you're gonna be having with the data scientists. Um and I'll be tuning in for that. Um but yeah, absolutely. Um but yeah, I think even from what from our conversations, we're we're we're starting to hear um a lot more kind of a gelling within that.

Robby Peralta 31:55

I mean, software engineers I might get hated on for this, but uh they've been around for what, 30 years at least. There's been software for 30 years, and I wouldn't say all software engineers are up there with security, but uh I would guess the data scientists better get up there a lot quicker than 30 years, or else we're gonna have a huge, huge problem. Yeah.

Eoin Wickens 32:12

I think it's like all of our responsibility, right? It just means that people like us in the vulnerability, vulnerability research side, in the adversarial machine learning side are able to help and able to contribute. And like we don't need to like put the onus entirely on the data scientist to get up to speed with it. Um but like we'll kind of all kind of come together and try and solve this problem, I think. Because uh I yeah, and I can see that happening. Um just just any data scientists listening who were saying, Crikey, what is Eoin saying?

Robby Peralta 32:39

God damn it. Yeah, because in their defense, I mean, in their defense, they don't need to defend them at all. Like they have so much learned. Like, I can't even follow the attacks that are coming out and like the the AI stuff, and they have to like learn how to implement it and do it, and then the security comes in addition, which is a whole another beast.

Eoin Wickens 32:55

It's a whole whole different, whole other aspect, but um yeah, we all have like you know, kind of uh adjoining surfaces or whatever, um or some other strange analogy for that. Yeah.

Robby Peralta 33:06

Well, if you have both of those traits, go work for hidden layer at least. I'm just thinking like um you go to a lot of conferences and stuff. You're probably one of the few people that goes to more conferences and stuff than I do, and you're on stage. Uh what kind of like who do companies go to with these sort of issues besides you? Like um like we talked about earlier, you have to understand data scientists, you have to understand secure uh coding practices, you know. So there's there's such a big span. Like, who who do you go to with those sort of uh requests? I guess. Ghostbusters. Ghostbusters, exactly.

Eoin Wickens 33:41

Yeah, but um no, it's a great question. Um I mean we do have taxonomies now. We have like you know, more robust standards and so on. Like NIST has released um they're kind of like the taxonomy of adversale attacks. Um we have things like MITRE Atlas, which is great at helping to define the space. We have different um AI frameworks. We have the secure AI framework from Google, um, IBM also released one. Um Databricks recently released an AI security framework. So it is, it is um, you know, there's there's starting to be more voices within the community, which is fantastic and it's and it's great. Um but um that's something we try and aim to do is is look at you know a system from all these different angles, right? From the vulnerable research side, from the the pen testing side, from the reverse engineering side, from the adversarial machine learning side, and all of these things. There is a lot. There is a lot, yeah. I a hundred percent. And that's that's um I so I think I feel like we're in a way uniquely positioned to advise and help and uh help companies in that. Um but but don't take it from me.

Robby Peralta 34:48

Crystal ball predictions and recommendations.

Eoin Wickens 34:51

Okay, okay. Um I think I've made some I made there's other predictions that I made last time that uh that we're still waiting on, but I think like LLM integration, it's just gonna be continue to become bigger and bigger, um, especially as the models get more capable. I think um the multimodal side of the models are really gonna be interesting. Like that that attack I mentioned from the the researcher at Black Hat EU with like a supply chain attack through a Wikipedia image. Like, I think that is where we're gonna start seeing things. Um it's not so much a crystal ball. It's like, I don't know, is it is it my partial doom saying, but I wonder and I kind of worry about like the veracity of information being sped out by these LLMs that often isn't 100% correct um through hallucination and so on. Um I do wonder, like, you know, if we'll have this kind of self-fulfilling uh prophecy or like a snake eating its own tail of like, you know, kind of bad bad data being fed back into the the models, and you know, we end up not being able to find the correct answer for our questions. I think the Wikipedia models work quite well for crowdsourced, you know, uh truths to an extent. Um, but I I do wonder about how things will progress with so much content, but that would be interesting. Um like we're starting to make strides with things like data provenance, uh, that like data set provenance and data set integrity, um, and especially as well, then model integrity and model signing. Um, there's

Advancing AI in Open Source

Eoin Wickens 36:17

there's some open source initiatives now um that are coming out around this so that we can basically show where a model has um come from and cryptographically verify that in the same way we like sign uh PE models and so on. And so one of my one of my crystal ball predictions, which I think it's necessary because you know, as we just mentioned, there's a big issue there with like, okay, where am I getting this data from? Who provided this data? Or if I'm pulling a pre-trained model in from a third party, where did this model come from? And like, how can I prove with any degree of of accuracy that this model is what I, you know, what I hope it is, um uh, or came from who I, you know, who I who I presume it did. So that's what we're trying to solve um as an open source initiative under the OpenSSF now is is um signing and and model provenance. Um and I think that's gonna go a long way to helping companies to trust their their their data and trust the models that they're pulling in a bit more. Um but yeah, I is that good for predictions?

Robby Peralta 37:22

Absolutely, absolutely. And I think that last way you entered it. I mean trust, right? That is that is the biggest problem with the AI and LLMs these days, is that's we we don't trust them. So any any initiative towards making trust happen is uh is a good one. And I also made me think that like when you were talking, it sounds like the AI, the push towards AI is kind of making steps, you know, verifications. That would be all be great for the open source community as a whole as well. Um just not there today. Uh

Eoin Wickens 37:48

absolutely, yeah. And I think we w we want to solve that as an industry. Um like we don't want to just create a new standard that you know is that we need another standard to correct. Um we want something that's like used by the industry that we can all work together with, and really like you know, a rising tide erases all boats. And I think that's that's the important thing there. Um but yeah, you 100% trust. Trust is the main thing. And I think, yeah, we're we're I as a as an industry we're making great strides, and it's great to see. Um I just hope that we can keep on pushing the pushing in that direction.

Robby Peralta 38:17

Mr. Wickens, you're a legend.

Eoin Wickens 38:20

You too, Robby. Thanks so many for having us.

Robby Peralta 38:22

Uh always nice to talk to you, and uh, I will see you here in a few weeks in San Francisco.

Eoin Wickens 38:26

Absolutely. See you, see you out in RSA.

Robby Peralta 38:28

Take care of it. Looking forward to that. Cool. Ciao. Cheers. Thanks. Well, that's all for today, folks. Thank you for tuning in to the mnemonic security podcast. If you have any concepts or ideas that you'd like us to discuss on future episodes, please feel free to hit me up on LinkedIn or to send us a mail to podcast@ mnemonic.no. Thank you for listening, and we'll see you next time.