LLMs in Security Products

In this episode, Robby speaks with Harry Wetherald, Co-Founder and CEO of the security platform Maze, about the current wave of LLM innovation in security and how to separate real progress from marketing fluff.

Drawing on his experience building security products, Wetherald shares how large language models are changing the way we approach vulnerability management, what to ask vendors about their "AI" claims, and why UX may be just as important as the models themselves.

Transcript

Speaker 3 0:03

From our headquarters in Oslo, Norway, and on behalf of our host, Robby Peralta, welcome to the mnemonic security podcast.

Robby Peralta 0:14

A few years ago, when AI was becoming a thing, my good friend and boss introduced me to something he called the God of the Gaps. When we didn't understand thunder and lightning, we assumed it to be Thor and his hammer. When diseases and the plagues were killing us, we assumed it was punishment from the gods. If it was beyond our ability to understand or describe, we let mythology do the explaining. And fast forward to this episode in 2025, AI seems to be the new god on the block. Now I've seen lots of PowerPoints during my years as a friendly neighborhood sales guy. And although I haven't seen God in any of them, I'm pretty sure I haven't seen any in the recent past without AI. So for the rest of this conversation, our guest will walk us through what he is thinking every time he hears AI in a product pitch.

Harry Wetherald 1:19

Hey, cheers, Robby.

Robby Peralta 1:21

Where are you joining us from today?

Harry Wetherald 1:23

I'm joining you from London and the UK. Uh hence the accent.

Robby Peralta 1:27

My future home. Soon in August. I was looking at uh looking at your profile, preparing my brain for this episode, right? And it said Stealth. I was like, oh, that's an interesting name, Stealth. I'm surprised that somebody else didn't have that company name. And then I realized later, like I'm just super noob to like this whole venture capital or whatever world. And I was just like, oh yeah, it just means they haven't released their name yet.

Harry Wetherald 1:49

I get we get that all the time. My favorite is on uh on LinkedIn, you get all these automated outreach messages. And I I frequently get, I'm gonna see stealth is just like what thousands of people around the globe are using to basically hide their company name for now. Uh, but you you get all these automated messages being like, hey, hey Harry, I actually love what you're doing with stealth. You know, uh love I've loved reading about everything you're doing, and it's like we literally have nothing public, so um, yeah, it reveals them pretty quickly, which is always quite fun.

Robby Peralta 2:17

They must be using AI then if they know all the answers to the to the questions, right? Yeah, exactly. Speaking of AI, um, is there anything you can say about it just as an introduction to yourself and what your background is?

Harry Wetherald 2:29

Yeah, yeah, absolutely. So yeah, I'm currently CEO and co-founder of a company called Maze. We are in stealth at the moment. We'll like slowly be sort of revealing more and more about what we're doing uh soon. The very simple kind of story is we're using modern AI, i.e., you know, the latest and greatest models, large language models, agents, et cetera. And we're throwing them at a bunch of kind of old but very painful security problems, in particular around kind of vulnerability management and how people are dealing with vulnerabilities day to day, which is a problem that's tried to be being sold a million times over, still is a huge, huge problem for most companies. So we're seeing if we can basically build uh you know an AI-based system that uh that addresses that.

Robby Peralta 3:04

Cool. Well, this is your first podcast. This is Maze then, maybe, or yeah, yeah, yeah, exactly. I think it is. I think it is. Oh, nice, nice. Awesome. Well, thanks for coming on. I um I spent a lot of time uh tuning my LinkedIn. I kept seeing like posts from you come up. And one of the posts that I that caught my eye, a simple way to call BS on security products that use AI. Do you remember that post?

Harry Wetherald 3:28

Yes, yeah, yeah. Just a few weeks ago.

Robby Peralta 3:30

Do you want to just explain that for me? Because I I I thought it was really interesting.

Harry Wetherald 3:34

Yeah, yeah. So um, I guess a bit of context behind it. Like I've worked my whole career, last 10 years or so, has been basically working in and around machine learning in all sorts of different ways. Um, and so I've been in and around that world forever, right? And um seen a lot of the good in that world of people actually using machine learning to create really interesting novel products. But it but with every wave of machine learning and interesting products you get with it, you get these waves of people that basically a team of marketers or executives in an office somewhere have decided if they just add the word AI or machine learning or whatever the kind of like buzzword of the moment is to whatever they're already doing, suddenly the whole world is going to take notice of their product, right? It seems to be the logic that goes through a lot of people's heads, which makes life really difficult when you actually are trying to do interesting things with machine learning. Um, so you know, this was the same. Previously, I used to lead product management for a company called Tessian, which was using kind of a more like 2015 kind of era machine learning, that kind of era of machine learning to basically do interesting things in email security. Now obviously we're doing the same thing again. And what that post was really just trying to demystify for people a little bit how to read when a company says to you, hey, we're we're using AI, like what does that actually mean? And some questions, some simple questions you can maybe ask to kind of like decipher it a little bit and try and get to the bottom of actually how interesting is the AI in their product. Just because there isn't AI doesn't mean the product's bad in any way, shape, or form. But if they're trying to tell you that it uses AI, then maybe asking a few simple questions could kind of help you understand a product a bit better. So that was the background of that post. I guess it was like uh 10 years of frustration like bottled into one post kind of.

Robby Peralta 5:02

Yeah. What were those questions that you uh you think that you should uh ask those vendors?

Harry Wetherald 5:07

So um, so yeah, there's a few things. So firstly, like to set it up a little bit, there's machine learning as a technology has been around for, I don't know, 20, 30, maybe plus years. Uh people who've been working on AI since the, I can't remember, like the 70s, 80s when they were working on expert systems. So none of these concepts are new, but there's kind of been a few waves of them over recent years which have like drastically improved their capabilities to a point where it's kind of what people understand by I use AI or I use machine learning, right? So those waves in recent times was around kind of like 2012 to 2015, you had um this kind of wave of machine learning becoming much, much, much more useful. Right. So it was about that time I think Google or or similar kind of were able to prove that they could uh recognize images better than humans in the early kind of deep learning models that happened around 2015, 2014 kind of time frame. And there was a bunch of innovations that happened around then. And it was kind of like you had um cloud, uh, big data processing, and these like improvements in machine learning all happening at the same time. And what you got out of that was this big wave of machine learning backed products that suddenly went, hey, let's take an old problem, let's build a product from the ground up to use machine learning, let's take all that problem. And you see that across a bunch of different security categories today. Um, email security being one I obviously know super well, but there's a number of other categories that kind of followed a similar uh trajectory. So that was one wave. And then around kind of like obviously like 2022, 2023, you've got ChatGPT coming out and large language models kind of coming into the consciousness of the wider population. Um, language models have been like uh been built on since about 2017, 2018, uh, but they only really started to get really, really good around 2022, ChatGPT then explodes it into everyone's consciousness, which means you then have another wave of companies starting to come out now that have been building on that kind of like modern AI, uh, meaning large language models, to build new products. So, all this is to say like one of the best ways to understand how what AI like behind the scenes a product is actually going to use is just when was it founded? Right. So if a company was founded in let's say 2010, right, the core of their technology was built before that kind of wave of ML-based innovation that happened around kind of like 2012, 2013, 2014, et cetera. So naturally, even though they've been built in 2010, they could, of course, in 2015 have built parts of their product using machine learning, built new features using machine learning, et cetera. But the way that it works is just nine times out of 10 or 99 times out of 100, they don't go and rebuild that core bit of their technology, right? Um, because it's it's their crown jewels, it's what they're known for, it's what they feel uh works for them. Um and they basically phase this kind of innovative dilemma where they don't just go and press delete on their code base and start again, right? They try and like bolt stuff onto the edges. So one of the best ways for me that you can understand uh what kind of like AI machine learning a product actually uses and how critical it is is just when was it built, right? And so products built around the mid-20 teens, right, to late 20 teens could or could not use machine learning in a really meaningful way, right? Some of them could have been built with a lot of machine learning from the ground up, others could not. It like doesn't really matter, but you know that there's a possibility. Something that was built in, let's say, like 2005 or 2010 for the first time, probably doesn't use very much machine learning at its core, right? It's probably just kind of bolted on features later in various places, um, or at least like the modern variants of machine learning. And then you just just apply the same logic to recent products, right? So that's where it gets interesting because LLMs and ChatGPT, et cetera, are still so new. Anything that was built pre, honestly, like 2022 for the first time, 2021, probably isn't using large language models at the core of their product, right? And this is where you particularly see it in security today. If you go around like RSA or any conference, honestly, eight out of 10 like conference stands are just like, hey, we do this with AI. You know, like everyone has just kind of stuck AI into their marketing, which I totally get why they why they've done that. But nine out of ten of those companies or or even more were not started pre-2022, right? So the crux of their technology, by definition, does not use large language models, right? Because it wasn't around then. So the core of their technology was built on something else, either kind of like previous waves of machine learning, or honestly, most of the time, just more on like traditional software, traditional logic, rules, stuff a human has predefined up front, basically. Doesn't mean to say those products are bad, right? Some of those products are like the leading products in the industry. It just means to say that they're not really using modern AI, meaning large language models, right at the very core of them, because they just can't have done. And what you then see is because they don't want to go and delete all of their products, right? That's not a rational thing for them to do. They start looking at how they can integrate AI in large language models. So they start kind of like putting it around the edges, right? So, like um, you know, the copilot wave that we saw around the seams was a great example of this. No one wanted to go and delete everything they'd built with their seam. So they thought, hey, what's the easiest way we can do this? We can just layer a copilot in our top, which means that you know, a SOC analyst, when they come in, they can write natural language and we'll translate it to a query that goes and queries our scene. Right. That was like a classic thing you saw in the early phases. That is technically that product using large language models, but it's not a core, you know, um critical part of the product, basically. So that was one of the main ways I kind of uh summarized an article of looking into this. Just like when was the product built and what does that tell you about how core AI is to the system and what generation of kind of AI or machine learning is being used?

Robby Peralta 10:19

I'm sitting here trying to think is what is the difference between a product if an LLM was bolted on afterwards, it's a core feature.

Harry Wetherald 10:26

The best way of thinking about this is just like the the kind of um physics, if you will, of the situation of building products, right? So you've got to imagine there is thousands and thousands and thousands of well-funded companies around the world all trying to solve the same problems, right? And there's lots of very, very smart people trying to trying to do that. People that have got every kind of conceivable version possibility around software to do it with, right? So you've got to imagine that we are uh we can maybe get better in places, but we're pretty efficient at basically finding the best solutions using the available tools we have today to solve problems, right? What that means is that it's unlikely that you could go and find a company that was built like five years ago and be like, hey, we're gonna like go back in time to the technology that was available five years ago and we're gonna come up with something that's like five times better than what they did. Because they probably chose pretty much the best approaches for their time, right? And so the the reason why it's important to understand how core AI is to a product is to understand if it can actually be 10 times better than what you expect it to be, what you've seen before, right? Because basically, if you encounter a product and it says it and it's it's going on and on about how it uses AI, but the core of this technology, you know, 90% of the value you get from it is not using AI, that does not mean to say it's a bad product. It just means to say it's probably gonna act and behave much like products you've seen before, right? So you should not expect it to be, you know, 5, 10x, 50x better than what you've used before, because it's still using the same technology at its core that was available three, four, five years ago. And there's so much competition in the market that we're very, very good at getting the most out of technology at any one point in time. Which means basically that um the only way you can really expect a new product to be groundbreakingly different and better is if it's using a completely new technology. And right now, large language models are you know the most prevalent example of that. But you can imagine, you know, you can take the same example with lots of different kind of technology, technological innovations that happened over the years. If you're still using at the core of your product something that was available three, four, five years ago, it might still be a good product. It's just not going to be groundbreakingly different in most cases, basically.

Robby Peralta 12:25

So you said earlier that you and your team, Maze, are using uh LLMs on security challenges where you think they're most effective, correct? What are those areas if you don't mind going into that?

Harry Wetherald 12:38

So for us, for us, we're basically focused on how to how to help companies understand where they need to resolve security issues in their environment, right? So that there's a whole wave of companies focusing on trying to help with threat detection and response, right? Find attackers, deal with SOC incidents, etc. We're not looking at any of that stuff. We are just purely focused on where are the potential risks in your environment, vulnerabilities, misconfigurations, these kind of things. And basically, how do we give you this kind of like AI-assisted um way of dealing with those risks, right? So investigating them, triaging them, resolving them. For us, like it's the only like meaningful path forward that exists around that world because there is so much noise starting to pile up for so many teams, right? Their backlogs are growing bigger and bigger, everything's getting exploited faster and faster. So we're basically trying to kind of like shift the advantage back to um bite a defenders where we can, um, which has been yeah, it's been super interesting so far.

Robby Peralta 13:27

Cool. Why are LLMs useful for that exact issue or challenge?

Harry Wetherald 13:33

Yeah, yeah, interesting. So uh they're useful for that challenge for the same reason they're useful for like almost every problem we're facing in security today, honestly, which is like if you look at problems that we've tackled across security, they're almost always some kind of variant of we have a bunch of different data sources, right? So like a security product almost always looks like give me one to n data sources, I'll look at them, and I'll tell you if thing is good or bad, right? That is basically, you know, every single security product in the world distilled down to its like basic elements. The problem is that almost every security product in the world struggles to understand fully the context of those inputs that's been given. Right. So it looks at a few different data points and it tries to kind of like guesstimate is the thing good or bad or not? Should I allow it or not? Should I flag an alert or not? Should I um should I call this a vulnerability or not, etc. The reason why like I believe modern AI is so effective for um not just from we're working on, but uh all sorts of problems in security, because it solves that kind of like context problem. Right. So basically what it allows you to do is um feed many different dates of sources of data in, and the AI can basically understand the true context and relationships between those different sources of data. And you can actually get AI to kind of step through step by step through different investigations, going and looking for new data where it needs it. So they can start at one place and say, hey, right, I've got this bit of context, I'm gonna understand it, I'm gonna then figure out where I need to go next. I'm gonna take that bit of context, understand it, figure out where I need to go next. Right. So that kind of like iterative process of understanding the context of the different kind of piece of data you're looking at is basically a roadmap to solving like problems across what we're working on and across like almost every other sector of security, which struggles with some kind of like accuracy problem, basically. I think it'll take a bunch of time. Like in some places, the technology isn't quite like ready yet to fully kind of replace what we're using today. But I think what you'll see over the next five years is basically some level of kind of like AI-based reasoning starting to basically just replace all the different kind of rule-based or maybe traditional machine learning-based um kind of like products that we've we've used for many years, uh, which is really, really exciting, right? Because I think what it finally gets us a uh opportunity to do is basically stop having so much noise generated by all our security products, right? If we can make them all more accurate by a factor of 10 or a factor of 100, we get way, way less noise. And all the people involved in security waste way less time and spend way more of their time actually dealing with you know real incidents and actually trying to help kind of improve the security of their companies, which is uh yeah, I'm really excited. Right.

Robby Peralta 15:59

You said something there, the rule-based and and ML-based versus LLMs. Where how do you distinguish those? Because in my mind, machine learning based is kind of or LLMs are kind of like machine learning, but not really, I guess. Yeah, yeah, no, they are.

Harry Wetherald 16:13

They are they are yeah, I terminology in this world is is just such a nightmare. Like everything is like a Venn diagram of each other. Most things are actually subsets of uh of other things, but then the way people use them varies. So oftentimes when I'm using the words terms machine learning, I'm referring to that kind of wave of machine learning that was in use, like it started to become used around that kind of time frame. Technically, large language models are like a subset of machine learning, right? And so technically we are still using machine learning in kind of modern AI systems, just people seem to have shifted more towards using the terms AI or generative AI or large language models, I guess, to try and differentiate from the more traditional machine learning models that um that were maybe a little bit more limited in terms of what they could do, basically.

Robby Peralta 16:53

Yeah, can I say like the old wave of machine learning 2015 that would kind of like flag something then and there, but the new ones, I'm kind of thinking of like uh perplexity has this deep research thing, right? Now I might be kind of going out on a tangent. But what I like about that is it like it doesn't just flag it, it does flag it, but then it runs it over runs it again through all the information it has. So it's not just a point in time thing that which is kind of machine learning, but it's continual.

Harry Wetherald 17:15

Deep research is a great, is a great example. Deep research is using the exact same kind of like step-by-step reasoning that I think is gonna completely change how most security kind of segments work. Because that same step-by-step way of reasoning where you basically like start with an objective, like with deep research, right? You start with something you want to know, and you basically step by step work through different forms of information until you gather up the answer to that question you had. That same approach can be applied to so many different parts of security, and it's that steady, like step-by-step going through the information that I think is going to be like the key approach that that a lot of people um end up using. So, yeah, deep research is a great, great analogy. And yeah, whereas a more primitive machine learning model would maybe would maybe instead just kind of like take a few simple data points and kind of give you a guess of an answer, right, immediately. You know, deep research kind of stepping through the information more methodically and kind of processing a lot more information as it goes.

Robby Peralta 18:05

Is that is that that kind of reminds me of the word agentic or like agent LLM or can you explain that? Or what what do you think about that?

Harry Wetherald 18:13

Yeah, so agentic AI seems to become like the absolute buzzword of the honestly, since like since the turn of the year, suddenly see everyone in security seems to be talking about it. Um, but I like most people still that I interact with still don't really like know necessarily what it means yet. So agentic AI just basically means AI systems that rather than being a single call to a large language model, right, rather than being like a single call to ChatGPT or something like that, you basically stack the calls to the large language model one after another. Right. So rather than saying, hey, I have a question for ChatGPT, you know, what where should I where should I eat breakfast this morning in Paris? And it will give you an answer back. You ask, where should I eat breakfast this morning in Paris? And it kind of talks to itself back and forth for a while. Right. Um, so it's so it's basically passing information back and forth with itself. And it also is able to access tools and other kinds of resources to help it with its with its question, right? So for that same question, it could maybe um start thinking about what I might like. It might be able to access some kind of data source on me that I've given it that can help it understand where I might like to eat breakfast. It might be able to access the internet, it can take actions on my behalf, and it can basically sprint this all together into like a long, step-by-step answer to my question. And by doing it step by step, you get two big benefits. One, you get a much more thorough and accurate uh answer because it's basically been able to kind of like think through the answer. Uh, but B, it gets to go off and do stuff, right? So it can actually take actions, gather data, um, uh, and help itself, basically. So honestly, all these different approaches like deep research, building AI agents, um, like all the new chain of thought models that have come out recently, like Deep Seek and Chat GBT's kind of O1, they're all similar in what they're trying to do. They're basically just giving large language models a lot of time to think and a lot of time to pass data back and forth to themselves until they get to much better answers. You can use it, do it by your agents, you can do it by these models that kind of run for a long time. It's not exactly the same, but it's it's all roughly similar in what it's trying to achieve.

Robby Peralta 20:07

So when you're making a product, um, what Maze is making, I assume that you would have to be making using an open source model, correct?

Harry Wetherald 20:16

Not necessarily. So we're basically like we've built the product as many people do, like built it to be super modular. So you can basically just swap in and out models as and when you need to. So as you said, like, you know, the next the next great new model comes out. You want to be able to basically use as much of that power as you can rather than being stuck with like some previous generation of a model that you used uh before. So we built it that way around. So we're actually just swapping stuff in and out and basically seeing what's what's best. What we do do though is we currently build everything on bedrock on AWS. And that basically means that the data never leaves AWS. Right. So it's a bit like your data being stored in a server on AWS. Amazon holds it, it never goes back to some other kind of supplier of software. So that's what we do to basically limit, you know, we either use open source or we use models available on bedrock, and that means that we never give data over to the model provider, basically. But the ideal is that you can basically just swap in and out models as in where the best model becomes available, basically.

Robby Peralta 21:07

And that's because if you were to base it all on one model, that'd be really bad because next week there comes a new crazy model and you're stuck with the old model.

Harry Wetherald 21:14

Exactly. And there's a whole lot to like making it easy to swap things in and out and validating that your product still works as well when you swap it out a new model. You know, there's a lot of like infrastructure that needs to be built around that, um, which is yeah, partly what we've been doing.

Robby Peralta 21:27

Yeah. So um I had a chat with a bunch of other uh vendors that are in the vulnerability space, right? And I think their strategy would be to uh buy a company and hitch it on their current thing, right? What would you say is the problem to that since it's not core architecture like you guys have done it?

Harry Wetherald 21:48

You get this kind of question all the time, right? Like, oh, why can't company X just simply like they have all the access to all the same models, right? Like, why can't they just go and do it? The answer is like theoretically they can. It's just what you've seen time and time and time again. Of almost every market in history is when new technology comes out, right, the existing products are just slower and they find it more difficult to get the most out of that new technology. Right. So absolutely what's going to happen is some of the big vendors are going to start playing around with large language models, start using them in different ways, different places. I just think what we've seen so far has kind of proven the trend, which is that in almost every case, you've seen them integrate large language models somewhere in their product. And it's been a fairly superfluous part of their product. Right. It's been like, hey, we took our existing alert and we threw it through a large language model to make it more readable, right? Or something like that. And it's like, cool, like that's useful, but it's not actually getting to the crux of the problem. To get to the crux of the problem, they have to go back and they have to actually delete what they've already done, which is much, much harder for them to do, requires much more change on their side, and requires them to almost like give up on what they've done previously, which is tends to be why it happens slower. But yeah, it will happen. Like I hope that happens. I honestly hope like every security product in the world tends towards using more and more modern technology. My kind of assumption is just always that the older companies are just slower to do it and they very rarely kind of like fully embrace the new technology. They more kind of like layer it on top, basically. So yeah, I will see, we'll see what happens here. But yeah, as I said, I fully anticipate that you know most companies around us will stop playing around with this kind of stuff.

Robby Peralta 23:13

Yeah. Or buy your guys' company maybe one day.

Harry Wetherald 23:17

Well maybe. Hopefully we can uh we can live past that and actually get to get to kind of be in a scale company ourselves.

Robby Peralta 23:23

Yeah. So I would assume that you're like living in like the AI LLM world, right? You go to uh you live in London, it's like an AI hub. Do you have insight into like sort of the roadmaps or like the um like what's coming next? I know it's a big question, but like, you know, for me, I just sit here and like, oh shit, perplexity has deep research. That's awesome. And then like it seems it seems like every month something awesome and revolutionary comes out. Do you see these things coming before because they're giving hints to her, like beta mode here and there, or like a little bit, honestly, not not too much.

Harry Wetherald 23:53

I mean, there's so much competition for that kind of information out there in the world. And the model providers actually, because they're under such intense competition themselves, they're actually pretty, they're trying to get ahead of it as much as possible and share as much as possible. So yeah, we try and keep an eye as much as best as possible. We have you know a few contacts within within certain places, but in general, the model providers are just kind of sharing a lot as they go. What we like a few of the assumptions we try and make are just to look at how fast the rate of improvement is across so many different things, right? Like you've got um probably unprecedented amounts of capital being invested in these underlying large language models across the across the world in a way that you could argue isn't entirely rational, right? All the big companies are kind of like just trying to outcompete each other on scale and how much they can invest, um, for very, very, very marginal gains each time. Now, that's interesting for them. I I have no idea who's gonna win in in that world. What it's amazing for is all the companies that are then building on the back of them because they're getting this huge amount of investment on the kind of foundations of their product. So each six months, your product just gets like, you know, almost exponentially better and exponentially cheaper. So we're just trying to live in that world as much as possible, which is just to anticipate that things will keep getting better, keep getting cheaper. And we just want to kind of live on the borderline as much as possible in what's possible, uh, basically.

Robby Peralta 25:10

Uh you say capital, like uh, like I just think of OpenAI and you know, Google and everybody's using so god so much money. Like, where what why is it cost so much money? What are they doing?

Harry Wetherald 25:22

So it all basically comes down to the principle that they basically think that the that there is no kind of plateauing as you scale more and more compute trained models, essentially. So the like go back again to like that 2015 era of like deep learning models becoming kind of popular for the first time and and starting to have an impact for the first time. There were so many orders of magnitude smaller than the models we're now talking about. And what's happened over that kind of like 10-year period is we've had a few innovations around how the models actually work, uh and in particular the big one around kind of transformers and and language models. But what's happened is basically we've kept throwing more scale at them, meaning more compute. Um, and we've kept just seeing the improvements happen at a linear rate rather than kind of like plateauing and falling off. What that basically means is all these big companies are basically going, ha, if we just can throw even more compute at this problem, we're just gonna get an even better model in a kind of linear fashion, rather than it kind of starting to tail off at that point. And that's why you're getting this insane talk of, you know, like I think the soft bank founder was talking of like seven trillion investment needed to make AI like truly, truly work in the way we think it should, which is such mind-boggling as amounts of money, it almost makes no sense. Um, so that's basically where the competition is coming from. They basically, the the perceived logic is that we just keep throwing more compute at it and the models will just keep getting better, and the best models will capture almost all of the value in the market. And therefore, if there's a lot of money in the market, then you just always want to have just about the best model. But it makes for a very interesting kind of competitive landscape because they're all fighting so fiercely just for like five, 10% improvements on each other. Uh, and there's so much investment needed just to like creep forward against one another, which is super interesting. And then you obviously have like the deep seek uh guys from from China who didn't have unlimited money. They had like, I think, a lot more money than than it than they made out in terms of behind them, but they had to operate under some constraints. And so, because of those constraints, they figured out ways that they could achieve more performance with less cost, right? And so I think some of the companies in the West maybe have been like over-indexing on just throwing more and more and more compute at it. We probably need a balance of actually trying to get more out of like a certain level of investment whilst also continuing to ramp up investment over time. So, yeah, we should, it should be really interesting this year. I think we'll start to see more and more like efficient investment happening this year rather than just the like throw money at it at all costs, which seems to be the trend of the last like six months or so.

Robby Peralta 27:43

And when you say like throwing compute at it, that means it just means making NVIDIA stock price go up because they just have to buy those expensive. Exactly.

Harry Wetherald 27:49

And they're like the ult, they're the ultimate winners of that, exactly.

Robby Peralta 27:52

Yeah, I'm not gonna lie. I was scared shitless when all that happened. I was just like, oh fuck, my tech all of my pensions and technology stocks and everything's tanking. And I'm like, fuck.

Harry Wetherald 28:00

Yeah. No, they're they're straight back to value everything.

Robby Peralta 28:03

Uh there was one other thing in your in your post, the valuation of open AI, and how their their goal is not the Chat GPT revenues, it's the connection that it's going to have with the underlying software, like that argument.

Harry Wetherald 28:18

Oh, yeah. Yeah, they like a lot of these model providers are getting valued at this kind of obviously insane multiple on their current revenues. My guess would be I I don't know so much how everyone's kind of valuing this kind of thing, but my guess would be that the investors are basically writing those investments not on the basis of some kind of multiple on today's revenue, but on the basis of uh one of these companies or or a few of these companies being able to own a lot of the economic value that's derived from AI in total. Right. So if you think if we if we play by the assumptions a lot of people are that AI is going to basically automate a huge amount of work that currently is done by humans, or automate a lot of work that could be done by humans but isn't, right, at the moment because it's too expensive or too slow or whatever, then that should be a huge amount of value in total. My guess is that a lot of investors investors are basically trying to underwrite investments in these companies on the assumption that they can be the underlying provider to um to a lot of that kind of work and therefore capture, you know, huge, huge, huge amounts of revenue in the future, even if their revenue delay is relatively smaller. So yeah, I don't know, I don't know exactly, but that would be even be my guess.

Robby Peralta 29:19

Yeah. So if I try to unpack that, that like let's say Maze, uh when what Maze just fucking makes it and is worth you know a bunch of money, hopefully are the investors that have invested in open AI are kind of hoping that you and as Maze are using one of open AI models. And that's why Exactly. Right? Okay, interesting. Yeah, yeah, exactly.

Harry Wetherald 29:36

Exactly. It's a bit like to be honest, it's kind of analogous to like why Amazon, Google, Microsoft are worth so much today, right? They've become the world's computer, basically. Right. So they've become, you know, any any almost any kind of startup that gets started today, they'll go to AWS, lots of enterprises, they'll run a lot of their um computer on Azure, right? So these three companies really have become like the world's computer. Um, and as a result, they've become these giant, giant companies because obviously there's so much value in in doing that. And I think this the same thing is roughly gonna end up happening with these AI model providers, which is the bet is they can become the world's kind of intelligence engines, uh, if you will, and play a similar role in the kind of like landscape to something something like AWS does today. The like bear case for it, which which some people would say is that open source models are basically just gonna come in and commoditize like everything they're doing, and that there is gonna be no, you know, like uh like value in something like Open AI. I don't know what's gonna happen, honestly. We kind of don't care ultimately, as long as all these people just keep throwing money at making the models better, like we're happy, other people other companies like us are happy, and we'll just keep keep building on top of them. And you know, if they end up being open source, great. If they end up not being, you know, we we don't care too much.

Robby Peralta 30:50

Interesting that you say that because I'm just thinking in my I'm a sales guy, right? In my sales guy's mind, if you're building your product on an open source model, then that means your product or either your product is cheaper or you have more margin that's coming into your pocket. Whereas if the model that you're using is open AI or perplexity or one of these that is not for free, then you got to pay somebody. So doesn't that have like a piece? Doesn't that have something to say for you?

Harry Wetherald 31:14

Not necessarily. Like again, the analogy there is kind of like um in theory, you can run like if we built our own data center, right? In theory, we could make that cheaper than running all of our computer on AWS, for example. Um but there's a ton of work involved in doing that, right? So there's a lot of operational cost. We then have to focus on improving things ourselves rather than just letting the like cloud provider improve things for us, right? So whilst theoretically it might have been it might be cheaper, we might it might end up costing us more in the long run, basically. And there's like a very similar analogy here, which is um, you know, theoretically, maybe you could get you could save money by doing things with open source models, but maybe it operationally ends up costing you a lot more because you've got to host them, you've got to do all sorts of different things atop of them. So for now, it's kind of like a bit of a neck and neck um between the two. The one thing I do think see happening, and like one thing we'll definitely invest in in time, is people taking uh smaller open source models and then fine-tuning them for very specific use cases. And that is a very effective way to make things cheaper. So you basically go from having to have this huge model that knows everything about the world to a much smaller model that's much less intelligent, but you get to train it really, really specifically. Right. It's like um it's like taking an intern that's not very smart and training them for like 10 years on how to be an accountant, right? Um, and then they're a very, very good accountant, they're very bad at everything else. There's a there's an analogy that's gonna happen here, which a lot of companies are already working on, which is basically like take an open source model that's relatively cheap to run, but is relatively kind of dumb, really, really customize it to a specific use case, and then you have a cheaper model that's very good at your use case. I think that's where a lot of the world will go. It's just tricky to go in that direction at this point in time because the big models are getting better so quickly. We could go and spend six months like tuning our own small model to be really good at a use case, and then the next big model could come out and it could just completely wipe out everything we just did. Um, so there's like, you know, I think things will settle down in in a few years' time. Like, uh, and you'll and you'll see all this kind of stuff starting to sell out, and people can focus on cost more. For now, I think most people are just focusing on like trying to get as much performance as they can. And you know, one day in the future, they'll they'll start to kind of like figure out their plan for open source versus closed source.

Robby Peralta 33:18

Oh, you must have a you must be stressed. You have some okay should I which horse should I ride? And where should it, wherever we use the time on?

Harry Wetherald 33:25

We yeah, we I mean, as I said, it's it's like the best, it's the most amazing time to be building a new technology company. I there's just there is just as I said, there is almost endless money getting invested into the building blocks that we can use to build a build a product. And those building blocks are so fun to work with, so interesting to work with. They do genuinely like you know, and everyone's seen how Chow GPT can work, but if you spend a bunch of time with them trying to get them really, really good at a specific use case, they can just do like mind-blowing things. So it's such a fun time to build a company. We're just super happy with how much money is getting invested in the underlying models and just having fun, having fun using them, basically. Awesome.

Robby Peralta 34:00

So, last question for you when it comes to your product yourself. So I guess the model and all that discussion, that's a lot of hard work, and you know, we just use 37 minutes talking about that. But I guess the front end is also just as much as uh focus area, or how what do you think about that?

Harry Wetherald 34:16

Yeah, yeah, yeah. Great question. So, my honest take is that like uh the biggest problem that exists around AI products at the moment is user experience. Like the the normal, like the standard paradigms of user experience around AI products in almost every industry is not what I think it will be in a few years' time once people figure out how to build the right user experiences. And I don't know exactly what they should be across lots of these things. But the general thing is like, hey, here's an open text box. Type whatever you want, and we'll try and do something for you. Right. That is, on the one hand, this really, really powerful thing, right? Especially with the large language model behind it, because it can do like thousands of different things for you. But from a UX perspective, it's actually it actually breaks a lot of like common UX kind of uh pieces of advice or like you know, rules because it's so open-ended. It has like in UX pilots, it doesn't have like it doesn't have good affordances, right? It doesn't actually tell you what you can do with it. And so you end up with these just empty boxes where you're like, cool, I guess I'll try typing something in and see what happens. And then you have to basically become like an expert on typing things into this big empty text box until you figure out how to use a tool. So I think a lot of the right UX experiences, they might use that as like a bit like you use like an advanced control panel today, right? It's like the thing that 20% of users use a lot, or it's the thing you use when you can't get the normal product working, right? You go and like figure out where the advanced settings are. I think the way that a lot of the best products will work in the in the future is they won't actually rely on that like free text entry kind of thing as much, but they'll uh build a UX kind of flows that allow people to get so much power out of a large language models, but they show the way that they show you what you can do with it basically. So that's where I think a lot of stuff will be going. It's a lot of the ideas we're trying to like play with, which is asking the user to do like as little work as possible and us to do as much work as possible and make it as clear as possible what we're doing. We're super lucky that one of my co-founders is uh is like a 15-year experience designer, uh, which I think is pretty rare in security. I've not seen many security like founding teams have a have a designer designer on them. You can kind of tell when you look at security products, sadly, like in uh for the most part. Um, so yeah, we're having fun figuring all that stuff out. We definitely won't get it all right the first time around, no way. Um, but um, it's like a super exciting area over the next few years in like all sorts of different fields. I hope we're gonna see a bunch of new ways of interacting with products that actually help people get the most out of out of what AI can do. And I think as a result, you'll start to see people trusting AI a lot more because like, oh wow, that was actually useful. Useful for me in like one click rather than useful for me once I spent like two hours figuring out how to use the thing, which obviously is not great.

Robby Peralta 36:46

Yeah, I was at RSA last year and uh I just randomly out of lunch sat next to this guy, the owner of this company called Sodium Halogen. And long story short, all they do is UX for security companies. And you know, it's like, wow, are you guys really you're that many people doing UX for security companies? And then yeah, he was just but you know talking about it, and I was like, oh shit, that's actually really important. And now that I'm fast forward to today, I look at you know, perplexity and and deep research and just the way they like bolt on they already you ask a question and then they ask all or they know what you are all the other things that you're wondering about, and that's you are that's UX, and they just understand it read your mind basically. So yeah, you get um a lot more value out of it or perceived value, and then you don't want to switch over. I know Chat GBG has the same thing, but uh I just want to use that one now because I like it, it looks cool.

Harry Wetherald 37:30

Yeah, exactly. Perplexic, great example, like they they are actually trying to innovate around the UX. Um, and it's made them a super successful company. It's not because they have unbelievably smart models under the hood, although they increasingly do, like it's just more about the fact they are just coming up with easier ways for you to access these models. And you're seeing it in like so many industries. A cursor, another good example of that in in engineering. Like a lot of that is just basically a really easy UX to interact with LLMs to help you code, right? And UX is one in that world as well. So I think we'll see this happen again and again and again, where basically, like, weirdly enough, in this like AI era, actually UX will be the thing that that helps a lot of the products win. Um, which is cool.

Robby Peralta 38:06

Cool. Harry and Maze, I'm rooting for you guys.

Harry Wetherald 38:11

Thanks. Thanks. Yeah. Well, as I said, we'll start sharing with the world a little bit, a little bit soon, more about what we're doing. Uh, but yeah, until then, yeah, thanks so much for you. It's been really fun.

Robby Peralta 38:20

Thank you. I will reach out when I uh when I touch down in London. I owe you uh a pint, whatever that means. Sounds good. All right, man. Thanks, ciao. Well, that's all for today, folks. Thank you for tuning in to the mnemonic security podcast. If you have any concepts or ideas that you'd like us to discuss on future episodes, please feel free to hit me up on LinkedIn or to send us a mail to podcast at mnemonic.no. Thank you for listening. We'll see you next time.