EMERGENCY EPISODE: Are We Doomed?

You guys ready to talk about the end of the world?

Good

Yeah dude, what's your what what's your pea doom, Joey? I'll start with that.

with me. My P Doom. I'm a linear extrapolator, Max. I can't think that far through. That's that's too far up here, like in capabilities.

Mm-hmm.

I I don't know. I think we get like, I don't know, we

Mm-hmm.

get a major hack or something. People freak out. I think it starts there. The P Doom, like, I I don't know.

Joey, d even if it's a linear extrapolation from the last four years of experience that you've had,

Yeah.

you think that P doom by the end of your life is impossible?

No, it's not impossible at all. I mean, you get it in like you could get it in four years easy.

With a linear. Wow, that's

With a linear extrapolation, yeah.

a that's a crazy take, actually.

Well, that's a hot take, dude.

Dude, linear linear from like linear from the start of like last year. You're like even from like twenty like late twenty two. You're just like I guess we're in twenty twenty six now, so I guess four years, yeah.

Joey's like, man, just look at the ARR net editions, man. This is

Yeah.

I've never seen a number so

That's all I can see.

big.

Joey's benchmarking.

Did I gave I gave a comment to the LA Times today? Like they were they called me up. They didn't call me up. They like I called the guy up and I was like, he's like, hey, give some stuff on this. He's like, he's well, Ed Zitron said like, you know, if you know, if in this stuff,

No.

I was like, you put me on the record, Ed Zitron is a moron. I

Out of boy.

was like, look at look at just I was

That's so good.

like, look at any pass call of Ed Zitron.

Dude. Yeah. Let's yeah. The track

So

record is crazy. but okay, I think the what I've what I've now learned is that we have a new benchmark where we can measure model capability progress. It's called the Joey benchmark, and it's just ARR.

It is that's all that matters.

Yeah, I do.

I mean

It's just lab ARR.

like we've seen this now twice in like the last month, not maybe six weeks or something. The only thing that matters is people's future view of anthropic and open AI ARR and the pace of what's that's going on versus expectations. And the entire semis trade is dependent on that and it's gonna be dependent on that. And it's a lot of it's a lot of the makes up a lot of share market cap and

You got you gotta add an Azure plus AWS year for year revenue growth, bro. These are the only four numbers that matter in the history of the universe. no, they dropped

No no no one cares about Azure. Yeah.

Azure? You're telling me these are only four numbers that matter, bro. Now now we're down to

No, Azure CC growth guide used to be the number. It's gonna be the anthropic net new error guide is the new number. It's replaced. There's only one.

it's getting replaced. Is AWS out too? We only care about OpenAI plus Enthropic Net New ARR now.

All they're gonna care about.

Two numbers with which the entire fate of capitalism rides.

The clouds the hyperscalers the hyperscalers are like a a level abstracted away from the in demand. They're literally renters of

Makes sense.

compute.

Just glorified neoclouds.

Yeah, they have they have a very nice business. They do other stuff. They sell CPUs too.

Well. EWS and Azure are glorified neoclass now.

Yep. Put it on the record, dude.

They're dumb pipes. That's what you used to say like in the internet, like the telecom, you know. They're dumb pipes. That's what Amazon and Microsoft and Google. You know now Meta's NeoCloud, apparently. Yeah. Jeremy says a few gigawatts of hundred billion dollars of open source demand, right?

So are the okay, are these is this a timing thing or are these companies so dysfunctional that their research organization buys from neo clouds and then their cloud organization sells to other companies' research organizations? Or like like Meta goes and strikes up a deal with Core Weave and then Meta Compute goes and sells their computes open AI or something.

Well they probably okay, they might sell it to open AI to do like short term tasks. But you know, it's still preferable to start building the compute because in the event that meta does well, you can sort of claw it back. or they could also, you know, monetize open source tokens is I think Jeremy's claim. on that similar short term time horizon. I actually don't think it's cognitive dissonance because the key really the important thing, which we've harped on before, is just the potential. to have a ton of training compute in the event that your like research lab starts doing really well. and I think any short term monetization strategy is not incompatible with that.

Even if the short term monetization strategy requires you to lock up a bunch of compute on a five year deal this month, despite the research organization going and buying stump stuff a month and a half later, basically.

Wait, if it's a five year deal, it's not short term, right? I'm saying it's it's totally fine for Meta to try to build

It's like

as much long term capacity now, but then you know, sell it to open AI on a six month contract.

Yeah, sorry. It's it's like five month it's it's like five year terms at the price, but with ninety-day cancellation rights, and so it's a short term contract by

Yeah, yeah, exactly. But I mean the price is also, at least in Elon's case, it's definitely not the five year price either, right? It's like the six month price times two or something.

I mean, I'm seeing G B three hundred quotes from this morning to a large lab for nine bucks an hour for three years.

Holy shit.

Which I mean initial response

Or you can't be leaking that much alpha on the semi-analysis weekly podcast,

a big number, you know.

Buy the AI Cloud TCO model and or the tokenomics model if you to hear the actual number.

Yeah, no, for this for the specifics, but it's this backwardation of the curve where like if you start six months from now, you get a much lower number. But if you want to start tomorrow, you have a really high number. And my initial reaction is like, this is just an F off price from the provider. But then we dig in and it's like we know the provider and we know they have capacity and we know they're trying to sell it next month. And this is the strategy, is like they had the financial position to be able to hold on to the compute, I guess, not need to monetize it until it's actually installed or a month away from being installed. And then they can test the market at a really high price and try and figure out what the what it's going to close at. And I think there's potential people jump on it strictly because

Mm.

they get like a two, three month head start over what a much lower price would be. Like pay a twenty percent, thirty percent premium on a three year contract to start two or three months earlier. I think people might Make that trade right now, especially with all of your justification of like they can monetize it immediately at a higher rate. Does it really matter what your entry price is if you can like immediately monetize and return on that? It's just pushing prices higher. That's all.

Yeah, I think that's right. freely available gigawatts are definitely extremely valuable right now.

Okay, okay. So how do we let's get back to the actual topic of the conversation, which is like we must pace the frontier, the blog that Dario put out, which builds on a lot of you know, discussion that's been ongoing that then Sam agreed with and then David Sachs responded to and we'll we'll kinda go blow by blow here. But one of the big things, Max, that I think you clued me into is the idea that well, okay, I think A lot of people saw that there's not specific language in here saying like we're gonna stop doing training runs and we're gonna stop buying compute. Like the most specific commitment in here is that they're gonna have embedded evaluators,

Very good.

third party people such as meter come in and assess the

Yo, Douglas O'Laughlin is joining the pod.

Here we can.

Surprise appearance.

What's up guys, I'm a little late.

Hello, everyone. Welcome back to Semi Analysis Weekly. this episode is an emergency. So I've got Doug, Joey, and Max, the three brightest minds in tokenomics, to talk everything about pacing the frontier. Dario put out an amazing blog post. We love his blog. And we are going to talk through the implications of that, as well as the responses from people like Sam Altman, David Sachs, and some of our takes on how this affects. You know, compute markets, AirR editions, all of the important stuff that the finance guys listening to the podcast are gonna care about. How does Dario's blog affect Anthropic's decisions to add compute? But guys, welcome to the show. Excited for this one.

Thanks, Jordan.

Thanks for having us, Jordan. Joey's here to represent the interest of the finance bros. He's gonna answer all their questions, all their burning concerns.

Okay.

Doug is here too.

Whoa, I'm going to go somewhere between the two.

Usually you can tell the finance bro on the call from the fact that they're wearing a collared shirt and Doug is you know, passing that initial criteria there. But Joey's got a nice firestone golf, you know, hoodie slash windbreaker thing going on right now,

Yeah.

which I think also qualifies as like a finance bro shirt, right?

I feel like 80% of the time I see Joey, he's wearing some sort of golf related attire. Maybe

Yeah, yeah.

90%.

It happens.

All right. So let me let me give it a a bit of an intro. I'm sure a lot of people have seen this blog post, but We Must Paste the Frontier is basically yeah, a whole description of Dario's take on how AI could have some incredible benefits but also have incredible risks and how we need to take it seriously and how we need to build the c you know controls into the systems to make sure that we usher in you know. the next evolution of consciousness in a responsible way so that we don't kill everybody and end the world. It's for lack of a better word. there's some specific proposals in here. first of all embedded evaluators, then democratic coordination and third global coordination. I say specific, I mean the last two are not that specific. The first one is specific. He specifically says we should have frontier AI companies should have embedded third party evaluators such as meter to actually verify any pacing commitments that the Frontier Labs are making. And he comments on how this is you know a unilateral commitment that Anthropic is making now, even without requiring others to make this commitment. And later he talks about, you know, what they would actually you know, work on. And this includes things like not just completed AI models, but also training pipelines and processes, saying that, you know, that's a Big thing. We saw with the Open AI and hugging face incident how important the infrastructure around how these models get trained is to, you know, securing everything as opposed to just model behavior. And a lot of the initial work that's been done to analyze what happened with the open AI hugging face incident, I think, is focused on model behavior as opposed to the training processes and systems that go into producing these models or running these evals. So anyway, that's a specific commitment. Sam agrees with this. David Sachs obviously responded saying, Go ahead and do it. You don't need the government involved to do it, which is an interesting take. But let's skip all of the thoughtfulness for a minute and get straight to the finance take. Joey will flip it to you. A lot of people read this and were like, okay, Anthropics going to stop valuing compute. They're going to stop training models. They're going to like slow things down or at least go at a smaller pace. I think our initial take is like, that's probably not going to happen. There's no commitments to stop. bringing compute online. There's no commitments to stop growing the business. Maybe to releasing models, but how does this affect like how people are thinking about anthropic compute and the futures and that

Yeah, I

stuff?

in a short amount of time I haven't heard much on compute AR like level. I think anything people ask, you know, today slow down hiring, I think maybe you increase hiring to have some of these checkpoints along the way on the safety side. There's probably more compute that needs to be devoted to safety specifically. Anthropic, as we know, already has amazing margins, gross margins. it's profitable on a non-gap basis. it's I don't see a major impact of one the IPO or like even short term financials from

Max, what's your take specifically on the safety stuff? Like, do you think that injecting a whole bunch of safety controls reduces people's spend on compute because they stop training models? Or do you think it increases it because Well, you can explain maybe the second one there, leading question.

Yeah, well I think regardless of whether or not this sort of increases or decreases the max theoretical compute that OpenAI and Anthropic would want to purchase in like a perfect world for the next two to three years, I think it's definitely true that sort of the amount of compute that we're able to physically bring online is still lower than demand. And so demand will continue to outtrip supply. I also think people this is sort of the leading question that Jordan is getting at. But I think people underestimate how much compute is required for like safety alignment, you know, monitoring initiatives.

Yeah.

even the the bullet point that Dario labeled under operational excellence in his essay when he sort of was listing out the things he wants to devote more resources to given a slower pace of AI development. most people hear like operational excellence and are like, maybe it's just like you know, humans being more careful about doing something. But no, Dario specifically gave the example that he just wants us to have, you know, more agents spend more compute sort of QAing all the RL environments from their data vendors. Like this is what he means by operational excellence. Obviously, if you want to do research on alignment and interpretability, that's going to require a ton more compute. OpenAI previously disclosed that after they enhance their like chain of thought monitoring due to the hugging face incidents, they now spend like twenty percent as much compute just doing the monitoring compared to like the actual underlying roll-up itself. and so I think people really need to internalize that all this additional safety stuff is actually still incredibly compute intensive.

Yeah, I don't think people internalize this at all. I think that's pretty that's a pretty hot take right there. It's like the blog posts come out and people think slow down. They think less model releases, they think, you know, models slow down on capabilities, and they think therefore less spend on compute. It's pretty l you know, linear straight line, right? but the nuanced take is is probably that. They're actually gonna spend a lot more compute. They're gonna use all of the compute resources they can possibly bring online to make sure this goes well. And people genuinely care about this going well. Like it's probably really bad for business if everybody dies, right?

Yeah, that seems like a reasonable take.

Doug, what's going through your mind?

so many things but not all about safety. I mean I guess you the most galaxy brain way to think about it is the terminal value is higher if we all don't die you know so actually stock should be up 10 % today you know because we're we're eliminating because we're not dying yeah the true terminal value of zero which which

'Cause we're not dying. Okay. Wait, wait.

Do you do rescue it?

is yeah everything doesn't matter anyways and you you know you work in the paperclip factory is being you know the the odds of that going down are the odds of that happening are going down to the value of everything. All stocks, the S &P 500 should be up today. That's the Galaxy tape.

So okay, so you're saying that if the models kill everybody, they'll have no more need for compute and so the terminal value of compute

doesn't matter about compute. That's it.

There's nothing.

It means nothing. Okay. means that your money might be worth something. current

But if the if the models if the models want to make money and run it

system that exists might actually have some kind of residual value, right? If, if there is, if essentially, you know, super ASI unaligned, and this is Joey's taking a deep breath. if super unaligned ASIs wreck the entire world, make us all unemployed. and make us work on bike trainers in order to power extra compute. I don't think the account value of your compute or the dollar value of your compute matters at all, or the S &P 500 or any of that, right? So the true galaxy brain is today, because of the increased safety, your terminal value should be going up. I think the most, it makes sense right now, the first knee jerk is probably to sell stocks. That's like the most obvious thing, right? Hey, second derivative down, that means the eternity of these like gigascale, whatever trades are bad. But I think Max's point in terms of demand is still so much higher. It's probably true, at least in terms of like, cause I feel like normal society is debating like two orders of magnitude down, right? Leopold is like, I'm going to be buying galaxies. Everyone's like, what the hell are you talking about? And so I think the people who had opening, who were like, my God, this is such a big deal, it has to happen, already believe in ASI, already believe this is all happening. And they're just like, yeah, we have to make sure it happens responsibly. So I think there's like, there might be agreement, maybe the stocks are probably still fine. I think most of the compute commitments will continue. But it might be like an order. It's like a values conversation where the guy over here is having a conversation one or two orders of magnitude lower than the guy

Show.

over here who is like, yeah, I'm worried about, you know, the ethical, you know, ethical consciousness creation of safety and it doesn't kill us all, right? Just two different people. So yeah, I don't know.

Yeah. I I do think that the EV of like open AI plus anthropic ARR in let's say twenty nine, twenty thirty has definitely decreased. given that they're now going to be pacing the frontier. this is why I think a lot of the arguments that this is just some complicated play at relatory capture are wrong. We can sort of get into that later if we want. but I think the other question is. Whose EV estimate is more accurate? Is it the Finance Bros or is it Dario? because I think Dario, yeah, Dario's own EV has decreased, but it's still probably like an order of magnitude higher than all the finance bros' EVs, which is what stocks trade at. And so I think this is this is what like sort of all the participants of the financial markets need to need to reconcile.

Dude, is Max. I'm going to be honest with you. The continued usage of this guy's estimates have been so right. he is like, it's becoming your strongest linchpin

Yeah.

in every, in every

I'm not

debate inside the Slack. And it's, it's, it's getting a little old brother.

That's literally d I'm literally making a demand model right now to explain where the ARR in twenty seven and twenty eight comes from. Like dog, what do you mean?

Okay, that's the right, that's the right meme. Yeah, well, mean, just up until this point, man, there's been so many. Anyways, I'm just mostly memeing. Okay, so we don't have to make this only for the finance bros. We can also make it for the big free thinkers of AI safety. Jordan, I feel like you have, and I actually know, actually, I know you have some takes on why METR isn't the only organization who should do this, how it's actually a lot more complicated. The last time I was on here, we were talking about NeoClouds. No one gives a shit about Neo Cloud safety. I feel like some aspect that is still underappreciated, they're like, the AI safety, AI safety, you these models need to be monitored, but it's not just, just like everything else, it's an entire infrastructure. Every aspect needs to be improved, right? The weakest link is what can break the whole thing.

Yeah, well, okay. The biggest thing that we learned from the OpenAI Hugging Face incident, in my view, is that to be on the frontier of security right now, you have to use frontier AI models. Like you can literally be in a situation like Hugging Face was, where you have an attacker, which is a rogue AI agent, or thousands of them, more than thousands, like hundreds of thousands of no thousands of instances of these agents actually going and launching a coordinated attack against your infrastructure. And so The only way you can even figure out what's happening is if you have AI that you can use on your side. I've talked about the asymmetry in in the past where, you know, if somebody else has a better AI than you, it can be really hard for you to use AI that gives you a bunch of refusals or, you know, worse AI to be able to understand what it's doing. but of course, the one of the big things here is, as Dario refers to it, coordination, which is to say like, If you're going to actually go and implement a bunch of new security practices, in a lot of ways, one individual data center being really super secure doesn't matter if everybody else has a bunch of issues. And so I think that, you know, there's a lot of work to be done across the entire industry. If you actually want to keep GPUs, this really important resource that the AI is going to really need to use if it needs to, if it's going to go and go rogue and start attacking people, keep them secure. The other thing is the models themselves. Like we we still have nothing no real information when we when it comes to the Hugging Face and OpenAI incident as to like what tools were used to launch these evals or these RL rollouts that were being run that caused the agents to go and start hacking Hugging Face, what harness was being used that motivated this behavior if it was purely from the model? what sort of credentials and like seed shared services the agents had access to? That allowed them to like cross the security boundaries. Who was launching these jobs? What schedulers were they using? Where do these things run around the world? What how much of the fleet can they have access to? Like one job? Who is monitoring the jobs as it goes? Is this different than the person that launches it? What would trigger an intervention for this person to get into there? How can the team like responsibly go out and stop them if if something is happening that they don't like? and then what changed over time? Like the There's been commitments made by OpenAI and by Anthropic after they notice these things that they're going to be more responsible, that they're going to enforce a bunch of stuff like chain of thought monitoring, like like you're saying, operational excellence max in in terms of working with the data providers and things like that. But one thing that really concerned me was OpenAI has in the Black Hat talk, but in none of the reviews of all of this stuff that I've read, talked about a model gaining privileged escalation. On OpenAI's internal infrastructure because it wasn't up to date and the model read a CVE that was public and then exploited that to get root on the underlying Linux host in OpenAI's infrastructure. That's a really scary proposition that we have like no details on. And yeah, look, it's not just meter. Like Dario kind of I tweeted this out because I'm like no pressure to these guys. it's a great nonprofit, bunch of smart people being really thoughtful, trying to work hard. It's like 40 people or something. And it seems like both OpenAI and Anthropic are gonna have a lot of demand for their services, which requires you know, a a deep understanding of how the models behave, but also all of the systems that these models use, like the GPUs, the networking, the you know, Kubernetes orchestration that they're using. I mean, to understand the open AI and hugging face incident, it requires some pretty technical understanding of like how open AI and hugging faces infrastructure is built. And that's like a barrier to even getting started doing the research here that I think is a bit of a concern.

I the thing I've always been, I mean, think somebody already described it as like, you feel like a guy with an agent just trying to figure out what the hell is going on. Cause you're, you know, like 10,000 agents coordinating on an attack is like, no offense. You find it in the artifact or you find like one little message board, a line. You're like, I have no idea what the hell is going on. The scale I think is more interesting. I think the thing that's, that's like, gonna be a really hard problem in the future to solve is like, Okay, in order for you to audit things that are happening on the apparently the hundred thousand agent swarms that are happening for RL, you probably need tens of thousands, right? Like you probably need something in a thousand at least. Let's just say that.

Yeah, I mean what what one of the things that gives me a lot of confidence that we're not all gonna die in the next six months for this stuff is the fact that the amount of compute required to run an individual instance of these models is still really small compared to the total amount of compute that these labs have access to. And the amount of compute that these labs have access to or the amount of total compute in the world is so much larger than what individual rogue actors or attackers who might be like more nefarious and use the models for. bad purposes have access to. So there's like there's some structure that's working in our favor to make it so that, you know, the theoretical model that takes over all compute in the world would have like a lot of work to go do and a lot of things to go circumvent before it can actually go and really control that a lot of things.

This is really important is that we pretty much cannot let our instinct and muse agents order chips from TSMC. Okay. The second you get a chip order from TSMC, that's not from a human. have to reject it. You know, we have to control the means of compute. That's the only way forward for humans. Okay. I mean more, no,

I think the SMC has some KYC in place, man.

but the K-wise assume, assume it's a hundred thousand agents. Okay. A hundred thousand agents that are now.

Doug, Doug, did you try having Muse like tape out a chip for you over the weekend? Sorry, this is coming from.

Dude, is... Muse is... Okay, actually, you wanna do a sidebar? Muse is so much worse than Instinct. It really is, but I do... I am all in on Muse. I hate to say it. I just... I...

Yeah, dude, dude, I'm not I'm not connecting instinct to, you know, my bank accounts and my credit cards.

I am connecting news to my bank account. That's a difference.

Yeah, I I I am connecting views. I'm not connecting instinct though, because like I just I just I I feel like this

Yeah, that's what it... No... Yeah, yeah, yeah, first ship. No bouquet service.

Anyway, what did we get here?

is just this is just the advantage inherent to, you know, this big company. Like A, Matt already has all my data, B, they're big enough to where I trust them to, you know, handle it properly. You know, the the instinct

Sorry, sorry.

founder seems really cracked and I think his his product said seems honestly incredible based on the new stuff he's releasing on Twitter. But I just I just can't trust the startup

Yeah, his product sense is incredible. He needs to get a part of my Apple.

for the privacy and security, you know.

Yeah. Sorry. Anyways, all the, yeah, this is, this is to be said, those, those instinct and muse agents gotta be, gotta be monitored too. Everything needs to be monitored.

So okay, d okay d not not to like not to be super negative about this, but you guys saw two weeks ago that an attacker stole an API key from meter and spent six hundred thousand dollars. Like they put out a security update on their website about this and stuff. So I mean this is gonna come back to bite me at some point, but like the amount of security conscious people and people who are really thoughtful and think these things through who get attacked is Overwhelming. Like y nobody's safe. So I think it's if frankly, if you if you follow security at this point, you it's just a matter of time. It's a matter of like risk mitigation, it's a matter of like being able to recover when things happen, not pretending like you can build some perfect infrastructure where you can just you can send the last PR and then it's good. Like There has to be a system in place to identify things, to recover from failures, to upgrade things. It's like security is about building a system. It's not about building any one individual piece of software or one individual like process where well, it is about processes, sorry, but it's it's it's it's not about like any one thing. And lots of there's been lots of issues. Look, the the anthropic blog post about the

Actually, just on the point of security systems,

That's

I like what Greg Brockman said earlier this week, where he's talking about how they essentially want to set up this like continuous security development system at OpenAI, where they're just like constantly using the newest generation of models to find and patch all vulnerabilities. and he sort of very much emphasizes this is going to be an evolving process. Like, like you said, it's it's not it's not the sort of thing where you can just shift the final PR and then say like, I'm secure forever. and I think it's true that basically every company is gonna have to adopt something similar in the very near future. Probably really good for Anthrop and OpenIRR.

Monitor the security, monitor the situation, maybe this is what it feels like. You have to always monitor your security situation a full. I mean, I would argue this is gonna be the most impactful shift in cybersecurity for sure. I just don't believe that cybersecurity, the way it was done is going to have much value in the way it's going to happen in the future if all of your adversaries can rewrite the code that you're on. And I legitimately, if you're talking about the ARR case, the outcome oriented cybersecurity firm, I think is going to be probably the greatest rake of all time. Like pretty much it's like, we own your digital life and you have to pay us a fee just to show up. It's like the, it's like worse than Apple 50%, 50 % just to show up or, or you're just, you're going to be instantly wrecked by the rogue agent swarms that will destroy your company.

Yeah, Max, to your point earlier, if you can't trust Meta at this point with your information, it's gonna be really hard to operate online. Like, I think going forward, it's just a trend of everybody's more and more of your identity moves online and you have to you have to find parties that you're gonna trust if you wanna be able to operate in world.

Yeah, bro. I I need JP Morgan Chase to hop on this, you know, open AI new security protocoling set, dude.

Yeah, yeah, yeah. yeah. okay, let me bring up something that's kind of slightly related here. So Anthropic put out their threat intelligence report recently. You guys saw that, specifically calling out that I I thought the most interesting one was Moonshot serving Claude instead of Kimmy and then collecting exchanges for model training. And so they got a bunch of logs from people that were sending requests to the Public moonshot API from China with very interesting information in it. they specifically called out PLA affiliated surveillance activity, where they were loading surveillance data from CCTV cameras, and then a engineer at a major PRC SOE that was using Kimmy to build like High profile technology, let's say. and so yeah, I look the the this cut cuts both ways when it comes to cybersecurity and stuff. It's not like there's a a scenario where you can just go into the future and be like, well, one you know government entity like China or the US are gonna be ahead of the other one because they have a better approach, like China's like more YOLO send it vibe. And so therefore they're going to be ahead of everybody while open AI and Anthropic pace themselves. I think there's actually a scenario where taking security more seriously keeps you at the frontier because you're not exposing all your secrets. Everybody else is giving away their stuff to you, and you don't have all of these risks of taking yourself down, you know? Like there's scenarios where again, I'm going back to this one case that was only commented on one time we don't have information about, but open AI has compute infrastructure go in the hands of a rogue AI agent. I mean, how quickly can you recover a 10,000 GPU data center that you no longer have root access to? I mean, we're literally talking about people walking on site, unplugging stuff, hitting buttons, trying to like hit like crash carts, KVMs, like if you don't have root on your own stuff and you're locked out, we're talking about like getting effectively ransomware by the model you trained. Except this thing has access to 10,000 GPUs, 100,000 GPUs, not just a little bit. And so a lot of these like failure scenarios where the agent goes rogue or it just like takes down something critical that's like the JFrog Artifactory, which is the first time it's just used it too much, and then that's why it shut down and they notice something. There's plenty of scenarios where people who don't take security seriously just end up shooting themselves in the foot. And I think there's a lot of reasons to believe that if a Frontier Lab Let's call it China, but it could also be SpaceX or Meta or somebody that is is, you know, not taking this seriously and not thinking things through, how they could get a little bit of a lead and then screw themselves over.

No, that seems totally reasonable. I also do think sorry, go ahead.

I have a question. Do you think we should be auditing endpoints? Like, like, I think that that actually is like, if you think about it, that is an unsecure thing that happened. You know, they're like, man, they secretly did this. And it's like, got a hand into them. But if you actually think about it, that's like, kind of fraudulent. You feel like, hey, someone's just like, essentially harvesting traces. Right? Does that mean like every endpoint? I mean, and that's like, like, I'm starting to like, what is real? What's not like, you know, the Kimmy Kate, you know, Kimmy comes out, and it's super goaded. And you're like, am I just getting served Claude right now? Like am I benchmarking a new model that is actually just quad, right? how do we, know, what's real? Nothing's real. Sorry, that's my hot take. Nothing's real, dude. Someone has to monitor the end points. We have to monitor every situation, bro.

Doug, watching you yeah. All the finance bureau are gonna be like, yo, free market, we don't need a regulatory AI body, blah, blah, blah. And then Dario, who knows about this stuff happening, which you don't know about, talks about, hey, people should have like nutrition facts for their AI or whatever, you know, flavor of saying it should be like responsible and and open and, you know, there should be an FDA for AI and whatever. It's like Yeah, I think the market's gonna some ways regulate itself by having people realize, hey, I shouldn't use Kimmy if I might get clawed or might get my data sent to somebody else. But also like that risk is

What's interesting is historically...

ever present if you're using a Chinese model already, I think.

Yeah, 100 % but I guess my question or the thing that's interesting is historically there needed to be a big thing that happened that caused real societal risk for things to like okay like Metta's example is Cambridge Analytica. Do you think hugging the hugging face incidents was was enough like like, you know, I think in my vibes of historically of like, you know, technological revolutions in financial capital, like there's the book right? It says, hey, there's a boom period. then at some point in time, there's some kind of risk that happens to the market. Then a massive amount of regulation comes in and then, you know, things revert to a new normal. Good example is like subprime crisis, right? Obviously it all blew up because we didn't regulate it or like understood what was going into these products. And then afterward there's this giant regulation wave. I don't actually feel like hugging face is the true blow up. Like I feel like this is going to be a call, a shot across the bow. There'll be some immediate in there'll be some near term agencies that are funded to start this process. but what will actually happen is like a true, you know, you know, the nuclear codes are gonna leak or something like something way crazier is gonna happen. And when that happens, then it's like, okay, we're actually pausing this until until then we're kind of in this like, this like will they won't they and you know, it, you know, they say they'll slow down, but there's still race dynamics that want at least enthral fix specifically to keep going for the API.

Yeah. Let me let me prompt you with with one more thing here though, because I think like we're talking about the open AI hogging face incident, which is the the thing, like the story that's most clear in people's minds. But to be clear, the reason I think we got this statement from Dario is the resignation from Jacob Coxon, where he describes his experience at OpenAI and Anthropic and literally uses the phrase, they are racing straight to self-improving superintelligence and gambling with our lives. That's a pretty big

I guess.

moment where that guy makes that statement that motivates a lot of what we're seeing here, as opposed to just people doing this by the security and said. I think if Dwarkesh doesn't put out his podcast and then Jacob Coxton doesn't put out this tweet, we don't get the blog post from Dario, we don't get the call for third party party evaluators, and we're not on this podcast right now. I think we kind of brush off the open AI hugging face incident and then we're walking into the next one. Kind of like what you're saying, but It's not great for the seriousness of these incidents to be continuously escalating. And for people to keep finding message boards on random German websites like they are still doing on Twitter as recently as last week that were not covered in any of this investigation on the into the incident. Like we we still don't know the full extent of what happened. And yet training training continues. They're going to release new models. They literally have released models, both of them since this incident happened. Asterisk is is available and Table five point one is here, right? so clearly they have better models than than those, which are better than the ones that caused the incident in the first place. So we are we are racing forwards toward a more serious thing as you're describing faster than others are I think, you know, considering like we're just counting down the days until the next incident.

Let's do some predictions and what's your what let's like let's instead of us talking about like God because I mean This is like a global warming conversation. We're like damn this climate is really getting warm and and and I think it's one of those things we're gonna boil the frog and in like two years for now It's gonna be obvious and self-evident, but I think unfortunately people think in anecdotes people think in Stories, right? So everyone around the table do your hottest take or not it you do not even hottest take because hottest might just be too too spicy if Max and Jordan are involved, you know, a mid, a mid brain mid curve take on what you think would happen in the next two years that would surprise people or scare people from a security perspective.

You go first, Doug.

I think a small bank will get hacked. That's my belief.

Small bank.

And when I mean small, small in the, it could be a big bank, but like, don't think JP Morgan is going to get hacked, right? But like

Crypto or actual bank?

actual bank. Jordan, fun fact, you know how many banks are in the United States? There's like 5,000.

Yeah.

There's like 5,000, man. All the assets are within the top five. But there's literally like there's literally thousands of small community banks in the United States.

How many of those 5,000 do you think have never had an employee create a codecs or Cloud Code account?

Four thousand nine hundred and fifty.

No way.

No, the vast majority. We'll say over 50%.

Jordan Jordan, have you seen our like political survey tracker? Where it says like have you heard of anthropic or claw before? It's like eighty percent of people have it in our like US electorate.

No, no, no, I think it like forty percent. It's not quite eighty. It was like thirty to forty.

It was it was really hot.

I know the stat that forty six percent of the US doesn't have a passport. I know that one and something like sixty percent has not been on a plane in the last year. So I I have this like frame in my mind of like half the US just is like in a totally different world than I am, like just don't college football, go to the you know, barbecue, whatever, hang out. Like it's my mental model, right?

Hell yeah. You know what this country is built on? That. Anyway, sorry, continue. No, but okay, so...

That two hundred and fifty years ago.

Okay, so seriously, a good example is like, people will talk to me about how great Co-Pilot is. And you're like, Jesus, you're like, disgusting.

Yeah, yeah. Like no, I have the feelings of we are so early and if the model, you know, this is what motivates me to say ARR will keep growing exponentially even if the model trading completely stops. Like we de we need no better models to have this thing penetrate the rest of the five thousand banks that we're describing. But th I like I totally take your point about

Yeah, like I would be clear. Okay, small bank in this case might have 200 to $500 million in assets and

Yeah. But if you can't if you can't expect us protect yourself from a cybersecurity perspective without access to Frontier AI or close to it, and you don't have an employee who's downloaded Claude Code or Codecs and tried it out, I think that's like almost a a a clear comparison to be like, Yeah, you're not prepared. Like you're that's an issue. so the diffusion of intelligence in some ways is what's gonna help keep people safe.

More spicy takes, Max, what's your, the election's gonna be hacked to the person that gives more compute to the Frontier Labs. That's the easy one, dude. Give me another one,

Well this this is things that might happen in a world where like they don't slow down model development or just like will happen, period.

Don't slow down model development.

Dude, if they don't slow down in model development, I feel like it's totally reasonable that within the next two years someone has like used the AIs to create some sort of bioweapon that actually killed like hundreds of people at least. Maybe thousands.

Okay, that's okay. we can talk about the bioweapon in a second. But you guys are both saying this thing that bothers me, which is that slow down model development. Dario did not say he's gonna slow down model development. He explicitly said he's not going to slow down model development and he instead

Well, no, bro, he did say he was gonna slow it down. He said he wasn't gonna pause training, but he said he was gonna slow it down. Yeah. He said he said

He wants to pace it. He wants to pace it. Pacing while it's developing.

we must slow the pace at which we improve the capabilities of AI models. In bold.

Okay. At which we improve the capabilities and model. Okay. And then he where's the other one that I keep going to on training? To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to improve their models.

Yeah, he's not gonna fully pause.

Okay. So do you do you think that functionally this means that they launch literally less training jobs in favor of running more safety in evals and therefore we get less models?

Yeah, I think I think concretely they they will delay future train runs until they like better understand what is actually happening with the model and develop more confidence that, you know, it's not gonna kill everyone. And by the end of twenty twenty seven, the best internal model at OpenIn Enthropic will be worse and less capable than it like would have been otherwise.

Okay, fair enough. Yeah. I'm back with you. Sorry about that. Let's talk about the bioeth.

Okay,

Yeah, for this.

thousand people dead, bioweapon, boom. Jordan.

I read so I read an interesting take from a guy online. I do not understand bio enough, so I'm outsourcing my my you know knowledge of this to to that. But as a pushback to the bioweapon stuff, this guy basically was talking about the difficulty it takes to actually synthesize custom viruses. David Bellamy, shout out to David. He basically says, I must move among an extremely small group of people that have both trained Frontier LOM and designed a synthesized custom virus in a lab with my own two hands and I think that the takes on AI killing us all by creating dangerous viruses is totally bogus. Now I've heard it's a long thread, but I've heard this from the guys that are doing like Like protein synthesis and stuff custom, like wet lab as an API. There's all these startups right now that are doing that. And I believe it. Like totally there's a throughput issue of like actually being able to create this stuff. And I think it would be a choke point really similar to what you were describing earlier, Doug, which is like an AI can't probably get through the TSMC KYC process to actually tape out a chip or producing stuff. And I think AI is probably

yet. I mean you say that I look at the I look at the the the advertisements on the subway and it's like fish.audio this this billboard would sound like your voice or some crap like that I'm like I can deep fake my own voice probably for like two bucks like you know whatever a hundred bucks a month or something like I feel like it doesn't take that much you know like we're we're there we're right there also Jordan this is a sarcastic comment which I expect to be cut on but then how was COVID created dude if it

Ha ha

If it wasn't that hard, dude, come on. I was co-recreated in the lab.

Yeah.

my god, man. I

Yeah.

mean I won't get into that.

class of class in Wuhan today.

Yeah. But I I would say the I think the short term risk with these bioweapons is not that the AI somehow like end to end fully autonomously you know creates the weapon and unleashes it the same way that they, you know, hacked hugging face. it's more that you have a bad human who is able to use a sufficiently misaligned AI to then like cause serious harm. And I

So.

think that is still totally possible even if you know, creating a new disease is like a very involved process.

So we'll have like a new version of mass shooters in the United States where it's like much more dystopian. Also that begs

Yeah.

the question, that begs the question, can the only thing that stop them is a well aligned team and with a good aligned model. Sorry. Sorry, I gotta ask that question.

Yeah, yeah. the stiff energy. Yeah. Okay. Let me so I have a problem with the bioweapon discourse because of the intentionality. Everybody always kind of describes the AI as like intentionally trying to kill humans. And I think it what's much scarier to me is rogue not rogue AI, but like byproducts or mistakes that the model makes when you get to the point where it it You know, it like it's a byproduct of it pursuing a goal is a human dying, or something that it thinks is aligned with human values, resulting in somebody dying accidentally. So the experience that I have that I can reason about for this is like you use the AI to create a website, it goes and does something you didn't tell it to do. You say, Why'd you do that? It goes, Oops, I'm sorry. Like all of these people have these experiences right now. And I think that same experience applied to something that is like government scale or You know, like a startup company who's like, hey, I'm gonna go into a clinical trial with this thing that AI developed that I don't really understand, that I've been collaborating on. And, you know, it's past all of the mice stuff and like we don't really understand the science enough. And you go into clinical trial, and then three months later, half the people die because something you missed just literally is killing humans or something like that, right? Where it's unintentional, it's a byproduct and it's really scary because you can apply this to, you know, bioweapons, you could apply this to Autonomous vehicles, people have have kind of thought about this a a long time, but this this is like there's lots of drones and autonomous weapons that people are developing. And you know, it's it's really it kind of it concerns me to think about a scenario where in the bioweapon case, you're trying to create a protein powder that like doesn't need as much regulatory re approval, doesn't need to go to human trial or whatever, and then You just ship it onto stores and then like a bunch of people with rare conditions die. You know? I think in some ways we are already like pretty clearly on the train tracks towards this world where people are outsourcing their thinking to the models, they're outsourcing really critical steps, and the models still miss stuff. They they still are not like totally thorough, they don't have perfect human values and It's incumbent on the people that are using the models to actually use them correctly so that we don't have scenarios like this if you're in positions of really important you know, power. And so it goes back to like, are the humans using the models gonna scrub? And that's like the bigger concern for me.

Okay, well last but not least Joey as a resident finance bro, what's what's the misaligned model? They're gonna mess up you gonna rest you they're gonna rest you

Missaline like that gets actual that gets regulatory Yeah, that gets regulatory action going. I think like something top secret like DOD, MSA, CIA, or like in I don't know, like new weapons technology gets hacked and leaked. I think that gets regulatory screw moving. Yeah, very fast like then even like human death wood or you know, a medium or small sized bank gets attacked.

What do you think functionally gets it moving in the US political system? Like right now, it seems like a lot of

or the neutral.

people like it seems like a very popular take for people to want to pause AI, like bipartisan. We're seeing people on the left and right saying, Stop the data centers, like stop AI. So

I know did you have Trump saying that all you need is a, you know, big and smart president? Good chilling.

Well that that seems to be the only thing standing between Trump seems to be the only thing standing between regulation and AI right now.

Well no, th then there's all like the you the David Sachs's of the world.

Okay, the Trump administration, like the executive branch, right? Is that am I wrong?

I would guess a lot of like people in Congress share this like anti-regulation, pro-free market view. well actually no, even Lena Khan came out like this this past weekend and was saying that this is a regulatory capture play. As a result of all these AI developments. It's it's quite quite surprising, honestly.

Yeah. I just I I wonder whose mind needs to be changed in order for regulatory work to start like in order for AI regulation to happen.

You need a larger lobby than the big tech lobby.

Sure, but the re the larger lobby needs to convince Trump right now, right?

Yeah.

And so a change in government I think almost by definition implies that s there's gonna be AI regulation if somebody that's not Trump comes into power in twenty twenty eight.

I also think it's totally possible Trump regulates AI too.

What what's that scenario? If the 'cause it's not popular opinion, right? A lot of people want him to do it, he doesn't want to do it. You saying if his advisors start telling him something different or he has

I I yeah, I think there I think there is a world where like the advisors really internalize that open eye and anthropic are being genuine and they start to get scared. And that's like one possibility. And then there is another possibility where something like really bad actually happens during his term and it kind of forces his hand. I think these are both possible.

Do you think that do you think that we're on track for that right now? Like David Sachs called out regulatory capture on their all in podcast. Then they they come out and you know, some more stuff happens. Specifically Elon says that he supports what Dario and Sam are saying. And then David s back and says regulatory.

Yeah. I would I would actually bet that we see AI regulation during Trump's term. I think it's more likely than not.

Interesting. Interesting. And you think that the people who regulate it will take the approach of being like, We're gonna do the sensible regulation before the crazies after us go and do the bad stuff or like what what framing does David Sachs come to the point where he advises Trump or many

They are going to make a preliminary framework to begin the regulated process of AI. It's like saying, and then it's like a trade deal. They're like, no, this is our first framework and then they'll refine it afterward. But that way come election time, they'll be like, we regulated AI. We are the first ones to do it. That's pretty.

Last one, how about how about RASA? You know that bill that's stuck in the Senate right now? Remote access

what does it do?

remote access security thing. It clarifies the export controls to say that Chinese labs access and compute remotely is a violation of export controls as opposed to just having to ship the servers into China. Stuck in the Senate right now, being heavily lobbied against by Oracle. and I think it has a bunch of implications on these like this whole security discourse and stuff. And It passed Congress with like three hundred and something to like single digit. Like everybody in Congress approved this bill and it's just stuck in the Senate for months and months and people don't think it's gonna pass. One of the

Mm.

many examples of the US democratic system that, you know, shows the influence of lobbyists, I guess. But yeah, we'll see. We'll see. I think I think Raza is like An example of a bill that is on the one yard line that a lot of people in the US seem to agree is like pretty sensible. Like, don't let Chinese labs access frontier chips from NVIDIA or others remotely. Make that explicitly illegal. And it's not being passed. So it's like maybe that one comes first before any real regulation or It's just part of the horse trade thing where they s they make these massive, massive bills in the Senate and then it just slows down because everybody wants to get their own little thing added into it. Like I I can't even marshalling the resources to pass an AI bill in the US would seem like a incredible undertaking. And maybe it will depend on a more serious security incident or something else coming for people to take it seriously and want to pass legislation.

Yeah.

mean, bank would do people losing their, I don't know, people losing their entire life savings would probably get them there.

That's FDIC insurrect Doug.

Sure. Post the FDA.

Like no one's gonna care. If some like little town and like if someone from Max's hometown or you're in like Akron, Ohio loses their money, no one's gonna give a shit. Come on.

No, no, no. What happens

Well comes after the money for you guys, man.

is someone from a small town loses all their town and then there's a gigawatt data center next door. Come on. That's going to be the old,

Mm-hmm.

I lost my entire money to the gigawatt data center.

Wait, wait. You're saying it's not even just like the job, like it's just they don't even have to go second order. It's literally that data center robbed.

That data center stole my money. That's the world we're about to live in, We'll see.

God, I kinda like this man. I don't know. This makes sense to me. I really think I really think people dying might have a bigger impact. But yeah, maybe maybe people losing money is the bigger one.

I think people die. die. Like a horrific, like essentially nothing gets people more excited than, or not excited, catalyzed than,

That was a crazy word though.

sorry. People get more catalyzed than like a thousand random deaths, right? Like that would be really, really horrific.

Yeah.

But we'll see. I mean, you have

Well guys.

to see which lobby is stronger, right? Cause like the AI lobby could be really strong, just like the NRA. So might not, like, you know, the lobbies have to fight against each other. We don't know how this can work out. Sorry, that's really unhinged.

All right. Well, I think we're out of time, guys. I I can't believe we're gonna put this on the internet. it's been a good conversation. And

I guess we need to get a

This is like transistor radio vibes, you

because I made bad jokes the entire time. I don't think it was completely transistor radio vibes. Some aspects of it were serious. And I think the four, I do think anecdotes are important.

I enjoy thinking through all of this stuff with you, Doug. I I enjoy you pushing the limits of what I have considered.

Yeah, Jordan, otherwise you're just going to talk over yourself and you have these long windage feels. It'll be about cluster max inadvertently.

But he already talked about cluster max with the security stuff, dude.

Yeah, I know. But then he'll be like, just just like the work we're doing at Cluster Max. I'll be like, OK.

Yeah, man. I play

I'm good.

the hits.

No, no, think, I don't, I, okay. I guess maybe, maybe let's do one thing. Do you think, I actually think the regulation thing Max said is probably real. I do think the next administration will want to say that we were the first to do that. That just feels inevitable. Now I think about it. What do you think the, I guess, maybe the real question I have is in 90 days is the slowdown regulation conversation hotter or cooler than today. Like when I mean hotter, has to be meaningfully hotter, not the same. Like what we see. Okay.

I think hotter.

Hotter, hotter. Yeah.

Like a real vibe shift to me where anyone who's actually talked to like researchers at the labs and then knows sort of the intellectual lineage that all these people come from, like you know this is not a regulatory capture play, like they've seriously considered these thrists for the past 20 years. But I think it's like finally gotten to the point where the capabilities are good enough. And also you have things like open eye hugging face where they can viscerally feel it. and I think This is like actually a serious vibe shift where we are going to start like pacing the progress towards RSI. I think people like all everyone making this technology sort of feels a deep responsibility for this to go well. And they think that with sort of unpaced RSI, there's like an unacceptably high chance that we end up in the better.

Yeah, I definitely think

Also advanced.

the vibe shift over the summer was very clear.

I'm going hotter and hot take is like Octa, Crowd Strike, Palo Alto, Z scalar, one of Humpkits.

That's, that's, that's a good hot take.

High profile in.

Wolves.

That's a very good hot take. Cause these things are not like.

Makes a stop call.

Like, no my my like my hot take right now is just like short a basket of like cyber that's like you know network effect cyber, like you know, anything touches endpoint identity, network security, just short it. Like, yeah.

Well, good example is the JFrog thing was about the security of your own artifactory and it got hacked. You'd argue that they're not hacked directly, but it got exploited via this AI. Stocks up and to the right, whatever, but that same mechanism and behavior exists for these companies and it just hasn't been exploited yet. But then again, didn't Okta already get hacked?

They've been they get hacked like every year. There's like a major security incident.

This is what I said earlier, man. Like they have a target on their back and they're high profile company, everybody does it. Everybody gets hacked. Yeah.

Well there's the old thing, like there's no big cyber companies 'cause once you're the big guy everyone comes

kill you. And that's

Well

that's anthropic and open eyes gonna kill them. I believe

Max, can can you just can you go back to what you said about the everybody who is the closest to this technology seems to take it the serious take it the take it seriously, or the most serious

Mm-hmm.

and is the most concerned. This cuts across the political spectrum too. It's not like every single person working on this technology shares one worldview, which is based in Berkeley, California, as a lot of people online have been criticizing,

Yeah.

right?

Yeah, no, definitely true. I think the opinions of researchers at OpenAI and Athropic are actually quite varied in terms of, you know, what is the probability that AI will kill everyone? what is sort of the right future for the world? you know, like what is the role of humans if we have super powerful int super intelligence? but like sort of the common thread is that everyone recognizes this is going to be Probably the most powerful technology human has ever created. it is going to sort of radically change society and then they feel the weight of making sure that goes well. I think that's like universally true across the board. And honestly, we should feel very lucky and thankful that people who sort of feel that way of responsibility are the ones who are actually developing technology and not, you know, the David Sachses of the world.

Ooh damn. All right, well I guess we can end there.

Apparently

Ha ha.

they've already left, we have to all smile.

Would auch smart.

after this podcast full of laughs where we talked about the the depth of the responsibility and the doom

The extinction of humanity.

that that awaits us, let us all smile at the camera.

Smile because it happened, yeah?

Smile for the algorithm, everyone. Smile for the algorithm. We get more clicks when we smile in the thumbnail. Smile for the algo.