Anthropic vs OpenAI Usage, Margins, Meta Compute, Future of MSL

everyone. Welcome back to SemiAnalysis Weekly. We're here with episode twenty with the tokenomics team. Things are moving fast. We are recording this on Wednesday, July 15th. I assume by the time this episode comes out, there's gonna be a lot more releases of models, disclosures from these companies about their profitability, revenue, how many active users they've got. Regardless, we're gonna Record a point in time right now, talk about token budgeting, Meta's compute strategy, MSL futures, the release of Fable Soul and Anthropics profit margins. So excited to dig in. With me today, we've got Max. How you doing, buddy?

And great. Thanks for having me.

Welcome. we've got Crystal. How's it going, Crystal?

A Jordan.

Yeah, we Joey.

Enjoy.

Cool. So we're gonna start with token budgeting. Crystal, over to you. We got lots of conversations that we've been ha having with enterprises over time and just asking them like what's going on. at some analysis we're still token maxing, but others are moving into the time of austerity. Can you take me through a little bit about what you found with this article?

So there's a lot of pe like enterprises right now that are cracking down on their budgeting because they think their employees are spending too much and a lot of people are getting scared because they're like, what if I don't have enough tokens to do my work? I don't think that's necessarily true that and it applies to most people, right? Because a lot of people aren't even getting close to that limit. There's probably a few handful power users at each of these companies which will really be affected. But other than that, I think it's been it hasn't been as big of an impact, I think, on most people's workflows as it has been portrayed on social media and stuff. There's been a lot of different strategies that companies have been imposing, whether it's like on a per person basis or like a monthly budget for the entire company from what we've seen. And I think the per company basis obvious probably makes a lot more sense than like on a per person basis, just given that some users will generate more like use from it than others.

And how do you see people actually enforcing this? Like, clearly you can burn through budget faster if you're going with the super ultra premium max thinking model, but some in some cases when you're actually enforcing a budget, you take different approaches, right?

Yeah, yeah. So it de I've seen some people actually, so I was talking to some people and they are at like a smaller company and they have a much smaller budget. So the way that they do it is they try to optimize how they're using the like more expensive tokens for like anthropic or open AI. So they use like a cheaper model to like process a lot of it and like condense it down to like a smaller prompt and then they use the more expensive models or they use something that their company's not counting. So there's a lot of like discrepancies in terms of what models actually count. Some people or like some enterprises are counting their own models into the budget, but some enterprises aren't. So there's like if they aren't counting it, a lot of people will tend to like really push those to the limit, use those a lot, and then use like the ones that actually cost money.

And you're seeing people, maybe Max, I can bring you in on this one. Y you you've been seeing a lot more people burning tokens for coding rather than other use cases, right? Like that seems to be the market that's growing the fastest.

Yeah, I think coding slash like software engineering in general is just by far the most token hungry use case. And I actually think this is why a lot of the token budgeting discourse is pretty like I think a lot of the budgets themselves are pretty uninformed. Like I hear people saying that they want to make sure their sales guys aren't using Opus or Fable to write emails, so they should definitely be using Sonnet instead. It's like, dude, like generating your email is basically free. from a tokens perspective. Like it does not matter if you use Opus or Sonnets to write this email. Like I have no idea why this is the policy you're enforcing to try to like reduce your token spend. I also heard lots of stories like certain companies where only the engineering team is allowed to use like Claude Code or Codex and then you know everyone else only gets like cowork or something like that. I think that's pretty short sighted and it's honestly you shouldn't have a cast system where your engineers are at the top. and everyone else is just forced to use like dumber AIs, you should really be giving everyone the opportunity to try these tools and figure out how you can become more productive with them.

Yeah. Joey, are you using these models right now? How'd you feel if I got access to it and you didn't?

I would probably be, I do use some access, not as much as like Max and other people at SemiAnalysis. But that's probably just the nature of our work or my work right now and that's changing a little bit. But yeah, I'd be pretty pissed because I still use Opus 4.8 a lot, like to pull in, to do more like agent research tasks that will pull in a lot of like different data sources. do more like research type tasks. I'd be pretty mad.

How do you think about the ROI question? I think a lot of people, the the whole motivation behind token budgeting is clearly that somebody is not seeing a return on this investment. either that or they're just like not seeing it yet. but I think in a lot of that research you're doing, you're seeing that there's a lot of benefits, right, to using it.

Yeah, I think like with the market being so coding focused today, so probably 70 % plus of ARR, lab ARR right now on the API side is encoding. And then two, like from what we've heard, it's very power user, a power company driven. So it's pretty widespread like coding. There's no like major big company for Anthropic. Like Meta is probably three to 5 % of total Anthropic ARR. But it's heavily driven by power users. if you're at a point where you're a power user at one of these companies, I think the ramp data is like the top of 1%, you know, the 99th percentile of companies. spend $100,000 per year on AI per employee. And if you've gotten to that point where you're spending that much, you're obviously getting pretty good ROI. And I think our conversations with lot of enterprises were these top engineers could go well above their limited, whatever their budget limit was. But if you were someone who is spending an insane amount of money, you were getting budgeted already pretty quickly if you weren't seeing ROI. And I think there's always projects that you'll try that don't get ROI. scrape that, you'll scrap that and then move on. But I think it was, yeah, it was pretty obvious that because the market is so coding focused, like people are getting ROI, they continue to spend. I think we continue to hear that NetNew ARR, both Anthropic and now OpenAI with like Codex and 5.5, 5.6, like since late March, like spend is broadening. there's clearly lot of API tokens today. And yeah, there's a lot of organizations that aren't traditionally doing a ton of development type work or focused, but they only allow their middle back office workers to use Copilot 365 or Sonnet, things like that. But I think the market will continue to broaden out.

Yeah what? Do you do you guys do you guys have a take on the balance between API usage in applications versus like coding plans when it comes to people's individual usage? Like do you ask them when you're talking to them if they are using a cloud code subscription versus API paying per token?

Well, I feel it.

I mean, anybody who can use a subscription definitely should use a subscription just because it's so subsidized. it's just, you know, OpenAI and Enthropic are very wise to this. And so when they have their enterprise plans, like same analysis is on a you know Anthropic Enterprise Plan, they just they don't let you use a subscription. They force you to pay for a token because they know that's where the margin is. But I think in general, like anyone who can use subscription does use subscription, and it's just the people who need more limits than that that pay for API.

I've even talked to like a few people at like startups and stuff and they're not on like the team or the enterprise plan at all. They just buy five however many like Claude or OpenAI subscriptions they need a month, charge it to the company card and it's just they just keep doing that. If they hit the limit on however many accounts they already have, they just buy another one for the month 'cause it's just more cost effective that way for them to operate than it is for them to actually use the API billing.

yeah, if you're a startup that you're like young enough to where you don't care about like security or any of the other enterprise guarantees, like just absolutely buy, you know, 10 subscriptions per person instead of you know paying for API credits. Like I I think the recent you know study we did on this, we had a decently viral tweet said that for Codex, I think $200 a month, you get like around 12K worth of API credits. And for Anthropic, it like 200 a month for around 8K. Like obviously that's a no brainer.

Yeah.

for anyone that is able to pick between those two.

Can you dig into that a little bit more, Max? Like when you're talking about the margin profile of these businesses and then you're considering the subscription plan versus the API, like what is that what does that look like? I'm gonna bring up the chart on screen that I think is the key one from that tweet thread.

Yeah. yeah, so essentially, obviously when you hear that like a two hundred dollar month plan can generate eight thousand dollars worth of API credits, the subscription is going to be much lower margin if the average user has anywhere near a hundred percent utilization. and so really the key question is sort of like what is the actual average utilization across our entire user base? And this chart that Jordan shows here is like kind of the break-even percentage for the various plans. We can see for the Cloud Max 20X plan, it's like just 10%. For the Pro and 5X, it's like a little better at 20%. OpenAI because they are sort of even more generous with their subscription limits. It's slightly lower at 11.4% for the Plus and Pro 5X and just 5.7% for the Pro 20X. We're pretty confident that The average utilization, especially for these like 20x plans, is much higher than 10% and 5.7%, respectively, for anthropic and open AI, which would mean that like forget being, you know, worse margin than API, which we think is like likely 85% plus for anthropic at least. it might just be like negative margin in general for these 20x plans. And so, of course, like these businesses want to move as much volume as possible to API. I think maybe Joey, since you wrote the newsletter on Anthropic's business, this would be a good segue to talk about why we think Anthropic is potentially in a stronger position than Open AI.

Yeah, and I think too, like once you move on to enterprise subscription plans at Anthropic, there's no usage included in your subscription fee. all done at API pricing. Yeah, I think at Anthropic, what we found was really interesting is because the business today is so much more, and this is changing, but because it's 80 % plus of ARR is on the API side, and that is, as Max showed, is really high margin, you know, they've gotten to a point where they're profitable today. You know, and some of that is on a non-GAAP basis, so excluding stock-based compensation, but like operating profit was profitable in Q2, you know, could be profitable to the tune of 1 billion plus in Q3. Some of that's a function of they've just grown so much, I don't think they can hire enough or like plow enough back into training as they'd like. But because of that margin, that kind of gross profit advantage, that they have today, they can plow as much back into training as they want and can kind of extend the model advantage and model capability advantage. We think that's something they should. obviously try to do. So just because it's very profitable for them today, it doesn't mean they should go run at 30 % even margins right away or as quick as possible. Show that this is a very profitable business model and continue to invest as much as you can into trading as possible.

Yeah, can you can you talk about the mix between consumer and enterprise, assuming that consumers those paid plans and enterprises pay per use on the API? Like when you when you look at these things, clearly the bulk of like Twitter users are using the subscription plans, but then you get like one really you know big customer who's spending so much more money on the API tokens that it kind of drowns out everything else and that

Yeah.

I guess makes it a good business.

Yeah, and because Anthropix really focused, they have been historically focused on enterprise. It's 90%, it's 90 % plus enterprise. It's, you know, versus chat GPT has gone into, you know, started having a consumer up in, you know, back in Q1, 60 % of ARR was in consumer and only 6 % of the 950 million weekly active users end up actually paying for a plan. and most people pay for the $20 a month plan. It's a much different business mix because then OpenAI has to subsidize and pay for it, not subsidize, but there's a cost to serving those other 900 million weekly active users that are free. So it lowers their gross margins on a blended basis by about 20 points. That's changing. I think, we think personally with some of the rise in OpenAI's API business since late March with Codex and then 5.5 and 5.6 that the 60-40 consumer enterprise split has flipped now to 60 consumer versus enterprise here in Q2. And it's, you know, from what we're hearing, like in the channel, especially out of, you know, token as a service businesses like Bedrock and Foundry is that, you know, OpenAI has recognized this and it's starting to shift and even, you know, ARR, like NetNew ARR is starting, you know, to look a little more even between Anthropic and OpenAI.

Yeah, and it's kind of shocking. Like with the release of 5.6 Soul, the guys like Tebow on X who are tweeting about the daily actor user counts are saying that they're celebrating going to seven and then to eight million active users. But to be clear, they've got over 900 million weekly active users on the free tier of ChatGPT. Like They've got a lot of room to grow to just get people using the number one paid product. Maybe they got a few extra users using other paid products, but I would assume the bulk of people using the paid products are using codecs, right? So that's not a great conversion rate at this point of in terms of who's actually using the full paid product from the you know base of 900 million.

Yeah, it's funny, you actually see it too in the paying rates for like even the consumer version of Claude. Claude is like 9 % of free users end up paying. So it's a much more focused user base. If we think of those 900 million weekly active users, a lot of it's like Google search type replacement, things like that. then, because of that consumer focus, they've been working on advertising a lot. consumers traditionally monetized. People aren't willing to pay. The retention curves are typically pretty poor in how many people are paying three months later, six months later, 12 months later. Ads is a tough business model. It's tough to scale. But maybe that's the eventual. They've put out some pretty crazy targets for 2030 advertising revenue. But yeah, consumers are tough business. Enterprise is obviously, given the margins on tokens for API, enterprise is the place to be. I think Anthropic had a pretty good lead because they were so lever decoding early the first six months of this year. And it seems like it's shifting to more of a kind of equal two horse race here in mid July.

Yeah, Crystal, what are you seeing in that two horse race? Like are people that you're talking to using anthropic, codex, both?

feel like a few weeks ago when I was talking to people, people were pretty heavy on using anthropic Claude Code specifically. And if I asked them why, they would say that's just what I've been using, right? And I'm used to using it now. But now as people in like the consumer section start using like ChatGPT a lot more than like Claude, that effect in like the consumer sector kind of starts trickling into like enterprise a little bit where like these people who were previously like Cloud code, like power users didn't touch Codecs at all. They're kind of like, okay, what if I like consider tinkering with Codecs? So even though consumer is like a really small portion of the pie and the margins obviously aren't as great as enterprise, like and historically speaking, OpenAI hasn't done as well in the consumer market as like Anthropic has, but like they have the that traction in the consumer space that's eventually like it's going to trickle in, and we're already kind of starting to see that in like just this past week already.

Max, what's your take? Opening eye versus anthropic. Where do things stand for you?

I think I remember last time I was on this podcast, we were talking about how it was it was right when five five came out. And we were talking about how things were looking really dire for OpenAI at the start of the year. I think ARR growth was likely flat in like March and April, which is like the alarm balls were definitely ringing, you know, in Sam's head when that happened. but then we said that five five would like to be an inflection point. Like this was actually a really good model, you know, probably Opus 4.8 level. and I think that prediction has played out even like sort of more quickly and more optimistically than we would guess. Like some people here are saying that five six is as good as Fable, sort of just like straight up, even though it's only half the price. And we also likely think it's like a much smaller model, which would be sort of a testament to open AI's I guess like training abilities. we think that like as Joey mentioned earlier, net new ARR like month on month is like the comparable between OpenAI and Anthropic now, which is Pretty shocking. Like people people were counting open AI out a little bit. I think Jordan might have been one of those people, but they're but they're back, guys. They're back.

I am nothing if not flexible. It's always been about the model, guys. Like everybody's like, when are you gonna stop hating on open AI? As soon as they ship a model, that's good. And it's good. I mean, for the things that I want to do, yeah, I'm yeah, basically default open AI at this point. despite the application not doing what I want it to do with multiple tabs and stuff like that.

Wait, are you using the Codex app or like the CLI? What's what's your foreign fact? Cli Yeah, I feel like they don't want you to use the CLI, dude. Like they they very actively want you to use the app. What is what is pulling you back?

See you live. Tabs. I need like multiple tabs going. Yeah.

okay, I think I think the way you're supposed to do this, because I agree, like the first time I tried using codecs and I was like every single time I open a new chat, it's just like lost in the ether on like you know the left-hand sidebar of all those tabs. I think how you're supposed to do it is like you have one pin thread like per project you're working on, and then you just like ask that thread to spin up sub threads anytime you have like a discrete ask. And so you actually like never even interact with the vast majority of threads like in a left sidebar. It's just like your couple pen threads that you use.

Yeah. that sucks. Second of all. second of all, I'm in remote SSHs all the time and their remote SSH plugin in the thing sucks. So yeah, both of those are I'm very against it. I think th they they are working on this, but yeah, I'm I'm just I'm not reading the code to be clear.

Ha ha. Let's see. Makes sense. Thanks for clarifying.

Just yeah, last time you were on, I you know, we had that discussion, but you know, still have to run some things, still have to run some things from the terminal and see some output in the logs at some point. And this is just like not part of the standard codex thing. But I'm not the primary market. Like I understand that they're targeting vibe coders around the world, you know.

Yeah.

We need to help people make more B2B SaaS is like the core business model of this. And that's I'm not the target market and that's cool. I'll keep using the CLI. It's all good.

Actually, I I wonder if like internally most of their software engineers are using the app with CLI. Summit analysis for sure, I think all of our real engineers are like CLI, you know, diehards, right? Like they'll never give it up. Yeah. Well, next time we I see some OpenI people I'll ask.

That would be a good question. You're gonna ask them that. Yeah. Sounds good. Okay, let's let's change gears and talk a little bit about who's gonna come in third. There's been some releases that we've covered a little bit. SpaceX AI is back. They've got Grok with cursor bolted on and things are going well. Meta has Muse Spark out in the world. They've also stacked up a whole bunch of compute and they've teased launching a NeoCloud, which of course SpaceX was first to do. So

Yeah.

what do you guys want to talk about first? The models or the compute for these guys?

Let's do compute. I think it's more interesting.

Jesus Max, I was going tell you to walk through the models. think, yeah.

I I quipped TLDR the models is like they're not bad, but at this point, I don't think it matters unless you can deliver like a true frontier bottle. And they're obviously not true frontier, so who cares? But as like as maybe sorry, go ahead.

Do you think anyone will use? Max, we want to use their API. Feel like product?

I don't think anyone's gonna use the API until it's a frontier model. Yeah.

Okay, so when they have an API, their token as a service business will be fired. They'll print 85 % margins too and...

It'll be fire. I mean that's only if it's like better than whatever the best anthropic and open eye model is at that time, which I think easier said than done. I think like Grok four five, Venom E Spark one point one, they're only useful as like just like proving points along the scaling curve. It's like, okay, these guys, unlike Gemini, they're like still like kinda in the game right now. you know, they can train like adequate models. It's not quite frontier yet, but there is a non-zero chance they can catch up. I think is the way to view them. I will personally be shifting zero of my token usage to either Grok four five or newsmark one point one.

okay, but you said the other name, which is shockingly, in my view, in fifth place right now, Gemini. Do you share that view?

Yeah. a hundred percent. I think they're like clearly in fifth place. and like unless unless Gemini three point five pro is like better than the you know industry chatter that we're hearing, I think they're gonna stay in fifth place, maybe forever. Yeah.

Mm-hmm. Maybe forever, Max. Come on. Okay. Talk to me about compute. So staying in fifth place forever would kind of be related to the quality of the model, but the quality of the model is kinda tied to how much compute these guys have.

Yeah, yeah, yeah. And I think so I have to say that when I first saw the SpaceX anthropic deal, where Yvonne rented like basically all of Clauses one to Anthropic, I think I incorrectly viewed that as Yvonne giving up. When in reality, like I think the really important question is like do you have the ability to claw back all of your compute, assuming that you have proven that you can truly reach the frontier by scaling up your compute? And I think that's sort of like the view that Elon is taking with like cursor and XAI. He's essentially saying, like, I'm going to leave you guys with enough compute to sort of prove that you're capable of reaching the frontier. And in the meantime, I'm gonna monetize all the additional compute by like renting it out on these like extremely high margin, you know, three X, maybe four X market rate rental deals. But I'm gonna make sure there's this like 90 day clawback clause. included in every single one of them, such that if you do really well, I'm just gonna clob it all the compute, give it to you so you can, you know, make digital God. And I think that is like a valid strategy for someone who still truly thinks they can put like build RSI. and I think you can really think of where it's like OpenAI and Anthropic are monetizing their compute by just like selling inference, Elon is doing these short term compute deals instead. I think this is fine. I think on the topic of meta compute, like you there are the rumors that they're gonna become a neo cloud. I think as long as they do these like SpaceX style deals with a clawback, or they do like, or they just like give it all to Rex, or they like do token as a service, like any monetization method that allows them to in theory claw it all back if MSL proves they're doing well. I think that's like an excellent smart decision on Mena's part. Probably allows them to be more aggressive with buying compute in the short term. And it still like gives them a chance to build RSI in the long term. I think the issue with Google and why they're in fifth place, aside from their model like Gemini three five flash being bad, is that they have signed all these long-term compute deals where they cannot claw back the TPUs that they're giving to Anthropic, right? That's just like committed for the long term. And I think it indicates like a lack of conviction on their part that they think they can build RSI. And I think you know, Noam Shazir, John Jumper, like all these really high profile guys at Deep Mind Leaving is sort of because of this fact. Like they are AGI pilled. Their leadership is not AGI pilled. and they think that the only way they can do the only thing they can do in this situation is to go work somewhere else.

Wow, okay. Hot takes. So two things I want to dig in on there. the first is related to the clawback. So this is a valid strategy, but you can basically only do the clawback once and then nobody's ever gonna buy from you and trust you again. Or do you think you can actually potentially oscillate between these two?

Well, I think in an ideal world you only need to do it once because y you only do it if you're confident that like I have I'm gonna be able to breach the frontier by continuing like scaling, right? but I also think it's possible you could do it more than once. I think in principle Enthropic like viewing this as a short term deal. and I think if Elon said like, Hey, actually I want it all back six months from now, and then a year from now he's like, JK, we actually still can train a frontier model. If anthropic is still ripping at that point and it's in desperate need of like compute capacity, why would they say no?

Yeah, I guess financial motivations can make friends out of enemies. so second thing is just on the topic of the business. Maybe Joey, I can bring you in here. What's a better business? Selling tokens of your frontier model or close to frontier model, or selling compute access on the short term at a premium over the market?

Selling your own tokens is probably the best model. mean, even selling someone else's tokens if you're AWS or Azure is a pretty good business model as well. If you can participate in some of the upside in revenue share. But if you can run above 3X market rates, it's not a bad business. I think before like... In like, from the MetaCompute perspective, it's been floated for a while. MetaCompute has been floated since like Q3 of last year. And it gives Zuck a backstop to say, okay, if MSL isn't successful, we have means to essentially make decent ROI on our CapEx spend. It allows them now to spend a ton more on CapEx than 27. you know, potentially 28 because there's a market and you you've gotten these market signals that Anthropic can pay, you know, 3X market rates and you know, you can do a small deal here and that's in, you know, the background investors can feel comfortable that, you know, if... If MSL is deemed unsuccessful, then Meta could have a pretty nice single client or a few client like NeoCloud business just serving the labs.

Yeah. But I I mean, I I thought this chart that you guys had in the Metacompute article was striking where you can see that the SpaceX plus Google deal, they're monetizing this compute at four X the market rate. So I think in the past everything you're saying makes perfect sense, but I would challenge a little bit that like this near term SpaceX GB three hundred rental deals aren't a Really good business. Like there's clearly high margin in this, right?

Yeah, mean, that's extremely high margin. think like when you think about the ARR per gigawatt, I think a lot of people would be happy to get 48 billion of actual token ARR, like of inference ARR. So I don't know. That deal doesn't seem very economical for Google, but you know.

They're telling a you think their metacompute is them telling a story to the market that they're going to be able to replicate this big XAI thing, which is probably a one-time thing, unless demand keeps going so exponential that they're such a compute crunch that everybody's fighting for access to the GPUs. And well, we have them and we can monetize them at like four X the market rate from Neo Clouds. Where NeoClouds, I think we already I think our house view is that NeoCloud is a pretty good business on its own at like twelve billion per gigawatt. Approaching fifty billion per gigawatt's like four times better. Yeah.

Yeah, and the labs have gotten so good at the ARR per megawatt or gigawatt numbers gone up a ton. So token throughput's been pretty impressive. Just the models as well, and then too how they've monetized them, the new models price higher. So if you look at Anthropic, revenue per megawatt was like 16 million per megawatt last year, or 16 billion per gigawatt. And now... you know, that's more than double than in three quarters. And I think there's some people that think that can double again in two to three quarters. So.

It looks like it's on that trend. I've got this chart on screen right now. So total token as a service market. I two things jumped out at me. One, how fast this thing is growing. Like I I thought the market was pretty big in Q4 of last year, and that looks puny compared to our forecast that you guys have on this chart towards the end of this year. like way more than doubling, as you said. But the other thing that strikes out that jumps out to me is just how bad Microsoft Foundry is doing here. Can you?

Yeah, and that'll be revised up with the recent OpenAI API success. But I think up until because OpenAI, were, Foundry was 90 % OAI and OAI was so consumer focused and API didn't really, this is not the latest version of this chart, but yeah, Foundry's doing a little better now, maybe Vertex is doing a little worse, given Max's thoughts on Gemini than that chart, but yeah.

Subscribe for more listeners. You gotta you gotta subscribe for the most recent tokenomics forecasts at these companies live.

Yeah.

There's always a sell.

We we can't we can't be giving this away for you know for free, dude. You gotta you gotta pay for the tokenomics model if you want the most up to date numbers.

I'm

Yeah. And we were nice enough to leave the axis on that chart this time. But yeah, I think we're like, we're of the view and like when we talked to enterprises, have token as a service is a very popular choice for consuming tokens.

Yeah.

AWS and Azure have massive businesses with Fortune 500, Global 2000 companies that spend nine figures plus on cloud a year. And if I can go to an existing vendor and have more model optionality, increase my ELA or my credits and then burn that down, it's pretty attractive for both AWS and Azure. think like... you know, if it was 5 % indirect like sources, like, you know, this token as a service for like Anthropic six months ago, like that's probably, it's probably 20 % now of, you know, the business.

Yeah. Crystal, what's your take on people using the big three hyperscalers, token as a service stuff versus the smaller startups like a Together, base ten, fireworks, anything they can get on open router?

I feel like as we see more of the bigger enterprises, so like I feel like a lot of the financial services industry still hasn't unlocked like and use AI to the level that they should probably can and should be. And that's where like Azure and Bedrock are gonna be benefiting from because most of them probably already like buy their services, right? And then once they do get like internal approval or whatever to be using these AI tools, like they'll just buy it through whatever existing channel they'll have already. Whereas it's like probably going to be like that's where the bulk of the market pretty much is.

Makes sense, yeah. Max, how about you? What do you think of token as a service from like the startups, how fast to gather fireworks based time they're growing compared to the hyperscalers that are kind of tied to the big labs, but maybe not actually selling a lot of tokens to the startups of tomorrow?

Yeah, I mean I think Together, Base and Fireworks, these are all super impressive businesses. I do actually expect like open source token volumes to grow slower than frontier token volumes, but that's only because I think frontier token volumes are gonna like absolutely explode. I think open source token volumes are also gonna explode just to a slightly smaller degree. and I think like it's kind of funny. I feel like if you talk to the VCs investing in like Together Fireworks and Base 10 and you ask them for the explanation why. Oftentimes it lead with like inference is gonna be like the largest market ever. And so it doesn't even matter if like these companies can only capture like a super, super tiny slice of this like extremely large market. That's good enough for us. And I think that thesis is like honestly more or less right. I don't expect these guys to ever be doing like more volume than an open air anthropic. Or like nowhere near that. I I actually think like this is in terms of like global token volume, they might be like the highest percentage they've ever been, like they ever will be rather today. but they'll still be like good businesses in the future, I think.

Yeah, how about the rumors that they can hitch their wagon to some of the bigger labs? So like SpaceX AI starts to win a little bit and then fireworks grows because they're exposed to cursor and actually have some I don't know exposure to that growth.

Yeah, well, I think that's gonna like last in the long term. I think the only reason why Fireworks got that exposure is because originally Compose was post trained on Chimmy, right? I think if you're a Frontier lab that has made like a new model from scratch, there's no reason to believe why Fireworks' kernel engineers would be better at optimizing that model than your own engineers. so I don't think they can get like any share of that market in the future.

Okay, so why do the hyperscalers get a share of that margin with the big three?

I mean because they just have like and I'm sure Joey can speak more on this, but they just have like the enterprise distribution that the NeoClouds do. Like Enthropic is not giving up this margin to Bedrock so that Amazon engineers can like make Cloud run faster on you know B300s or whatever. Or it's just so yeah, sure trade him. But it's just so like all the existing customers that are reliant on bedrock can also use cloud models.

Bring him, but yeah. Sounds well.

Yeah, I think the deal, mean, there's, we hear more, we don't hear anything, but people tell us, you know, they think the deal could get restruck. Cause it's obviously, you know, it was very beneficial for AWS. I think we've read none AWS and their margins and how they monetize, you know, versus just selling the bearer, you know, compute for Anthropoc to run inference on, you know, to get that 30%, you know, 20 to 30 % revenue share, you know, that just falls down to the bottom line. It's pretty attractive. AWS, Azure, lesser extent GCP, obviously have massive customer bases. People's cloud estates are there. enterprises are very comfortable security and compliance and everything now with cloud, even your more regulated industries are. So it's a natural place for them to want to buy. But to see that much of the economics for Anthropic is obviously, I think a lot of that was because when the deal was struck and kind of the incentive to make tranium work on that, they were able to run with those terms and make it pretty favorable. So yeah, we'll see what happens if those revenue share deals can continue.

Yes. Okay, let me make one more attempt at the bull case for these token as a service companies that aren't the hyperscalers. for AWS and Google, obviously a big portion of the benefit of getting token as a service from them is that their engineers are are working on training and TPU, and that might be different than the experience that the lab has with the GPUs, where they have more experience like running this themselves. There's a whole class of chip startups that are coming to market right now. And an obvious way in which they come to market is by partnering with these token as a service companies where, hey, the the biggest frontier labs aren't going to spend a whole bunch of time optimizing for, you know, the seventh best chip startup that's coming to market right now. But if they do and that chip startup strikes something really nice for a given model that makes it a lot more cost effective or a lot higher performance to run instead of NVIDIA, then they've got a shot at doing something super unique. Do you think that's a, you know, potential future In terms of where these kernel engineers who have learned a lot end up going and spending their time over the next few months or years.

This is a good point and it's reasonable in the short term. But if any of these new chip startups actually reach like sufficient scale, I think the labs will just dedicate teams to making their model like run really good on their chip. I think there's just I think there's no world in which you have a new accelerator that's like meaningfully better than Nvidia, and it's also being like sold at large volume that OpenAI is like relying on together to serve their model like on that chip instead of just like working directly with that company to, you know, develop the first party capabilities to run their model on that chip.

Yeah, you think it's gonna go the way of Cerebrus, where the chip companies to be successful are effectively gonna have to become a neo cloud for themselves and have to have the relationship with the frontier labs. Okay, makes sense. maybe we can finish by talking a little bit about MSL with Meta, particularly Max. I just loved the like crash course on RL that this article turned into. Not necessarily

Yeah, exactly.

How it works from a technical perspective, but how it works from a business perspective. Like where people buy data, how they build these environments, what sort of what the market looks like. Can you kind of give a an over like previously in the world where everything's pre-training, it's just like whoever has a You know, frontier class team and the most compute can just train the biggest model and win. They follow the scaling laws and go there. But now there's a scaling law related to RL. How how is this playing out in your mind at a high level?

Yeah, yeah. I I think it's I think A, it's like really important for everyone to understand that reinforcement learning or RL is like probably the most important scaling law for improving model capabilities today. And there are a lot of people who believe that the only thing that's stopping the models from being able to do like literally anything a human can do on a computer is having sufficient RL environments. This is this is sort of like data that lets the model like try to complete light collar tasks itself until and it can sort of like repeatedly try including the task until it fully learns how to solve it. And there's like an entire new industry slash like supply chain of these RL environment startups whose like entire job is to convert real world economically viable tasks into these R environments that can then sell to the labs and allows us to use it to improve their models. yeah you see like most of the main players on this chart Jordan has pulled up here. I actually think like some of these AR numbers might be slightly understated. I think it's like pretty consensus that the total sort of data budgets at the Frontier Labs this year. So this is like primarily OpenAI, Anthropic, Google, Meta, and XAI, but also the long tail companies like you know, Meta, sorry, Amazon, Microsoft, like Thinking Machines. I think all those summed together is gonna be like well over $10 billion. that's sort of like a 10x relative to last year. It's very possible that 10x is again in 27. and this is just like a hugely important market for improving AI capabilities. this is probably like maybe the only market in the world where in customer demand, I guess maybe compute is the other one, but like in customer demand is just not even a question. For all these startups, like they will never have a contract turned down from the labs because it's too expensive. It's just a question of like, can you scale up, you know, creating high quality data fast enough? And if the answer is yes, I do will pay any price for it. so yeah, this is like I guess maybe as one other like side note, one of the reasons why anthropics models are the best at coding today, or at least like they definitely were before five six soul came out, is that they were by far the most aggressive. From buying coding data from all these oral environment startups. I think some Delta Labs are starting to like realize this and catch on, but it is definitely like a super important industry everyone should be aware.

Can you d dig into a little bit of the process to create some of these tasks? You you kinda ran through this in the article by describing how Meta has moved thousands of engineers to doing this work and and also dispelled the notion that this work is like meaningless soul crushing stuff and is actually, you know, pretty both economically valuable and I don't want to steal your thunder, but you had a nice line on that one.

Yeah. I think I said it was both potentially more economically valuable and intellectually stimulating than like your average big tech job. so like yeah, I mean I think a lot of people think like they hear the the phrase AI data and they still think like, we have some branded people in the Philippines who are like drawing bounty boxes or like labeling text as like NSFW or something. And that's just like like dude, the models are already fully solved that. like your data is only valuable if it's doing something that the models don't already know how to do. And so what this means in the case of software engineering is that in order to make a good like software engineering task today, it typically needs to be something that would take like a really good human engineer maybe like a full day of work to solve in order to be something that like the model can't already one-chart itself today. And so if you want to create this data, like essentially you need like a really good human engineer To sit down and like think of an example problem that he would actually want to do. You then need him to like create a verifier, like usually a set of integration tests, maybe along with the rubric that can check if the model actually slightly completed this like day-long task or not. Obviously, this is easier said than done. And then you also need this engineer to like write a prompt for the model that like asked it to do this task, but also in it's sort of like needs to fulfill these like two competing factors where it needs to be like a hundred percent fair and and like unambiguous what you want the model to do because you can't sort of like incorrectly fail the model during training because it like successfully did what your prompt asked for, but your prompt was just not specific enough. And so like the model didn't know it actually had to do some extra thing. But at the same time like you need your prompt to be realistic and natural sounding and representative of something like a human would actually ask an AI to do in the real world. And it's like very difficult. To get this balance right. we heard for like the highest quality coding tasks, like the labs are willing to pay well over five figures for a single task. And you know, this is obviously already entering into the realm of like how much you would pay a decent engineer for a full week of work. and so I think that should sort of just like dispel any myths about this being like easy, you know, mind-numbing work. I To all the listeners out there who are looking for a new job, like if you are really good at creating RL like tasks, you can make seven figures plus annually at this point. So maybe maybe consider that as a new job option.

Okay. Yeah, that maybe we have some listeners excited. Can you maybe actually say like one click lower? Do you have personal experience with striking that balance between easy for the AI to do and not impossible for the AI to do? Like what sort of intuition could you give to a listener about what that means?

honestly it's like it's always changing. so so back in the day, like I actually did sell some RL environment data to the labs myself. don't do it anymore. but it was like much easier to create data even just like eight months ago when I was still doing it than it is today. I'd say like today it often looks like you need to identify a specific failure mode that like you are aware of in the model, and then you need to create like RL tasks. specifically target that failure mode. And really the only way to know if it's like at the right difficulty for the model is like you have the model try solving it 10 times and then you see how many times it's successful. You sort of just like repeat that iteration. So it's like I I don't know if I can really provide any like blanket advice on how to find the right difficulty other than if this is like your first time trying to do this, your first thought is almost certainly too easy. Try making it 10 times harder, like try that. And then maybe, maybe it'll be like in the rectangle level.

Interesting. Cool stuff. Okay guys, we've got a worldwind tour. anything you think has that I've missed as we've gone through token budgeting, anthropics profit margins, metacompute, msl. What what what have we missed? Crystal, what have we missed?

Yeah.

I don't know, nothing I can think of.

Just been too busy watching the World Cup here. We got we got Wednesday, July fifteenth, right when we're recording this. We just got to watch England get knocked out as Argentina stormed from behind for a nice two one victory.

That was crazy. Dude, there's like a bunch of British people in the office right now, and they were just like all depressed downstairs. It was crazy.

no. Joey, how about you? What's what's on your mind as we wrap up here?

Yeah.

Nothing, token spend is up and to the right right now, so it's good. I'm excited for hyperscale earnings. Starting next week, we've got Google, Amazon, and Microsoft the week after. It should be pretty good on the top line.

Okay, let me go around the horn. We'll close by getting a vibe check from everybody. Joey, what's your vibe like on the market?

On the market? As long as LabA R is going up and at a good pace, it doesn't decelerate, I think things are fine. And right now it's going up, opening eyes, catching up, should be good. That's not what you're seeing in semi and see in the last few days, but that's just summer, know?

Yeah.

Summer momentum, doesn't work. it'll come back. People come back to the office from vacations and semis will rip in a year and things will be good.

Yeah. yeah. Yeah, we're gonna see an acceleration after the summer pop. Joey's gonna shoot eighty four and see lab ARR go up and be happy. Okay, Crystal, how's your vibe?

I it's going I have to say I think it's going good. Same thing Joey said. I need to leave San Francisco before the AI bubble pops. So very quickly moving away from San Francisco before it's too late.

Cri Crystal, you think it's a bubble?

Yeah.

yeah, Max, we should publicly talk about our bet here.

Yeah. Do you guys want to start a new podcast right now?

yeah. So Joey Joe and I

Yeah, we have the loser. So Max chose the over under of Anthropic ARR at 400 billion to end 2027.

twenty twenty seven.

Twenty twenty seven.

Yes, 2027. we're at like, you know, we're in probably, yes. Yeah, Yeah, you can figure it out on the token-arms model, but you know, it's...

Yeah, yeah, yeah. You can't give them the actual number right now, Joey. They gotta they gotta subscribe to the token optics model for what the actual profit AR number is right now.

Yeah. Okay, but i it does does somebody have who took the over, who took the under?

I took the under a 400 million. Yeah.

So obviously I I took the over. Jeremy and Joey actually both took the under. But we haven't we haven't decided what like the actual bet's gonna be for yet. I think at the off site this year we'll figure it out.

Know that the bet is like,

Did you guys see us wagering in

when you lose, have to write.

did you guys see our pot at raise last week? I had I pulled out a Canadian twenty, Rake pulled out two hundred Singapore dollars, also some rupees. we we had Dylan pull out some Euros, like we had five different currencies going on the table. So I whatever you guys are betting, I will throw in Canadian currency and and take the over with Max because we are exponential we are exponential extra extrapolators here.

No. Ha ha. It was funny. Let's go. Exactly.

Yes,

Yeah.

Exactly.

The loser has to write a newsletter post of why they were wrong.

Yeah.

Okay, that's good. That's good. I like that. Well, I didn't realize we agreed to this. Like I I must have missed that in the in the Slackbird.

No, that's what it was.

Yeah. Are we doing like Frontier Lab Fantasy here? We need to pick a model for a given week and set up our team?

Yeah, we'll bet on Jeremy's spend. We bet on Jeremy's weekly spend and we'll have like an over under and get everyone involved. It's like, isn't meta building an internal like polymarket or kal-she? Like we'll do that for semi-analysis. Someone can vibe code like polymarket for betting on Jeremy's. We'll have a polymarket for betting on Jeremy's tokens.

Yeah.

Jeremy stuff.

Battle Jeremy said.

Over under.

Yeah. Dude, Mena Mena needs to shut down that effort right now, dude. Like what are they doing? That's just ridiculous. There we go.

I like it guys. There's no way that market could be manipulated.

Yeah.

no way. Totally fair.

Hey Jeremy, I've got six thousand rupees writing on you. Need you to hammer the data center model dashboard this week, buddy. Okay. Well guys, I appreciate you you taking the time. Hopefully the listeners enjoyed it. Nice devolved a little bit at the end. Let's let's all get back to work. Keep tracking those tokens.

Yeah.

Yeah.