Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Fine-tuning is the industry’s favorite blunt-force instrument. It is expensive, computationally heavy, and — more often than not — it breaks as much as it fixes. In our collective quest for more efficient model control, Activation Steering promised a surgical alternative: an inference-time “nudge” that costs nothing and changes everything.

Yet, in production, steering often feels like fighting a ghost. You push for a specific concept, and the model’s distribution leaks into generic prepositions and hallucinations. You try to steer a model to be “more professional,” and suddenly it starts obsessing over the word “the” or “to,” losing the very nuance you were trying to preserve.

The problem isn’t that steering is “weak” — it’s that we have been fundamentally miscalculating the “shape” of the space our models live in.

We tend to treat the internal representations of an AI like a flat, simple map where you can just draw a straight line from Point A to Point B. But the moment a model uses Softmax to turn raw numbers into a probability distribution, that map warps. Much like mountaineering, you want your AI to account for the curvature of your landscape to get the best outcomes (this was one the things that made Kimi’s MuonClip work so well).

The paper we are breaking down today, “The Information Geometry of Softmax: Probing and Steering”, introduces a fix called Dual Steering: “We prove that dual steering optimally modifies the target concept while minimizing changes to off-target concepts. Empirically, we find that dual steering enhances the controllability and stability of concept manipulation.”

However, instead of going deep into their algorithm (which is shared above), I want to use this research as a starting point to talk about some of the grounding concepts in the research around a model’s information geometry (how it organizes and navigates internal knowledge representations), so that you can go beyond this paper and start understanding the larger space around LLM Geometry and Activation Steering.

In this article, we’re going to walk through:

• Why Euclidean Math Lies to You: We’ll explain why treating a model like a flat grid causes “probability leakage,” and why a tiny nudge in the wrong part of the model’s “terrain” can flip the entire output in ways you didn’t intend.

• The “Two-Map” System of Softmax: You’ll learn how models actually use two different coordinate systems at the same time. One map is for “freedom” (where the raw vectors live), and the other is a “cage” (where the probabilities live). If you don’t know which map you’re using, you’ll accidentally crush the very ideas you’re trying to blend.

• The Difference Between Movement and Measurement: We will break down a common “type error” in AI research. We often try to “add” a measurement tool (like a probe) directly into a representation, which is mathematically as nonsensical as trying to physically add a thermometer to a room.

• Navigating the “Exit Nodes”: We’ll look at why this math currently only works at the final layers of a model and where the research needs to go next. This will help you make high-signal judgments on where to invest your developer attention and capital.

We’re moving away from bludgeoning models with scale and toward a more precise way of understanding the math of intelligence. I hope you’re as excited about this as I am.

Executive Highlights (TL;DR of the Article)

Our current failure to make Activation Steering — an efficient, inference-time intervention — as effective as expensive Fine-Tuning stems from a fundamental geometric “type error.” We are treating AI representation space as flat (Euclidean) when it is actually warped (Bregman).

The Euclidean Illusion vs. Bregman Reality

• The Problem: Standard steering assumes adding a vector moves a concept in a straight line without distortion. In practice, this “naive” addition causes probability leaks. Steering a model toward a specific verb might accidentally spike the probability of a random preposition like “to,” degrading the model’s overall intelligence.

• The Geometry of Softmax: Once a representation passes through a Softmax operation, Euclidean rules break. Softmax creates a Bregman geometry governed by the log-partition function A(lambda). In this space, distance isn’t a straight line; it is measured by KL Divergence.

• The “Type Error”: Researchers often treat the Linear Probe (a measurement tool/covector) as a Vector (a displacement). Adding a probe directly to a representation in the residual stream is mathematically akin to trying to “add a thermometer to a room.”

Dual Steering: A Geometric Fix

Two Coordinate Systems: Bregman geometry necessitates two systems: the Primal (lambda), which is the raw, infinite vector in the residual stream, and the Dual (phi), which is the probability-weighted “center of mass” of the model’s vocabulary.

• Primal Interpolation acts like a Logical AND, crushing unique traits to find a “safe,” generic consensus (often resulting in bland outputs).

• Dual Interpolation acts like a Logical OR, preserving the union of concepts and allowing for distinct mixtures without collapsing into prepositions.

The Solution: To steer effectively, one must map the primal vector to the dual coordinate, add the probe there, and translate back. This Dual Steering ensures a “KL Projection” — changing the target concept while minimizing the shift of the rest of the distribution.

The Bottleneck: Current math for dual steering only applies to exit nodes (where Softmax occurs, like final-token distributions or CLIP retrievals). It does not yet exist for the intermediate layers where most surgical steering is actually performed. This is a huge problem and will have to be addressed. That’s why we treat this work more as a jumping off point into the larger space as opposed to focusing most of our attention on discussing the algorithm.

Final Takeaway: we should stop trying to “bludgeon” models into submission with compute-heavy fine-tuning and start mastering the information geometry of the models themselves. Unlocking the math of how models represent knowledge is the path to efficient, precise control.

Things you might find interesting:

1. Our research on Latent Space Reasoning, which allows us to unlock new capabilities not present in the base models, without any training.

2. Our work on Fractal embeddings, which allows us to integrate a sense of hierarchy directly into embeddings, allowing for lossless architectural depth hierarchical classification.

Both streams of research are promising pointers to how exploring geometry can unlock low-cost, powerful solutions that improve the AI landscape.

PS: Personal update. A good friend of mine is hosting this event. Y’all might find it interesting to go to (I’ll be there as well, lmk if you want to come say hi).

Website: silsilasounds.org | IG: @silsilasounds

Hope I’ll see you here.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

What Does Activation Steering Actually Do to a Model’s Output Distribution?

Generally, when we want to impose a behavioral change in a model, we tend to rely on Fine-Tuning. When it works (and it never does), fine-tuning a 70B parameter model requires dataset curation, regression testing, and hundreds of GPU hours; thousands of dollars per behavioral tweak (at the end of which you’re hoping that your expensive lil trip hasn’t broken something else in the model’s ability (which it generally does)).

Activation steering, on the other hand, costs nothing. It is an inference-time intervention — a single vector addition during the forward pass. So why don’t we do it everywhere? It hasn’t had the kind of results we were expecting from it. The reason why might have been in the way we were thinking about the space our vectors occupy.

The standard approach to steering assumes the model’s representation space is Euclidean — a flat mathematical environment where adding a vector moves a concept in a straight line without distorting the rest of the distribution. Here, you find the vector direction for a concept and add it. Let’s look at an example:

• Feed Gemma “Under Al-Teta, Arsenal play…”.

• The model predicts base verbs like “haramball” or “set piece ball.”

• To steer it toward positive verbs like “exciting football,” you calculate the “exciting games” vector, magnify it (you have to gaslight the model a lot for this example) and add it to the active representation.

The naive mental model assumes probability mass shifts cleanly from the base to the exciting concept. In production, the probability leaks. At intermediate steering strengths, the preposition “to” might suddenly pull more mass than any verb in the distribution. You ask for a conjugation and the model might hallucinate a preposition. This is not unique to Language Models; we observe the same failure in vision models. Steer MetaCLIP-2 away from “cat” toward “dog,” and the top retrieval becomes an image containing both a cat and a dog. The target concept moves, but unrelated concepts get dragged with it.

This explains why steering consistently loses to fine-tuning in head-to-head benchmarks. Subspace patching creates illusions of control while actual model behavior degrades.

To reiterate, since this is an important point, standard steering commits a type error: Once a representation passes through a softmax operation, it no longer exists in a Euclidean space. Softmax creates a Bregman geometry — a space that is mathematically flat, but requires two different coordinate systems to map distances correctly. Standard steering collapses these systems into one.

The research we’re going to look at derives a fix they call dual steering, and proves that under a clean probe and a concept-factorization assumption, dual steering is the KL projection that changes the target concept while minimizing off-target distribution shift.

The exact geometry derived in the paper only applies to representations that directly parameterize a softmax distribution. This covers final-token distributions, CLIP retrievals, and attention layers. It does NOT cover arbitrary intermediate layers deep inside the network. Most production steering targets those intermediate layers to intercept concepts before they propagate. The math for Bregman geometry at intermediate layers does not yet exist. So, as we get into this research, consider this more the start of an interesting discussion as opposed to the final say. My hope is that by surfacing this research (and others like this) we can encourage more contributors in our open source community to start exploring the geometry/math of AI, instead of purely looking at the standard axes of scale, tweaking, and model tuning.

To explore this research in-depth, we must first ask ourselves a very fundamental question — 

How Does a Representation Become a Distribution, and What Should “Close” Mean?

When we want to know if two representations are “close,” we naturally measure the straight-line distance between their vectors. Why? Because torch.dist() is easy to type, and we like to pretend AI happens in a clean, flat space (the Euclidean assumption we just about).

But as we saw in the last section, Euclidean distance lies to you. A microscopic nudge near a decision boundary flips the entire output, while a massive shove when the model is 99% confident does absolutely nothing.

The model doesn’t care about the geometric distance. It cares about the output distribution. To define “close” correctly, we have to look under the hood at how a vector actually becomes a distribution.

The model holds an active representation vector for the current context — let’s call it lambda (personally, I don’t love the use of Greek letters, but I’m keeping it here since most of the literature uses them). It also holds a massive lookup table of “unembedding” vectors for every possible token in its vocabulary — let’s call those gamma_y. To score a specific token, the model calculates the dot product of lambda and gamma_y.

Why a dot product? Because a dot product is fundamentally a measure of directional alignment. It asks the math: “How much does our current context vector point in the exact same direction as the ‘exciting football’ vector?” High alignment means a high raw score.

But these raw scores (logits) aren’t probabilities. To get probabilities, the model shoves them through a Softmax function. Softmax does two things

• It exponentiates the scores

• Then it divides by the sum of all the exponentiated scores so everything equals 100%.

Exponentiation is a bloodbath. Let’s say “haramball” scores a 10, “set piece” scores an 8, and “exciting” scores a 3. In raw score space, 3 is behind 10, but it’s in the same zip code. Once you exponentiate them (e¹⁰ vs e³), “haramball” shoots to roughly 22,000. “Exciting” is sitting at 20. Divide by the total, and “haramball” owns 88% of the probability mass. “Exciting” gets 0.08%. Softmax takes a mild preference and turns it into a blowout.

This extreme sharpness is exactly where standard mathematical tools break down — specifically, the covariance matrix.

All of this chaos is controlled by the denominator in that Softmax equation — the normalizer that divides everything. Because we usually work in log-space to keep our GPUs from throwing underflow errors, we take the log of that massive sum. This term is so fundamental it gets its own name: the log-partition function, written as A(lambda).

The actual probability of a token is just its exponentiated score minus this A(lambda) function. A handles all the normalization. This creates a very interesting outcome: the geometry of this space, the duality we need, and the steering fix we are about to build — all of it is hiding inside the derivatives of A(lambda).

To see why, we need a way to measure how different two softmax distributions are. And that leads us to our next point of exploration…

Why Does KL Divergence Measure the Change We Actually Care About?

We just established that the log-partition function A(lambda) controls the shape of our representation space. But before we can use it to fix our steering vectors, we have to solve a more immediate problem: how do we measure distance on this new terrain?

If Euclidean distance is a lie (yet another reason to not trust the Greeks)— if it treats a massive shove at 99% confidence exactly the same as a tiny nudge at a decision boundary — then what is the truth? We need a function that takes two probability distributions, compares them, and returns a single number representing how different they actually are in practice.

Our hero comes from Information Theoery: Kullback-Leibler (KL) divergence. If the true distribution is P, but your model assumes the distribution is Q, KL(P || Q) is the exact mathematical cost of that error. It calculates how surprised you will be when reality actually happens.

The formula is a sum over all possible outcomes x: P(x) * log(P(x) / Q(x)).

There are three moving parts here, and they each do a specific job to keep the math grounded in reality:

1. The Ratio: P(x) / Q(x) For any token x, how much more likely is it under the true distribution (P) than your steered distribution (Q)? If they agree, the ratio is 1. If P thinks the token is a sure thing and Q thinks it’s impossible, the ratio explodes.

2. The Logarithm Log converts multiplicative ratios into additive scores. A ratio of 1 (total agreement) maps to exactly zero. But more importantly, log makes KL sensitive to proportional changes, not absolute ones. A token dropping from 50% to 40% is only a 1.25x change. A token dropping from 0.1% to 0.01% is a 10x change. Log mathematically enforces the rule that relative probability determines model behavior.

3. The Weighting: P(x) The whole thing is multiplied by P(x) before summing. This is the “Do I actually care?” filter. KL only punishes disagreements where the true distribution P actually puts probability mass. If P thinks a token is garbage (P(x) is near 0), that term vanishes. What about Qs thoughts? Respectfully, who gives us a fuck what a grunt like Q thinks when a baller like P has already made up its mind.

Putting everything together, KL(P || Q) asks: “If P is the absolute truth, how surprised would you be if you had to navigate the world using Q?”

This weighting creates a profound asymmetry. KL(P || Q) does not equal KL(Q || P). Being wrong about P when Q is true costs a different amount than being wrong about Q when P is true. In the Euclidean space, the distance from New York to London is the same as London to New York. In information space, the penalty depends entirely on which distribution is actually generating your data. This asymmetry is exactly what causes forward and reverse KL to produce entirely different behaviors (which becomes the crucial AND-vs-OR distinction when we talk about interpolation later).

Why does this matter? Let’s dig into this math just a wee bit more. If we do this step-by-step, the geometry of the entire model falls out of the algebra.

What Happens When You Compute KL Between Two Softmax Distributions?

First, let’s put the pieces back on the board so we don’t lose track of what we are building:

• lambda is our original, unsteered context vector.

• lambda-prime is the steered vector (after we add our behavioral tweak).

• gamma_y is the unembedding vector for a specific token (the dictionary definition the model checks against).

• A(lambda) is the log-partition function — the brutal normalizer that forces everything to sum to 100%.

The KL formula asks us to compute a ratio for every token: log [ P(y | lambda) / P(y | lambda-prime) ].

In plain English: take the logarithm of the true probability divided by the steered probability.

How do we calculate those probabilities? Remember from the previous section: the probability of a token is just exp(score - normalizer).

Because we are taking the logarithm of an exponentiated number, the math simplifies beautifully. The log simply deletes the exp, leaving only the raw terms inside. Division inside a logarithm becomes subtraction outside.

So, taking the log of that probability ratio strips the math down to just the raw scores and the normalizers. For the top part of the fraction (the original state), we get: lambda * gamma_y - A(lambda)

For the bottom part (the steered state), we subtract it: minus [ lambda-prime * gamma_y - A(lambda-prime) ]

If you group the similar terms together, the log ratio becomes a clean, linear equation: (lambda - lambda-prime) * gamma_y + A(lambda-prime) - A(lambda)

Now, the final step of the KL divergence formula tells us to multiply that result by the true probability P(y | lambda) and sum it up over every token in the vocabulary.

When you do that, something fascinating happens to that gamma_y term. You end up calculating the sum of P(y | lambda) * gamma_y.

Stop and think about what that is physically. You are taking every single token vector in the model, weighting it by how likely that token is to be generated, and averaging them all together. It is the probability-weighted center of gravity for the model’s current state.

As it turns out (and we will prove exactly why in the next section), this center of gravity is exactly the mathematical gradient of our log-partition function A. Let’s just call it grad-A(lambda).

If we substitute that gradient back into our equation, the final KL formula reveals itself:

KL = A(lambda-prime) - A(lambda) - grad-A(lambda) * (lambda-prime - lambda)

Look at the physical architecture of this final result.

A(lambda-prime) - A(lambda) is exactly how much the normalizer actually changed when we steered the vector.

grad-A(lambda) * (lambda-prime - lambda) is how much a straight, flat tangent line predicted the normalizer would change.

The KL divergence is literally the gap between the true change and the linear approximation.

Why is this gap always positive? Because the normalizer function A is convex — it curves upward like a bowl. For any convex function, a straight tangent line will always sit below the curve. The true curve always bends up and overshoots the straight-line prediction. The gap only hits zero if the two vectors are exactly the same.

This gap — the error between a convex function and its linear approximation — has a formal mathematical name. It is a Bregman divergence.

KL(P || Q) between two softmax distributions isn’t like a Bregman divergence. It is the Bregman divergence induced by the log-partition function A.

Nobody chose this. The algebra forced it. It means Euclidean geometry is not the natural default once a representation passes through a Softmax distribution. Every LLM, every CLIP model, and every attention layer is living and breathing in a Bregman geometry that most researchers have never explicitly mapped out. We’ve been using Euclidean wrenches on Bregman bolts since 2017.

Information geometers have studied Bregman divergences since the 1980s. And the very first thing their toolkit tells you about a space governed by Bregman geometry is this: the representation space does not have one natural coordinate system. It has two. This has some very juicy implications.

What Are the Two Coordinate Systems, and Why Does the Duality Matter?

The first coordinate system is the one everyone already uses: lambda. This is the raw representation vector sitting in the residual stream. Let’s call it the primal coordinate.

The primal space is the Wild West. It is entirely unconstrained. You can take your lambda vector, multiply it by a million, and point it absolutely anywhere in that 4,096-dimensional space. The model won’t crash. Softmax will just take those massive numbers and turn the output into a brutal step-function where one token gets 99.999% of the mass. The Primal Space has that good ol’ Murican freedom, baby.

The second coordinate system is completely different. It comes directly from that gradient term we isolated earlier: the gradient of our log-partition function. Let’s call this new coordinate phi.

phi = grad-A(lambda)

Let’s walk through the actual derivative (in case you don’t remember, derivatives give us the rate of change of something with respect to something else) to see what phi is physically made of. Don’t skip this, because it is the most elegant piece of math in the entire architecture.

Remember that our normalizer function is A(lambda) = log [ sum of all exp(scores) ].

To find the gradient, we take the derivative. The chain rule in calculus tells us that the derivative of a logarithm is simply 1 / x multiplied by the derivative of whatever is inside the log.

• The bottom of our fraction (the x) becomes the inside of the log: the sum of all the exponentiated scores.

• The top of our fraction becomes the derivative of those scores. If a token’s score is lambda * gamma_y, its derivative with respect to lambda is just the token vector itself: gamma_y.

So, for any given token, the gradient gives us this exact fraction: exp(score) / [sum of all exp(scores)] ... multiplied by the token vector gamma_y.

Look very closely at the left side of that multiplication. That fraction is literally the exact formula for Softmax probability.

The algebra just handed us a massive gift. The gradient of the log-partition function is simply every single token vector in the model, multiplied by its Softmax probability, and added together.

phi = sum over y of [ P(y | lambda) * gamma_y ]

This is our dual coordinate. Physically, it is the probability-weighted center of mass for the entire vocabulary. If the model assigns 70% probability to “maintains”, 20% to “operates”, and 10% to random noise, then your dual coordinate (phi) sits exactly at 0.7 * (maintains) + 0.2 * (operates) + 0.1 * (noise). It is a physical coordinate telling you exactly where the model’s attention is currently hovering across the dictionary.

Because of how Bregman geometry works, lambda and phi are just two views of the exact same distribution. If you have the primal vector, you can calculate the dual center of mass. If you have the dual center of mass, you can reverse-engineer the primal vector.

But there is a massive physical asymmetry between them. We already established that the primal space (lambda) is infinite. The dual space (phi) is trapped in a cage.

Why? Because phi is built by multiplying token vectors by probabilities. Probabilities obey strict laws: they can never be negative, and they must add up to exactly 100%. Because of this, phi can never step outside the boundary drawn around your vocabulary. Imagine stretching a massive mathematical rubber band around every single token vector in the model’s embedding space. That rubber band is called a convex hull. You can move phi anywhere inside the hull by mixing different token probabilities, but you can never push it outside. If you try to steer the dual coordinate outside that hull, the math shatters, because no valid probability distribution could ever put you there.

What Do the Two Coordinate Systems Mean Semantically?

Why do we care that there are two systems? Because they answer the most basic question in geometry completely differently: how do you draw a straight line? This might seem like a silly questions, but drawing a straight line between two concepts is how we blend them. When you want to combine two ideas, you take their representations and find the midpoint. But on a warped Bregman surface, your midpoint completely depends on which coordinate system you use to draw the line.

Let’s look at a concrete example. You have two representations.

• Vector 0 is the model’s state after reading: “Q: What is the capital of France? A: It is” (Probability sits heavily on the token “Paris”).

• Vector 1 is the state after reading: “Q: What is the capital of Germany? A: It is” (Probability sits heavily on “Berlin”).

You want to find the exact 50/50 midpoint between these two concepts.

If you do a primal interpolation, you draw a straight line between the two raw lambda vectors in the unconstrained residual stream. Because of how the Bregman algebra shakes out, moving in a straight line in primal space mathematically forces the model to minimize the reverse KL divergence.

Think back to our KL section. Reverse KL is the “Do Not Hallucinate” penalty. It looks at the target distribution and says, “If the target thinks a token is garbage, you will pay a massive penalty for putting probability mass there.”

Look at what happens to the math when you stand at the primal midpoint. The France endpoint looks at the word “Berlin.” It sees that “Berlin” has near-zero probability in the France distribution, so it slaps the model with a massive penalty for including it. Simultaneously, the Germany endpoint looks at the word “Paris,” sees near-zero probability, and slaps the model with a massive penalty for including it.

What survives? Only the tokens that both endpoints agree are mathematically harmless — generic words like “The”, “is”, or “called”. Primal interpolation operates exactly like a logical AND. It crushes everything that makes a context unique and leaves only the safest possible intersection.

If you do a dual interpolation, you draw a straight line between the two phi vectors (the centers of mass) inside the convex hull. Moving in a straight line in dual space minimizes the forward KL divergence.

Forward KL operates under the exact opposite philosophy. It is the “Do Not Forget” penalty. It says, “If the target thinks a token is highly likely, you will pay a massive penalty if you fail to cover it.”

Look at the midpoint now. The endpoints are no longer allowed to veto each other. France demands you keep “Paris”. Germany demands you keep “Berlin”. Instead of crushing them, dual interpolation forces the model into a compromise. It creates a mixed probability distribution that holds both truths simultaneously, allocating roughly 50% mass to Paris and 50% mass to Berlin.

In other words, Dual interpolation operates exactly like a logical OR. It preserves the union of the two concepts.

This behavior is a fundamental law of any model that uses Softmax. If you take a vision model like CLIP and try to interpolate the concept of a “black dog” with a “white dog”, you get the exact same split.

• Primal interpolation (AND logic) searches for shared traits. The colors fight each other, the model panics, and it spits out a single dog with black and white spots.

• Dual interpolation (OR logic) holds both truths. It spits out an image that literally contains two distinct dogs — one black, one white.

Primal finds the boring consensus. Dual preserves the contradiction. Two completely different philosophies of blending concepts, arising purely from which ruler you picked up.

This is why the geometry actually matters for your pipeline. When you blindly subtract two vectors in the residual stream to build a steering direction, you aren’t just doing niche math. You are accidentally making a product decision. By calculating your vector in the raw residual stream, you are locking yourself into the primal coordinate system. You are forcing the model into that destructive AND logic. You are explicitly telling the model to crush anything unique about the prompt and only keep the safe consensus. This is exactly why standard steering leaks probability mass to generic prepositions like “to” — it’s abandoning specific concepts to find the mathematical middle ground.

If you actually want to add a behavioral concept without destroying the original context — if you want the OR logic — you cannot just add vectors in the residual stream. You have to translate the vectors, do the addition in the dual space, and translate them back.

Isn’t exploring the math of intelligence so much cooler than being a training grunt? Imagine learning about some of the coolest topics in the world only to be forced to debug GPU crashes and benchmark tests 24/7.

Interpolation shows that the two coordinate systems produce different behaviors when you blend representations. Steering is a related but sharper operation: instead of blending two representations, you’re modifying one to change a specific concept. The question is the same — which coordinate system are you operating in? — but the stakes are higher, because steering with a probe means adding a specific mathematical object to the representation. What kind of object the probe is determines which coordinate system it belongs to.

What Kind of Mathematical Object Is a Linear Probe?

Before we fix the steering math, we have to look closely at the tool we are using to steer: the linear probe.

When you train a linear probe to detect a concept — say, “third-person verb” — you are building a very specific mathematical object. It takes your 4,096-dimensional representation vector (lambda), runs a dot product against its own weights (beta_W), and spits out a single scalar score.

In linear algebra, an object that eats vectors and spits out scalars is a linear functional. A covector. Covectors live exclusively in the dual space.

Most of us learned the difference between vectors and covectors in undergrad and immediately dumped it from RAM. Why? Because if you live in flat Euclidean space, you don’t need to care. Flat space lets you cheat. You can add them, subtract them, and treat a covector exactly like a normal vector.

But they are physically different objects.

A vector is a displacement — a physical direction you can move. A covector is a measurement — a tool that assigns numbers to states. Think of a thermometer. A thermometer measures a room and returns a temperature. The thermometer itself is not a room. You cannot take a physical location and mathematically “add” a thermometer to it.

In the warped, twisted Bregman geometry of Neural Network Information Spaces, we can’t get away with conflating them. Technically, we can, but that is why so much of the activation steering research has sucked for so long and all of the attention has been to the fine-tuning tards.

What Does Standard Activation Steering Get Horribly Wrong?

Look at the formula the entire open-source community currently uses for activation steering: lambda_t = lambda_0 + t * beta_W

Take the original context vector (lambda_0), add the probe (beta_W) multiplied by some steering strength (t).

As an array operation, it runs perfectly. PyTorch will execute the addition without throwing a warning because both objects are just float32 arrays of the same shape. But PyTorch doesn’t know geometry.

As a geometric operation, this equation is a disaster. It blindly mashes a measurement tool (the probe) into a physical displacement (the representation). Because Softmax forces the model into Bregman space, vectors and covectors are not interchangeable.

But Dev Dev, you can say it sucks, but I don’t understand why. What does this type of error cost you in production?

When standard steering executes that addition, it is mathematically asking the model to find the closest point that satisfies the new concept. But because it used Euclidean math, which we already proved has zero relationship to the model’s actual output distribution. Put two and two together and we see that — that gap (the distance between the Euclidean guess and the true Bregman reality ) is exactly where your probability leaks. It is the physical reason the preposition “to” steals all the mass from your verbs. It is the reason your vision model spits out a dog with black and white spots instead of two distinct dogs.

The fix is one line of math.

phi(lambda_t) = phi(lambda_0) + t * beta_W

Map your primal vector to the dual coordinate system (phi). Add the probe (beta_W) in the dual space, where measurement tools actually belong. Then translate the result back to primal space to hand off to the next layer.

So what does this actually accomplish? A lot, actually.

The Payoff: What Actually Happens in Production?

What happens to Gemma-3–4B?

The hallucination dies. When you run the dual steering vector to change a base verb to a third-person verb, the probability mass shifts directly from “maintain” to “maintains”. That generic preposition “to” that spiked out of nowhere in the primal space? It stays completely flat. The probability leak is sealed.

What happens to the vision models?

Steer MetaCLIP-2 away from “cat” toward “dog” in dual space, and the model directly transfers probability to images of dogs. The off-target “cat and dog sitting together” image never spikes.

Why Does Dual Steering Actually Work?

When you steer in the primal space, you are just adding raw numbers to the model’s logits. Softmax then exponentiates those inflated numbers. Because exponentiation amplifies differences, the artificially massive logits crush the model’s original context. To make everything sum to 100%, the model abandons the specific context and falls back on the safest, most frequent tokens it knows. As with most things in life, an overabundance of playing it safe leads to bland, generic output (tokens like to).

The dual space (phi) operates under different laws. Because it is built entirely from probabilities, it is strictly zero-sum. The total mass is locked at exactly 100%. You cannot inject raw, unconstrained numbers here.

If you want to push the center of mass toward the third-person verb “maintains”, the math forces a direct trade. This trade is the physical reality of a “KL projection.” The projection takes your target concept and solves for a new probability distribution using two strict rules: the new distribution must satisfy your steering target, and it must change the original probabilities as little as physically possible.

To satisfy both rules, it steals the required probability mass directly from the base verb (“maintain”) and hands it to the target verb (“maintains”). The generic token “to” stays flat because the distribution was never shattered.

That’s not to say this is without flaws. Dual steering fixes the geometric type error, but it hits a hard wall in production: it only works at the exit nodes.

The Bregman geometry derived by Park et al. relies entirely on the log-partition function. That means it only applies to representations passing directly through a Softmax distribution. In modern architectures, that restricts you to the final-token unembedding layer, CLIP retrievals, and attention matrices.

In production, almost nobody steers at the final layer. By the time a concept hits the unembedding matrix, the model has already made its decision. Surgical steering happens deep inside the network — say, layer 15 of a 32-layer model — to intercept a concept before it fully forms. But intermediate layers don’t have a Softmax attached to them. They are just unconstrained states floating in the residual stream. For intermediate layers, the mathematical map of Bregman geometry simply stops.

So…. we can never really apply the math we just talked about to hit real-time steering. That is a slight problem.

So where does this leave us? Do we just retard-max and let fine-tuning handle all the work? Let’s bring our little exploration to a close.

Conclusion: Where Does Dual Steering Go From Here.

Let’s end our journey with a trip down memory lane, back to the days of classic ML. Think about how LLMs changed production. We used to build massive, brittle scaffolding — wiring ten specialized ML classifiers to two generators just to complete one basic workflow. LLMs replaced all of that with a single general-purpose model, collapsing infrastructure costs overnight. That drop in cost didn’t happen because we optimized the classifiers. It happened because we completely changed the interaction pattern.

We are at the exact same threshold with model internals.

Right now, the industry’s default reflex for bad model behavior is to treat it as an infrastructure problem. When a model fails, we throw compute at it. We run brittle fine-tuning jobs and try to bludgeon the network into submission using scale.

But as more research is showing, the model layer might be the wrong layer to solve these problems. Perhaps the true solution is deeper. By changing how the space of how we represent knowledge in the language models, and then how we navigate it, we might be able to unlock capabilities that our current paradigm deems unrealistic.

And even if it doesn’t work, isn’t the idea so much more fun? Do you really want to spend the rest of your career profiling GPUs, fiddling with random seeds, and writing 20 skills/agent templates so that your manager can show your teams AI readiness? Does your heart (and brain) not ache to do more, to push humanity’s knowledge forward?

Think that over.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

I spoke to Ryan Estes about Legal AI, Open Source Research, and why access to Legal Services needs to be more accessible over here. The conversation was very well received, so I’m sharing it here with Ryan’s permission.

I hope you enjoy it.

Companion Guide to the Livestream:

This guide expands the core ideas and structures them for deeper reflection. Watch the full stream for tone, nuance, and side-commentary.

1. The Three Hallucinations Hiding Under One Word

The Event — I broke hallucinations into three categories on the stream. Category one: the AI makes up a case that doesn’t exist. Category two: the case exists and the quote is real, but it’s from the wrong jurisdiction or doesn’t apply to your domain. Category three: the AI gives you an argument that looks correct on its own, but somewhere else in your documents there’s something that contradicts it, and the system never connected the two.

Why this matters — Everyone talks about category one because it’s the most obvious. A lawyer cites a fake case, gets sanctioned, it makes the news. But category one is also the easiest to fix. You just check whether the case exists. A basic validator catches almost all of it. Categories two and three are where the real damage happens, and they’re much harder to catch. The lawyer reads the brief, everything looks right, the citation is real, the quote checks out. They file. Then opposing counsel tears them apart because the cited authority was overturned in their jurisdiction, or because a deposition transcript on page 723 contradicts the whole argument.

Most legal AI tools are RAG wrappers. You upload documents, the system cuts them into chunks, turns the chunks into vectors, and when you ask a question it finds the chunks that look most similar to your question. This works fine for simple retrieval like “find the indemnification clause.” It does not work for “is this argument actually supported across all my documents.” Cosine similarity doesn’t know what a contradiction is. It doesn’t know about jurisdictions or whether a ruling is still valid. Two laws from different states will sit right next to each other in vector space even if they say opposite things. And “document 47 invalidates the claim in document 12” is a logical relationship that vector search can’t represent at all.

Every category-two and category-three hallucination is a potential malpractice claim. The tool makes you faster, you trust it, you file work that has buried contradictions, and the first time it costs a client real money your insurance situation changes permanently. The speed improvement means nothing if the work product carries hidden liability. This is why legal AI has to move to architectures that handle context across documents natively. The wrappers will be fine for boilerplate. They’ll fail at everything that actually matters.

2. Why Irys Doesn’t Wrap, It Rebuilds

The Event — Ryan asked whether Irys’s “infinite context” works the same way Harvey’s does. It doesn’t. Harvey chunks your documents and uses vector similarity to find relevant pieces. Irys builds a knowledge graph that links entities, propositions, assertions, and contradictions across all your documents, and updates that graph every time you add a new file. When you ask a question, the system walks the graph instead of doing similarity search.

Why this matters — Vector search became the default approach to “my documents don’t fit in the context window” because it was cheap and it worked for simple use cases. Then people treated it as a permanent solution. It was always limited. Chunks are independent of each other. There’s no cross-document reasoning. There’s no way to bind a claim in one document to evidence in another. None of this mattered when the use case was “summarize this PDF.” It matters a lot when the use case is “build me a litigation strategy across 100,000 pages.” I was writing about these structural limits in 2022, before RAG was even a common term.

The fix isn’t bigger context windows or better embeddings. The fix is a system that explicitly tracks entities like parties, dates, jurisdictions, and claims, that maintains contradiction edges between propositions, and that reorganizes itself when new documents arrive. That’s how you catch “document 47 contradicts document 12” before the LLM ever starts drafting. That’s what Irys does. And this is why the wrappers can’t close the gap by adding features. Their entire stack assumes chunks are independent and retrieval is similarity. To move to graph-native context, they’d have to throw it all away and start over.

3. Why GitHub Copilot Couldn’t Become Cursor, And Cursor Couldn’t Become Claude Code

The Event — Ryan asked what stops someone from copying Irys. This is the answer Harvard Business School reached out about for one of their courses. Software doesn’t just have features. It has assumptions baked into it about what the user controls, what the AI controls, where data lives, what gets automated, and what gets left to human judgment. GitHub Copilot had Microsoft’s distribution and money. It couldn’t become Cursor. Cursor had the developer tool category to itself. It couldn’t become Claude Code. Each product was built around a different set of assumptions, and those assumptions aren’t portable. They’re in every layer of the code and they compound with every release.

Why this matters — When founders talk about moats they usually talk about data, brand, and distribution. Those are visible from the outside. Architectural assumptions are not. If you build assuming the AI is an autocomplete assistant, you get Copilot. If you build assuming the AI is an autonomous agent that the user occasionally interrupts, you get Claude Code. The surface features can look the same but the products are completely different underneath. You can’t retrofit one into the other because every API, every state machine, every piece of the UX was built on the original assumption. Unwinding it costs more than starting over.

This is the moat for Irys. We assumed in 2022 that vector search would not be enough for legal context. So every layer of the system was built around graph-based context aggregation. Harvey assumed RAG was enough. They have a year of rebuilding before they can even start the conversation we finished three years ago. And by the time they’re done, we’ve shipped two more iterations on top.

There’s a second moat on the product side. The more you use Irys, the more it learns how you work. It doesn’t train on your data. But it learns which argument structures you prefer, what memo formats you use, how you weigh jurisdictions. All of that accumulates in your private workspace. If you leave, you lose all of that embedded knowledge and have to rebuild it somewhere else from scratch.

4. Why Open Source Is Cheaper Than Marketing

The Event — Irys is free to sign up. We’ve open-sourced major pieces of our reasoning infrastructure, including a lightweight version of the latent space reasoning engine. By normal startup logic, this makes no sense. We have a defensible product, real funding, well-funded competitors, and we’re giving away the technical work for free.

Why this matters — The usual way to think about open source is as a cost. Every piece of IP you publish saves your competitor some R&D time. That’s true if the game is static. In a fast-moving technical field, the thing that actually matters is who has access to information about what’s coming next. Who’s in the room when the labs are deciding the next generation of model capabilities. Who knows what’s shipping in six months.

Published work is what buys access to those rooms. NVIDIA’s senior engineers don’t take meetings with random startups. They take meetings with people whose technical work they’ve already seen. DeepMind doesn’t share roadmap previews with companies that haven’t contributed anything back to the field. The newsletter and the open-source reasoning engine aren’t marketing. They’re what get us the partnerships with the labs. That’s how we know what’s coming before it’s announced, and that’s how we make architecture decisions that our competitors don’t know they need to make yet.

The competitor who copies our open source saves maybe a quarter of engineering time. But they’re still nine months behind on the decisions that actually matter because they don’t have the relationships that tell them where the field is going. We didn’t lose value by publishing. We traded code for visibility into the technical horizon.

There’s also a recruiting benefit. The engineers who read the reasoning paper, run the lightweight implementation, and reach out about it are exactly the kind of people you can’t find through normal hiring channels. You can’t buy that pipeline. You can only earn it.

5. The Newsletter Is Peer Recruitment, Not Customer Acquisition

The Event — Ryan assumed the Chocolate Milk Cult newsletter (250,000+ subscribers, about 1.5M monthly reach) was the lead generation engine for Irys. I corrected him. Lawyers don’t read deep dives on quantization math or GPU pricing curves. The newsletter doesn’t sell legal seats. It serves a completely different purpose.

Why this matters — Most founder content advice treats your audience as a sales channel. That works when your audience and your customer are the same person. A fitness creator selling a fitness app. A finance creator selling a finance tool. When your audience is a different group from your buyer, the sales-channel framing leads you to measure the wrong things. You track conversion and click-through, you report the wrong wins, and eventually you conclude the audience isn’t worth the effort because the numbers don’t show a return.

What the audience actually does, when it’s decoupled from the customer base, is give you information you can’t get any other way. Lab researchers DM me about unpublished work. Engineers send me preprints. Investors share data they wouldn’t put in a pitch deck. None of that shows up as a Substack metric, but all of it changes what Irys ships next quarter.

The point for founders is simple. Be honest about who your audience is and what they’re actually for. If you’re building vertical SaaS for accountants, an audience of ML researchers won’t sell seats. But it might give you a technical edge that makes the product worth buying when your actual sales channel starts working. The two things serve different purposes and you can’t optimize both with the same content.

6. The Eat-Shit Theorem of Legal Access

The Event — Ryan asked why I picked legal when I could have pointed the technical foundation at any industry. The answer is the moral case. India has a ten-year backlog on civil cases. If your employer steals your wages and you go to court tomorrow, you won’t get a hearing for a decade. In NYC, tenants have landlords who let buildings rot and overcharge rent, and they can’t do anything because they can’t afford a lawyer. Indian farmers get oversold pesticides, fall into debt, and end up dealing with loan sharks. I get hit with baseless defamation suits over newsletter coverage, designed not to win but to bleed me on legal fees. The pattern is always the same. The legal system is gated by money, and ordinary people pay the price.

Why this matters — Democratization here is not a marketing word. It’s the reason every other decision gets made the way it does. Irys is free to sign up because if we charged what the market would bear, the people who need access most would never get in. We open-source the reasoning infrastructure because the category needs to advance whether we win or not. We’re building Irys Lite because the full product doesn’t reach a wage-theft case in rural India or a tenant in the Bronx.

For founders, the closing point from the stream is the one I’d keep. Be honest with yourself about whether you’re building a business or a mission. Both are fine. The question is which one survives the next downturn, the customer churn, the eighteen months where nothing works. If it’s a business, say so to yourself, your team, and your investors. Don’t dress it up as world-changing because that confusion is what burns founders out around year three. If it’s a mission, then it has to be the thing that’s still motivating you when a competitor with twenty times your funding announces the same product. The mission is whatever’s still there when it stops being fun. Everything else in the company is downstream of that.

Full conversation with Ryan is on AI for Founders, here. Try the platform free at iqidis.ai. The open-sourced latent space reasoning work is on the Chocolate Milk Cult archives. If your attorney is still billing you in fifteen-minute increments for things a knowledge graph does in fifteen seconds, send them this guide.

Subscribe to support AI Made Simple and help us deliver more quality information to you-

Subscribe now

Flexible pricing available—pay what matches your budget here.

Thank you for being here, and I hope you have a wonderful day.

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Last year, I compared the major AI web apps: ChatGPT, Gemini, and Claude. The question was simple: which one was worth paying for?

At the time, the answer was also simple. ChatGPT was the best general-purpose app. Gemini had strong models trapped inside a messy Google product. Claude had a good underlying model, but the app was too limited, too forgetful, and too frustrating for sustained work.

That rubric is now old. The major AI subscriptions are no longer just chatbots. They are memory systems, research tools, coding agents, file processors, image generators, project workspaces, and billing systems. The model still matters, obviously. But the model is no longer the whole product.

So I’m redoing our comparison with an updated question: Which subscription will let you get the most done for the least cost? I’m keeping the comparisons limited to Gemini, OpenAI, and Anthropic for 3 reasons:

1. Performance: They still have the best models available for use.

2. Reliability: All 3 tend to have much higher uptime and usage.

3. Paperwork: Some of the subscriptions (like the Chinese models) are typically not approved in most orgs and require a longer review process. By the time it is approved, we will likely see 10 new model releases and updates.

For the ranking, we will review them over the following points:

• workflow coverage. Can the subscription handle writing, research, coding, files, images, planning, review, and daily work without forcing you into three different products?

• memory and state. Does the system remember useful context without becoming weird? Good memory helps. Bad memory overfits to stale preferences. Additionally, there is ease of use/effortlessness (Anthropic natively looks at memories, Gemini needs to be prompted explicitly) and the cost (Anthropic burns a lot of tokens on memory)

• instruction stability. Can it follow complex instructions across long, messy work? Not one prompt. That is easy. The real test is whether it preserves constraints after ten turns, two corrections, and a change in direction.

• judgment and taste. Does it know what to cut? Does it know when the technically correct answer is still bad? Does it update proportionally when you push back?

• research quality. Search is not research. A good research workflow finds sources, weighs them, notices conflicts, and tells you what remains uncertain.

• coding and agentic execution. This is now central. Codex and Claude Code are not side features. They are main reasons to pay.

• output quality. Can it produce writing, visuals, reports, artifacts, and polished work that is actually usable?

• economics and billing trust. Can you predict what you are paying for? Does the plan scale with heavy use? Does the product punish you for doing real work?

• product friction. Does the app fight you? Does it hide limits? Does it make you restart? Does the best model live somewhere other than the product you are paying for?

• Ecosystem: Does it work well with other tools in the ecosystem?

Let’s see where you should be spending your money.

To access the full article—and all premium breakdowns going forward/written prior—upgrade to a premium subscription below.

Subscribe now

If you believe deep insight deserves support, become a premium subscriber to allow me to keep doing the same.

Flexible pricing available—pay what matches your budget here.

Most companies offer learning or professional development budgets. You can expense this subscription using the email template linked here.

Read more

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Most of the AI infrastructure boom isn’t actually buying compute. It’s buying memory bandwidth.

Look at the market right now: SK Hynix and Micron are completely sold out of their 2026 HBM capacity. Cerebras is pushing a $22 billion IPO built almost entirely around high-memory inference chips. Google just split its 8th-generation TPUs into completely separate training and inference lines.

Hundreds of billions of dollars are being deployed based on a single, massive assumption: that AI generation will always be bottlenecked by the exact same things it is today.

But if you look closely at the research, that assumption is already starting to rot. Diffusion models are rapidly rising out of the image-generation niche and seeing aggressive, mainstream use across text and reasoning workloads.

(PS: if you want to dig into why Diffusion Models are the future, from an engineering/technical perspective (including why companies are already investing in them, we covered it in this article (Prediction 2 and 3). Since I don’t want to repeat stuff, this article will take the rise of Diffusion Models as a given. The image below is a good illustration of how diffusion models unlock a deeper category of reasoning. .

This is not a trivial algorithmic swap. Diffusion fundamentally alters the AI paradigm. It aggressively rewrites the underlying math of how models are served, flipping the hardware bottlenecks and turning the established inference market on its head.

In this article, we will cover:

• The Dependency Graph: Why standard autoregressive LLMs are structurally trapped by memory constraints.

• The Bottleneck Inversion: How diffusion models shatter the sequential lock and shift the hardware paradigm.

• The Algorithmic Timeline: The math required to make language diffusion financially viable.

• Inference as Search: How diffusion makes complex, inference-time verification cheap, threatening the $2 billion base-model moat.

• The Vendor Breakdown: Which hardware architectures (NVIDIA, AMD, Groq, Apple, etc.) are actually positioned to survive the shift to compound diffusion workloads.

Let’s play together.

Executive Highlights (TL;DR of the Article)

• The Autoregressive Memory Trap (Section 1): The entire AI infrastructure supercycle is a desperately expensive patch for data movement. Because autoregressive models generate sequentially, operations have an arithmetic intensity of roughly 1 FLOP per byte. A modern tensor core needs nearly 300 FLOPs to stay fed. This means your $40,000 GPU executes at under 1% of peak compute while waiting for gigabytes of history (the KV cache) to stream across the chip.

• The Bottleneck Inversion (Sections 2 & 3): Diffusion models refuse left-to-right generation, refining entire sequences in parallel. This changes the underlying math from bandwidth-starved matrix-vector operations to massive matrix-matrix multiplications. The KV cache tax vanishes, and the bottleneck flips to raw compute. This is a massive tailwind: hardware generations like Blackwell and Rubin have compounded FLOPs at 3x every two years, while memory bandwidth has stalled. The industry accidentally built the perfect silicon for diffusion workloads.

• The Algorithmic Timeline & Step-Count Constraints (Section 4): Perfect hardware is irrelevant if the math requires too many steps. Image generation collapsed its step count using continuous math, but language operates in discrete jumps, trapping text diffusion in a 4-to-16 step regime. However, software scheduling is proving cheaper than raw parameter scale. Simply forcing a model to unmask logic premises before conclusions (LogicDiff) spiked reasoning scores by near 40 points without changing base parameters, threatening the hyperscaler gigawatt-compute thesis.

• Inference as Search & The Verifier Moat (Section 5): Because early denoising steps just build coarse shapes, multiple search trajectories can share the exact same early path. Running 4 candidate branches that share 40 of 50 steps yields a 4x quality search for a 1.6x compute tax. Branching AR is financially ruinous; branching diffusion is cheap. This unbundles the AI market: value migrates from the $2B base model to the companies building elite, proprietary verifier suites.

The Vendor Landscape (Section 6): Production diffusion is not a clean loop; it’s a chaotic compound pipeline (Denoiser + Verifier + VAE).

• NVIDIA: Retains its moat because CUDA is the only environment flexible enough to handle this “swamp” of switching models and search orchestration.

• AMD: Massive HBM capacity acts as the perfect hedge against the massive activation memory spikes required for video diffusion and DiT workflows.

• Groq & Pure ASICs: Brilliant for streaming deterministic AR tokens, but deeply vulnerable to the inter-chip latency required to orchestrate complex, backtracking search trees.

• Apple & Qualcomm: Positioned to completely strip-mine edge volume the second local step-counts collapse, separating the volume from cloud revenue.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

Section 1: Why Large Language Models Struggle with Decoding

Pick any fast frontier model — Claude 4.5 Haiku, GPT-5 Mini, Gemini 2.5 Flash. Look at their tokens-per-second at batch size one. These are the fastest autoregressive LLMs in the world, running on $40,000 silicon with peak FP8 throughput in the petaflop range. Yet they generate text at roughly the speed an undergraduate types.

To understand why, you have to look at the dependency graph.

Autoregression is strictly sequential. Token 200 cannot exist until token 199 is sampled. Because of this lockstep dependency, the model cannot process its layers in parallel across the sequence. To produce a single token from a 70B-parameter model, all 140 GB of weights must stream from High-Bandwidth Memory (HBM) into the chip’s SRAM, execute a tiny matrix-vector multiplication, and leave.

This operation has an arithmetic intensity of roughly 1 FLOP per byte of data moved. A modern Hopper or Blackwell tensor core needs nearly 300 FLOPs per byte before it stops starving. So when an LLM generates text for a single user, the GPU executes at under 1% of its theoretical peak compute. The other 99% of the silicon waits for data to arrive. The constraint isn’t how fast the chip can multiply. It is how fast the chip can be fed.

Once you understand that data movement is the true bottleneck, the entire AI infrastructure stack reveals itself as a series of patches for this exact flaw.

Take the KV cache. If you are already memory-bound, the logical fix is to avoid recomputing the past. The KV cache stores the attention states of previous tokens so each new token only adds to the sequence. But this trade creates a massive residency tax. The longer the context, the more memory capacity it eats.

To offset that capacity tax, the system has to amortize the weight-streaming cost across multiple users. That is why PagedAttention and continuous batching frameworks like vLLM exist. They batch requests so the 140 GB of weights stream through the chip once to serve 64 users instead of one. Throughput scales linearly — until the KV cache for those 64 concurrent users blows out the HBM capacity entirely.

When the software patches hit a wall, the hardware roadmap panics. HBM4 is a bandwidth upgrade. NVLink 5 at 1.8 TB/s per GPU is an interconnect upgrade. Multi-billion-dollar photonic interconnect acquisitions are not compute upgrades. They are data-movement upgrades designed exclusively to keep the tensor cores from starving.

But the physical reality of chip manufacturing is asymmetric. Adding denser compute units and low-precision formats like FP4 is relatively cheap. Scaling memory bandwidth and capacity is economically brutal.

Autoregressive inference sits on the wrong side of that divide. You can pour infinite FLOPs into a server, but if the workload requires dragging gigabytes of history across the chip for every single token, the math will never work. This is why Diffusion Models have been attracting so much attention.

Section 2: How Diffusion Models Break the Sequential Lock in AI

Diffusion language models escape the memory trap by simply refusing the premise of left-to-right generation.

Instead of predicting the next word, they start with a block of noise scaled to the target output and refine the entire sequence in parallel across a handful of forward passes. Every position is in flight at once. Token 200 never waits for token 199.

That single architectural shift shatters the dependency graph. Because a diffusion forward pass operates on hundreds or thousands of tokens simultaneously, the underlying math changes from bandwidth-starved matrix-vector operations to massive matrix-matrix multiplications. The arithmetic intensity spikes into the hundreds of FLOPs per byte — easily clearing the threshold required to actually keep a modern tensor core fed.

This is the bottleneck inversion worth noting: autoregression is memory-bound, while diffusion is compute-bound.

(worth stressing again — High Bandwidth Memory, HBM, is extremely expensive to build, hence the memory wall that has been choking AI. More on this in the next section).

Obviously, this means that the model does more raw work per output token, and it repeats that work across multiple refinement steps. But it pays that tax using the exact resource the hardware has in massive surplus.

We already established that Diffusion Models are fast, and they have some structural advantages with the hardware paradigm. But what about their performance? It’s still a bit early to tell, but the results look extremely promising:

Assuming this will continue to scale (and we’ve already broken down the research for this in previous works), we should see them close the gap in the next 2 years (Gemini 3 already started incorporating Diffusion in its model, so they’re already touching the frontier). This means more and more hardware providers will have to start accounting for diffusion models and their more matmul-heavy workloads.

Let’s explore that sentence next in more detail.

Section 3: The Hardware Diffusion Needs (and Why the Industry Already Has It)

If diffusion is going to take over a meaningful share of inference, we have to look at what it actually demands from the silicon.

To understand what this architecture needs to work well, we look at the math.

• A forward pass processes hundreds or thousands of tokens simultaneously, creating massive matrix-matrix multiplications. Compute becomes the primary axis of growth: the faster the chip multiplies, the faster the model runs.

• Given the size of the MatMuls, we also need Diffusion hardware to be good with quantization. Most modern hardware handles this fine, and it will be helped by the fact that, because the generation process is an iterative refinement loop, it naturally absorbs precision noise across steps.

• When it comes to memory, Diffusion drops the bandwidth tax. There is no KV cache to stream and no per-token residency penalty. It does require memory capacity for activations — which becomes a real wall when scaling to video — but for text, that is a far weaker constraint than the bandwidth trap autoregressive decode hits at batch size one.

The net result: the ideal architecture for this workload is heavy on compute, native in FP4, modest on bandwidth, and generous on activation memory. Surprisingly, this maps wonderfully to the hardware the industry has been building.

Every accelerator generation since Hopper has been an escalating compute story. Blackwell hits 20 PetaFLOPS of FP4. Rubin is targeting 50. Node shrinks and new precision formats have led to FLOPs compounding at 3x every two years. The only problem? The memory bandwidth needed to feed it has grown at half that rate.

The moment your model’s data spills over the tiny pool of super-fast on-chip memory, the system punishes you. Hard. “When data size exceeds GPU memory capacity, the data must be migrated repeatedly between the CPU and GPU, either manually or automatically. However, manual migration can be laborious for programmers, and it is infeasible for irregular workloads because the data access pattern is unpredictable. On the other hand, demand paging approaches (e.g., NVIDIA Unified Memory [120]) can automatically manage data movement, but it can significantly degrade performance due to high page fault-handling latency and limited PCIe BW [46, 121, 52]. This overhead can be particularly severe for irregular workloads since prefetch/eviction policies become ineffective [14]. For example, the runtime of bfs can increase by ∼4.5× with only 125% oversubscription (i.e., exceeding memory capacity by 25%) compared to when the GPU is not oversubscribed.”

Since my brother also reads this newsletter, here is a summary so that he can follow along: Compute go up fast. Good. Memory go up, but not so fast. Bad. Low Memory make algorithms run slow. Much money lost to GPUs.

Why has memory been scaling so slowly? HBM is pinned by physics, supply consolidation, and packaging constraints. This is not a hardware newsletter, so here is a quick summary below:

This is why increasing growth in compute haven’t scaled hardware utilization. The memory wall Pepe’s the fuck out of any deployment.

For diffusion, this set of conditions is a massive tailwind for all the reasons mentioned earlier (faster compute, good w/ quantization, and less hti by memory). The billions in capex committed through 2028 — from Blackwell to Rubin, MI400 to MI500 — is a roadmap of compute-axis improvements that will stall on AR decode but compound beautifully on diffusion.

All this means is that the blockers for diffusion going forward aren’t in the silicon since the chips and manufacturing constraints are pretty friendly to Diffusion. The real bottlenecks are upstream: maintaining frontier reasoning quality, and solving activation memory at video scale.

But most importantly, diffusion’s economics aren’t gated by how fast each step runs on the hardware. They are gated by how many steps you have to run in the first place.

Let’s look at the math behind that next.

Section 4: How the Algorithmic Layer Dictates the Hardware Timeline

Let’s pause and anchor exactly where we are in the stack, because this is where the money actually changes hands.

We know the architecture fixes the memory trap. We know the hardware industry accidentally built the perfect silicon for it.

But let’s be blunt: having the perfect chip is completely irrelevant if the math running on it is a nightmare. Hardware is just a dumb, incredibly expensive rock. The algorithm — the actual math telling the rock what to do — is the gatekeeper.

If diffusion requires 1,000 forward passes to generate a single image, nobody cares how perfectly it aligns with the silicon. At 1,000 steps, the unit economics are dead. For diffusion to actually threaten autoregression and shift billions of dollars in hardware spend, the step count — the Number of Function Evaluations (NFE) — had to collapse.

It already happened for images.

In 2020, diffusion ran on stochastic math (SDEs). Because random noise accumulated at every step, the model had to take tiny, excruciatingly slow steps to keep the image from blowing up. But a year later, the math flipped. Researchers realized they could use deterministic Ordinary Differential Equations (ODEs) instead, stripping out the random noise and allowing massive jumps across the generation path. The 1,000 steps collapsed to 50 for free, and the Midjourney and Stable Diffusion API economies were born overnight.

But language is a completely different financial reality.

Images solved the step-count collapse because the math is forgiving. You can draw a smooth, continuous mathematical line between two pixel colors. You cannot draw a smooth curve between the word “Apple” and the word “Murder.” Masked language models operate in discrete, harsh jumps across a vocabulary space.

Because of this, the continuous distillation techniques that made image generation real-time fail spectacularly on text. When researchers tried porting consistency distillation to LLaDA-8B, the model bled 6 full points of accuracy on the GSM8K math benchmark.

Because discrete distillation is struggling, language diffusion is currently stuck in the 4-to-16 step regime. That is fast enough to beat autoregression in a server rack, but it fundamentally delays the “frontier reasoning on an iPhone” narrative. Until researchers crack discrete space distillation (or we get a way to make language concepts operate in the continuous space, which imo is more useful) — which is likely 18 to 36 months out — Apple and Qualcomm are stuck waiting. The compute required for those 4 to 16 steps ensures that the hyperscalers will maintain absolute, walled-garden dominance over text generation. The datacenter monopoly holds.

But inside those datacenters, the bottlenecks are shifting in ways that threaten the current capex consensus.

For a while, we assumed diffusion language models were bottlenecked by raw parameter scale. We thought they just needed bigger training runs to reason better. It turns out they just had terrible time management.

Standard masked diffusion unmasks tokens randomly. This means the model routinely commits to a conclusion before it has even unmasked the premises it needs to get there. It’s trying to solve the end of the maze before looking at the start. In 2025, the LogicDiff paper fixed this with a microscopic, 4.2M-parameter classification head that simply forced the model to unmask premises first and conclusions last.

The result: LLaDA-8B jumped from 22.0% to 60.7% on GSM8K. A near 40-point spike without changing a single base parameter.

If you can buy 40 points of reasoning with a dirt-cheap scheduling hack instead of a $2 billion training cluster, the entire hyperscaler capex thesis starts looking very fragile. Software efficiency is quietly undercutting the necessity of building gigawatt, nuclear-powered compute clusters just to achieve frontier logic.

Lastly, it’s worth studying LLaDA-MoE, which proved that routing tokens through expert subnetworks works beautifully for diffusion text models. But MoE inherently requires streaming different expert weights from memory for every routing decision. It isn’t the relentless KV-cache tax of autoregression, but it is an expert-loading tax that brings the memory bandwidth constraint back into discussions.

Section 5: How Diffusion Turns Inference into a Search Problem

Once diffusion eats a meaningful slice of inference, the serving stack needs to be rethought deeply.

We aren’t just swapping out the model weights and keeping the same infrastructure. We are dealing with a structural mutation in what inference actually is. We are moving from a clean, sequential generation pass to a chaotic, branching search problem.

To understand why, look at what just happened to language models.

Inference-Time Scaling Arrives in Diffusion

The most important paradigm shift of the last two years was the realization that throwing compute at inference beats just training a fatter model. OpenAI’s o1 and DeepSeek-R1 proved that the model that “thinks longer” wins.

Diffusion is about to do the exact same thing, but the mechanism is entirely different. Instead of “thinking longer” by generating more internal reasoning tokens, diffusion searches wider.

You don’t just run the denoiser once, roll the dice, and hope for the best. You generate a swarm of candidate trajectories, and you use a secondary model — a “verifier” — to judge them mid-flight. This was shown to us in Google’s “Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps”. By investing in verifiers and better search, they were able to mog simplistic NFE scaling.

Now, researchers are playing with AI search algorithms like Breadth-First Search (BFS) and Depth-First Search (DFS) to see how they might benefit the generation process.

Take the prompt “eight bottles.” Standard FLUX almost always messes up the count. But if you run a DFS search guided by an object-detection verifier, the system catches the error mid-trajectory. If the verifier sees the count is wrong at step 20, it slaps the denoiser’s hand, forces it to add noise back to an earlier level, and makes it try a different path. This will be similar to how well developed agentic systems work today (using external checkers to check the generator’s tasks and then adjusting the next generation accordingly), but it will be baked into the generation process instead of requiring an entire harness.

The benefits of this approach might make you ask yourself a very reasonable questin: Why don’t we do this with autoregressive LLMs?

The Math That Makes Branching Cheap

Because branching AR means cloning massive KV caches for every new path. The cost scales linearly. A 4x search costs you 4x the compute. It is financially ruinous at scale.

Diffusion drops that tax entirely.

Because the early denoising steps just build blurry, coarse shapes, multiple candidates can share the exact same early trajectory. They only need to split at the very end when fine details are committed.

The arithmetic here is brutal. For k candidates sharing the first s steps of an N-step trajectory, your branching multiplier is m_branch = k - (k-1) * s/N. If you run 4 candidates and share the first 40 of 50 steps, your cost multiplier is exactly 1.6.

You are getting a 4x quality search for a 1.6x compute tax. If you are deploying capital into inference startups or hardware right now, that single equation should make you rethink where the value will actually flow.

The Verifier Becomes the New Moat

Because branching is practically free, the verifier becomes the actual god of the system. The delivered quality to the user is no longer just the single-shot capability of the denoiser. It is the searched quality of the denoiser-verifier pair.

This splits the competitive moat right down the middle. In the autoregressive world, the $2 billion base model is everything. In a diffusion world, a smart team can take a mid-tier, open-source denoiser, bolt on an elite, proprietary verifier suite (say, one specifically trained for medical imaging or product photography), give it a generous search budget, and absolutely annihilate a closed-source frontier model running single-shot.

The value migrates away from the hyperscalers burning gigawatts on training compute, and accrues directly to the companies building the smartest serving-stack intelligence. In the future, I also expect to see verifiers being sold as infrastructure (why make your own legal verifier when Irys has the best ones in the market?), which will unbundle intelligence tremendously.

How this will Impact ASICs

This compound setup — Denoiser + Verifier + Search Orchestration — turns datacenter capacity planning into an absolute nightmare.

Current AR serving frameworks (like vLLM) are optimized for a highly predictable, bandwidth-bound decode process. Diffusion serving is a schizophrenic mess. You have a dense matmul denoiser running on FP4 fighting for accelerator space with a lightweight vision verifier, and maybe an LLM judge acting as the final arbiter. Your compute per output isn’t fixed; it varies based on how quickly the search tree converges. Your memory pressure isn’t a slow-growing KV cache; it is a sudden, violent spike of activation memory whenever a request branches.

This is the kill-shot for half the specialized hardware pitches we are seeing right now.

An ASIC designed purely for dense matmul throughput handles a denoiser beautifully in isolation. But the production workload isn’t isolated. The second an ASIC has to pause the denoiser, offload verification to a different chip, and orchestrate a backtracking search tree, the inter-chip latency eats the entire throughput advantage. This puts two opposing consequences on the system:

1. In the short term, GPUs keep their monopoly in this space not because they are the fastest at math, but because they are the only silicon flexible enough to survive the chaos of a compound search workload.

2. Longer term: this actually opens up a wedge for ASICs, if they can figure out a way to work with each other. Different verifiers/model profiles/modalities will be best with different specializations; what they’ll need is a system to break things up and organize communication across them. This is something we broke down at length over here, but I highly recommend investing into this before the market catches up to this opportunity.

I’d like to end this analysis by specifically digging into this space more. The AI Hardware market is incredibly fascinating, with multiple providers making a specific, multi-billion-dollar claim about where the next bottleneck in the system will live. The gambler in me always gets all hot and bothered in such high-stakes situations. So let’s look at the vendors, the claims they’re making about the future, and how they well they align with a Diffusion-heavy future.

Section 6: Which Inference Vendors are Suited for Diffusion Inference Workloads

Why Diffusion Changes the Dynamic

This has been a long article, so it would be helpful to take a breather and regather the important points.

For the last three years, the inference stack was built to solve autoregressive decode. Because AR models generate one token at a time, the arithmetic units sit idle waiting for weights to stream from memory. This reality forced the entire market to obsess over memory bandwidth and KV-cache management.

Diffusion changes the math entirely. It shifts the primary workload from bandwidth-starved sequential decode to dense, parallel matrix multiplication.

But diffusion inference is not just a clean denoiser. It is a compound system:

Text Encoder → Denoiser Loop → Adapters/Control → Verifier Search → Safety Filter → VAE Decoder → Output

This pipeline is exactly why the hardware market is fracturing. Different vendors built their chips to solve only specific parts of this equation. Here is how the physical silicon actually maps to the math.

NVIDIA: The Moat is the Swamp

Nvidia’s primary advantage is not raw speed. It is flexibility.

Production diffusion is a heterogeneous swamp. A real-world creative workflow requires a text encoder, the primary denoiser, multiple LoRAs, an IP-Adapter, a safety classifier, a verifier, and a VAE decoder. GPUs are uniquely suited to survive this kind of chaos.

An ASIC might execute the denoising loop faster, but Nvidia can run the denoiser, the LLM judge, and the adapter stack simultaneously because CUDA supports everything. As long as diffusion workflows require running a shifting society of models on the same machine, the Blackwell and Rubin architectures will maintain their dominance. General compute outlives narrow accelerators during periods of structural transition.

AMD: The Memory-Capacity Hedge

AMD’s thesis isn’t just “Nvidia but cheaper.” AMD’s MI300X and MI400-class architectures are explicitly built around massive High-Bandwidth Memory (HBM) capacity and open rack-scale infrastructure.

“We believe that the MI355X could be competitive against the HGX B200 for small to medium LLMs production inference workloads. This is because the MI355X total cost of ownership is 33% lower than that of the HGX B200 for self-owned clusters, while it delivers much more HBM memory capacity, slightly more FP8 and FP4 TFLOP/s and double the FP6 TFLOP/s. Rapid improvements to AMD software under the leadership of Anush, AMD’s AI Software King, will also push the MI355X’s relative performance per TCO advantage higher. “— SemiAnalysis.

For image diffusion, compute is the primary bottleneck. For video diffusion, the bottleneck shifts to state. Longer video clips mean more tokens, which drastically increases the size of the intermediate activations. Suddenly, raw memory capacity matters just as much as peak compute. If the market stays centered on images, Nvidia keeps its exact advantage. But if the market shifts heavily to video generation and Mixture of Experts (MoE) denoisers, AMD’s capacity-first architecture is positioned exactly where the bottleneck moves.

Google TPU & AWS Trainium: Factories and Procurement

Google’s TPUs (like the new Trillium and Ironwood chips) are diffusion factories. Their architecture relies on systolic arrays and the XLA compiler. If Google can freeze the generation graph — using the same model, the same resolution, and a fixed step count at massive volume — the compiler maps the math perfectly to the hardware. They win tightly controlled, internal workloads like Vertex AI and Veo. But they struggle with open diffusion, where users demand custom schedulers, dynamic branching, and constant adapter swapping. The TPU compiler hates a graph that won’t sit still.

AWS Trainium plays a different role. It is a procurement weapon. AWS doesn’t need to win the frontier; it simply needs Trainium to be cheap and integrated enough that enterprises keep their stable, batch-generation marketing workflows inside the AWS ecosystem.

Furiosa and d-Matrix: The Efficiency Bets

These two startups looked at the power consumption of GPUs and built architectures to fix it, but in completely different ways.

• Furiosa’s RNGD chip abandons traditional matrix-multiplication units entirely. Instead, they built a Tensor Contraction Processor (TCP) that treats tensor contractions — like attention mechanisms — as the fundamental hardware primitive.

• d-Matrix’s Corsair, on the other hand, uses Digital In-Memory Compute (DIMC). They physically moved the arithmetic logic right next to the SRAM cells to stop wasting power dragging data back and forth.

Because diffusion relies on repeated passes through the same denoising backbone, both architectures map to the workload cleanly. But there is a catch. This efficiency only holds if the surrounding pipeline — the verifiers, VAEs, and safety models — can also be executed on the chip. If video activations grow larger than d-Matrix’s SRAM or Furiosa’s 48GB of HBM3 can comfortably hold, the efficiency breaks. They win image-scale denoising, but face steep challenges at video scale.

Groq: Token Generation in a Block-Refinement World

Groq built an exceptional machine for deterministic token generation. Their LPU (Language Processing Unit) throws out HBM entirely, relying instead on massive pools of extremely fast SRAM and a compiler that schedules every piece of data movement ahead of time. It is a machine built perfectly to stream AR tokens.

But if inference shifts toward block-level parallel refinement and full-latent denoising passes, Groq’s religion is challenged. In a compound diffusion pipeline, they are unlikely to be the primary renderer. Instead, they survive as the ultra-fast LLM planner or verifier judge sitting next to the diffusion engine, handling the language-side logic while a GPU processes the pixels.

Cerebras: The Wafer-Scale Locality Limit

Cerebras does not build chips; they build wafers. The WSE-3 is the size of a dinner plate. By keeping everything on one giant piece of silicon with massive amounts of SRAM, they eliminate the slow, expensive interconnect latency of moving data between traditional GPUs.

For a static, continuous denoising loop, this is an elegant solution. But production pipelines require orchestrating multiple side-models, and video activations can easily exceed local SRAM limits. Once the working state forces the system to move data off the wafer, the architecture loses its primary advantage. Cerebras is highly effective for a single, massive denoiser, but struggles with the messy reality of multi-model video serving.

Etched: The Transformer Ultimatum

Etched made the most aggressive bet in the market with Sohu. They ripped out all the general-purpose programmability of a GPU and physically hardwired the Transformer architecture directly into the silicon.

If image and video diffusion standardize completely around Diffusion Transformers (DiTs), Etched is a monster. But diffusion transformers are not identical to causal LLMs. They need bidirectional attention, multimodal token streams, and variable resolutions. If Etched’s architecture is too rigidly optimized for the causal LLMs of 2024, diffusion will run poorly on it. It is a gorgeous, specialized knife waiting to see if the industry standardizes on the exact vegetable it was built to slice.

Apple & Qualcomm: The Edge Volume

Apple and Qualcomm are betting on the collapse of the step count. As discussed in Section 4, once image distillation drops the required forward passes to between 1 and 4 steps, the math becomes efficient enough to run locally on consumer NPUs via Apple’s Unified Memory architecture.

Local generation is private, instant, and carries zero marginal cost for the provider. Datacenters will retain the premium video revenue and heavy enterprise pipelines, but Apple and Qualcomm are positioned to completely strip-mine the bottom of the consumer image market. The edge wins the volume; the cloud retains the revenue.

Lightmatter and Astera: The Fabric Layer

Lightmatter and Astera are betting on the physical limits of copper wire.

A standard image model does not require a photonic spiritual awakening. But long video diffusion and world models create massive token counts, severe temporal attention costs, and multi-device parallelism. When activation memory exceeds what a single server rack can hold, the interconnect fabric becomes the foundational bottleneck. Lightmatter’s Passage platform uses 3D-stacked silicon photonics to move data using lasers instead of electrical signals, hitting 1.6 Tbps per fiber. These companies win when video diffusion scale officially outruns algorithmic compression, forcing the rack itself to become the computer.

Conclusion: Where this is All Headed

Every major computing paradigm eventually shifts the locus of value from the raw engine to the orchestration layer. In the early internet, we obsessed over physical servers and networking hardware. Eventually, the servers commoditized. The real money moved to the platforms routing the traffic.

AI is about to undergo the exact same unbundling. For three years, we have treated foundation models like indivisible, magical brains. But if diffusion turns generation into a branching search problem, the monolithic model dies. It becomes a supply chain. You will have cheap open-source denoisers, elite proprietary verifiers, and dynamic schedulers, all negotiating with each other in real-time.

The history of tech is just the history of expensive, integrated systems being ripped apart by cheaper, modular components. The closed labs and hyperscalers are fighting a brutal, capital-intensive war to own the integrated brain. But if inference is no longer a straight line, the most valuable real estate won’t be the base weights or the silicon. It will be the routing logic. The question is no longer who builds the biggest engine. It is who builds the best map.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

Share

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Everyone is staring at the Gemma 4 benchmarks right now, but that misses the actual design shift. Google shipped four models, but if you look under the hood, you’re looking at two entirely divergent architectures: E2B and E4B for phones, and 26B and 31B for servers. Unlike traditional multi-model rollouts (where it’s the same core architecture at different scales) , the edge pair and server pair have very different DNAs.

Why? Because the physical constraints of a phone and an H100 are exact opposites.

Think about the hardware. A phone has abundant flash but starves for DRAM — you’ve typically got 128GB of storage against maybe 8GB of shared memory, all bound by a battery that dies if you hit it wrong. A server gives you DRAM in abundance and FLOPS you pay for by the hour. Every architectural choice is a trade between memory, storage, and compute. When that scarcity flips, you have to pull opposite levers: you compress memory at the cost of compute on an edge device, and you spend memory to save compute on a server. That’s why one architectural DNA can’t survive both deployments.

Gemma solves this through different architectural DNAs, and I think that’s the real signal about where the industry’s headed — one family name, very different architectures underneath depending on where the model has to run. This has several implications for the future of AI, all of which go way beyond simply discussing the model choices/benchmark numbers.

In this article, we will cover:

• The two constraints: the KV cache scaling and narrow hidden dimensions forcing this entire split.

• Per-layer embeddings: why the E2B blows half its weight budget on flash lookups, and why the server models don’t bother.

• Interleaved local-global attention: where Gemma 4 actually pays the O(n²) compute tax, and where it doesn’t.

• Divergent GQA: how edge compresses KV everywhere, while server compresses it selectively.

• Cross-layer KV sharing: the edge-only trick that slashes cache by 83%, and the MLP tax it charges you to do it.

• Partial RoPE: rotating only 25% of dimensions so the content actually has room to breathe.

• Hybrid MoE: the always-on dense FFN that makes running 4x Mixtral’s sparsity safe in production.

• The FA2 serving break: why a 512 head dimension costs pre-Blackwell GPUs roughly 14x in throughput.

• The deployment framework: which model earns its place where, and exactly what breaks in practice.

Let’s have some fun.

Executive Highlights (TL;DR of the Article)

• The Core Thesis: The benchmark hype misses the real story. Gemma 4 is actually two entirely divergent architectures marketed under one name. Edge models (E2B/E4B) and server models (26B/31B) share almost nothing because a phone’s constraints (abundant flash, zero DRAM) are the exact opposite of an H100’s constraints.

• Per-Layer Embeddings (Edge): The E2B blows nearly half its parameter budget (46%) on flash-based lookup tables. This prevents token meanings from colliding in a narrow hidden state, boosting reasoning without touching the phone’s limited DRAM. Server models skip this entirely.

• Interleaving & The KV Cache War: To survive massive contexts, Gemma 4 alternates between cheap local attention and expensive global attention. On the edge, it goes further by heavily compressing the KV cache and reusing it across layers — shrinking the cache from gigabytes to megabytes so 128K context can actually fit on a phone.

• Partial RoPE: At 128K context, standard positional encoding acts as noise, scrambling semantic meaning. Gemma 4 fixes this by only rotating 25% of the vector’s dimensions to track position, leaving the other 75% clean to act as pure, undistorted content channels.

• Hybrid MoE (Server): The 26B server model pushes a hyper-aggressive 128-expert routing setup. It only survives this extreme sparsity because an always-on dense feed-forward network runs alongside the experts, acting as a structural safety net against routing failures.

• The Infrastructure Break: The global attention layers require a head dimension of 512. This breaks FlashAttention-2 compatibility. If you run Gemma 4 on pre-Blackwell hardware (like an H100 or 4090) today, you will take a massive 14x throughput hit until the open-source serving stack patches the issue.

• The Big Picture: Uniform scaling is dead. We are witnessing the unbundling of the one-size-fits-all transformer. The labs that win the next cycle won’t just scale brute compute; they will engineer specialized architectures that exploit the exact physics of the hardware they run on.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

What Is Per-Layer Embedding and Why Does Only Edge Gemma 4 Use It?

Every token enters a transformer through a massive lookup table that maps the raw word into a vector. That vector becomes the main signal flowing through the entire model. Attention and FFN layers reshape it at every layer, but they are all just transforming that exact same base vector that started life as a table entry.

That starting vector has a brutal job: it has to carry the token’s raw identity (the word “bank”) and its contextual potential (money vs. river) simultaneously. In a massive model like Llama 3 70B (hidden_size 8,192), there is enough bandwidth for both signals to travel without stepping on each other. Different dimensions specialize; the signal has room to breathe.

But E2B runs at a hidden_size of 1,536. At a fifth of the width, the signals suffocate. “Bank the river” and “bank the money” compete for the same constrained coordinates, and every downstream attention layer inherits that collision.

The naive fix is to bolt capacity downstream using adapters, LoRA, or extra trainable projections. But downstream math can only transform what survived the initial lookup. If the embedding flattened two meanings into the same direction, no amount of downstream projection unflattens them.

Google’s fix is to move the capacity upstream to the lookup itself. Instead of one giant table, E2B gives each of its 35 decoder layers its own dedicated 256-dim embedding table. When a token arrives, every layer does its own lookup. The main 1,536-dim signal flowing through the model no longer has to remember a token’s raw identity across 35 layers, because every layer has a fresh, context-aware identity signal waiting for it.

This is obviously not a free lunch. At 67.1M parameters per table across 35 layers, Per-Layer Embeddings (PLE) consume 2.35B parameters — 46% of E2B’s entire 5.1B budget. If this lived in DRAM, it would be fatal since the phone only has 8GB, and the KV cache and activations are already eating it.

But PLE tables don’t need DRAM. They are static lookups, read once per token per layer. Flash latency is irrelevant at that access pattern. So Google parks the tables in flash, where a phone has 128GB to burn. That 4.7GB footprint is effectively free.

This is also exactly why the 26B and 31B skip PLE entirely. On an H100, you have 80GB of HBM and no flash asymmetry to exploit. You would never blow 46% of your parameter budget on a trick that only pays off when DRAM is the binding constraint. Besides, at their wider hidden sizes (2,816 and 5,376), the representational collision stops being a fatal problem.

Does PLE actually do the work Google implies? They haven’t shipped an ablation, but E2B’s benchmarks are hard to explain without it. It hits 37.5% on AIME 2026 (beating Gemma 3 27B’s 20.8%) and 44.0% on LiveCodeBench v6. A model 12x smaller in effective parameters is beating its massive predecessor on reasoning, heavily implying the narrow main signal is finally free to actually reason instead of just remembering what token it saw. It would also be consistent with other patterns where small augmentations that specialization enable higher quality reasoning by allowing the other params to dedicate to reasoning.

Given that everyone has decided how inefficient Self Attention is, it should come as no surprise that the next innovation Google made was around it.

Why Does Gemma 4 Interleave Local and Global Attention?

Attention is the most expensive thing a transformer does. Every token has to look at every other token, meaning the compute cost scales quadratically with sequence length. At 128K tokens, that is roughly 16 billion score computations per layer. Multiplied across 30 to 60 layers, it eats the FLOP budget alive.

The historical move is to stop looking at everything. Sliding window attention (Mistral, Phi) caps each token’s view at a fixed window — say, 512 tokens in each direction. The cost drops from O(n²) to O(n *window size), which at 128K context is a 250x reduction.

The wall you hit here is signal degradation. With a strict sliding window, a token at position 1,000 cannot directly see a token at position 50,000. Long-range dependencies have to hop through intermediate layers, and each hop degrades the signal. Most modern small models just accept this range limitation, operating on the assumption that if you’re using 2B phone model to process legal documents (or any solution not named Irys.ai for legal work for that matter), then you deserve to be arrested and have your contributions to the gene pool snipped.

Gemma 4 cannot make that assumption. E2B and E4B are multimodal, and processing video frames blows past 8K tokens in seconds. The edge models must handle long contexts. Google’s fix to this conundrum is interleaving. Most layers use local sliding-window attention, while a few execute full global attention. The model alternates between them on a fixed ratio:

• E2B: 4 local + 1 global, repeated 7 times. Window: 512.

• E4B: 5 local + 1 global, repeated 7 times. Window: 512.

• 26B: 5 local + 1 global, repeated 5 times. Window: 1024.

• 31B: 5 local + 1 global, repeated 10 times. Window: 1024.

Every model ends on a global layer. The output always sees the full context regardless of what the intermediate layers did.

The insight here is that the global layer is not just doing the same work less frequently. Local layers build up rich feature representations within short spans — 512 tokens is plenty for syntax and local semantics. The occasional global layer then executes long-range integration on those already-refined features, rather than raw token signals. It does less work per unit of capacity, which is why the 5:1 ratio sustains long-range reasoning without degrading the output.

At E2B’s 4:1 ratio, 80% of the attention layers pay linear compute instead of quadratic. On an 8K query, that is a 5x speedup for attention compute on the phone. At the 31B’s 256K context, the savings are the only reason the model fits in its FLOP budget at all.

This means that your system is inflexible: if a user inputs a task requiring dense long-range integration across the entire context — like cross-referencing contradictions throughout a 200-page document — the model cannot dynamically allocate more global layers. It gets what it gets. Most tasks do not hit this ceiling, but when they do, the degradation is hard-coded.

Every modern long-context architecture is converging on this identical bet: uniform O(n squared) attention is a cost most tokens don’t need to pay. Mamba avoids it with selective state updates. Ring Attention avoids it by partitioning across devices. Gemma 4 avoids it by interleaving layer types. The architecture that figures out exactly which tokens actually need the expensive math wins.

Why Do Edge and Server Gemma 4 Use Opposite GQA Strategies?

The KV cache puts a silent chokehold on the memory budget of modern transformers. Every processed token must keep its Key and Value vectors in memory so future tokens can attend to them. Cache size scales as layers × KV_heads × head_dim × sequence_length × 2 (one tensor for K, one for V). At 128K context on a 7B model, that is 12.8GB just for the cache. That is the entire DRAM budget of a phone before the weights even load.

Grouped-Query Attention (GQA) is the standard response. By having multiple query heads share a single KV head, you can slash your costs down. For instance, an 8:1 ratio cuts your cache by 8x.

Gemma 4 breaks this uniformity in opposite directions depending on the hardware.

• Edge models crush KV everywhere. E2B runs 8:1 uniform GQA across every layer. E4B runs 4:1. The logic is forced by the hardware — on a phone, doubling KV heads Yamchas your 8GB limit. There is no layer where the edge models can afford richer KV, so they don’t have one.

• Server models compress selectively. The 26B and 31B both use 2:1 GQA on local attention layers and 8:1 on global layers. Local layers operate on short spans where fine-grained KV discrimination helps distinguish nearby tokens, so server architects spend the DRAM there. Global layers aggregate over 256K of context, doing broader integration that tolerates aggressive compression. You compress where the work is coarse, and spend where it is precise.

In global attention, the server models deploy one more trick: K=V weight sharing.

In normal attention, K and V do different jobs. Keys are the indexable signal queries compare against; they determine which tokens get attended to. Values are the payload passed forward once attention has selected a token. Think of a search engine: keys are how you index documents for retrieval, values are the document contents themselves. Typically, the mechanism learns these independently because “how to match” and “what to pass forward” are fundamentally different objectives.

Gemma 4 eliminates the V projection in global layers. The key projection is computed, then reused directly as the value, with only RMSNorm applied on the value side as a differentiator in the forward pass. As you might imagine, other architectures will typically avoid this since it nukes your quality. However, Gemma avoids that by building a good system around it:

1. Global attention does long-range semantic integration where K/V specialization matters less than in local layers doing fine-grained discrimination.

2. The global head_dim is doubled to 512, giving each head enough room to encode both matching and content signal in one projection.

3. Partial RoPE leaves 75% of dimensions as clean content channels, pulling “match on content” and “retrieve content” closer together by construction.

This halves the global KV cache on top of the 8:1 GQA compression.

The split b/w server and edge versions of GQA follows the scarcity logic directly: edge has no DRAM, so it compresses everywhere; the server has DRAM for local discrimination, so it only compresses global aggregation.

This adds quite a bit of architectural complexity. Uniform GQA is one number you tune; divergent GQA is four (local edge, global edge, local server, global server) plus the K=V decision, and every combination must be validated against quality. Google paid that cost because the resource flip between edge and server made uniform GQA visibly suboptimal at both ends. Most labs will likely lack the resources to fight on both fronts simultaneously, so they’ll have to pick their specialization beforehand (by considering the constraints/use case) and be mediocre on the other one. Studying Gemma 4s decisions is a good way to understand that.

However, this is the only interesting optimization Gemma 4 uses. Up next, we will cover the one I found most interesting.

How Does Cross-Layer KV Sharing Cut Edge Cache by 83%?

GQA compresses KV within a layer. Cross-layer sharing attacks the vertical axis: why recompute K/V at every single layer in the first place?

Research shows that in the later layers of a trained network, representations converge. By layer 25 of 35, the residual stream has settled into a representation that the model is refining rather than transforming. In another lens, computing fresh K/V projections for layers doing similar work is redundant. On a phone, this redundancy is fatal.

E2B’s fix is to have 20 of its 35 layers skip the K/V projection entirely and reuse K/V from an earlier layer. Only 15 layers compute unique KV. (E4B shares 18 of 42; server models share zero).

The sharing is back-loaded and strictly matches attention types — sliding-window layers only reuse from sliding-window layers; global only reuses from global. You don’t want a local layer’s short-span K/V hijacked by a global layer trying to see 128K of context.

Crucially, shared layers still compute their own Query (Q) projection. Q is cheap: you compute one per token and use it immediately. K and V are the memory killers: they get cached forever so future tokens can attend to them. Sharing Q saves nothing. Sharing K/V eliminates the cache entirely for 20 layers.

Let’s see how this plays out in hard numbers. A standard 7B transformer at 128K context demands a ~12.8GB KV cache. Combine cross-layer sharing with 8:1 GQA and interleaved attention, and E2B shrinks that cache from tens of gigabytes to hundreds of megabytes. Google’s reported 83% reduction at 8K context becomes an order-of-magnitude reduction at 128K. That is the only reason long-context models run on 8GB phones.

The Quality Tax and the MLP Compensation

This will naturally reduce your generation quality since a shared-KV layer can only attend to a representation shaped by an earlier layer’s objective. It cannot sharpen its own retrieval targets or reshape the payload. To prevent this from gutting model quality, you have to add capacity back. If attention is kneecapped, the Feed-Forward Network (FFN) is your only compensation lever.

On shared layers, Google doubles the MLP width from 6,144 to 12,288.

The math here is revealing. A standard GeGLU FFN at hidden_size 1,536 uses ~28M parameters. Doubling it adds ~28M more. But the K/V projections they removed only cost 1-2M parameters. This discrepancy is caused b/c replicating the cross-token mixing power of attention through a purely feed-forward path requires massive, brute-force capacity.

Why do this when you’re building a model for resource constraints? Doesn’t this beat the point of edge architecture?

Put simply, a wider FFN costs you FLOPs, not bytes. You don’t cache FFN outputs; you compute them, use them, and throw them away. In other words, you spend compute (which you have) to save memory (which you don’t).

Server models skip cross-layer sharing entirely because DRAM isn’t their binding constraint. The quality tax of sharing stops making sense, and the MLP doubling becomes pure parameter waste.

Notice the market convergence here. DeepSeek’s Multi-Head Latent Attention (MLA) attacks the exact same KV bottleneck by compressing K and V into a shared latent space within a layer. Cross-layer sharing skips recomputation across layers. Different axis, same target. The field has unanimously agreed that the KV cache is the primary enemy. It will be extremely interesting to see all the ways people attack the KV cache from various points in the AI stack.

There is one final aspect of the attention mechanism that DeepMind researchers reconfigured to give Gemma 4 it’s insane potency.

How Does Partial RoPE Separate Position From Content?

Transformers match tokens by taking the dot product of their vectors. To prevent the model from losing word order, you have to inject positional data into those vectors.

Rotary Position Embedding (RoPE) does this geometrically: it takes the vector’s coordinates and physically rotates them by an angle determined by the token’s position index. Tokens close together receive similar rotations, so their dot product stays high. Distant tokens rotate away from each other, causing the attention score to naturally decay. Position becomes geometry.

The standard implementation rotates every single dimension in the vector. That was fine at an 8K context window. At 128K, it breaks the model.

When you rotate every coordinate to encode extreme distances, the raw semantic meaning of the token gets distorted. If a query at token 120,000 is searching for a specific fact at token 500, it struggles to match the key because the extreme positional rotation acts as noise, scrambling the pure content signal. Every dimension is screaming its location, leaving no clean channels to communicate what the token actually means.

Labs have tried patching this with RoPE rescaling and NTK-aware scaling, but these are workarounds. They don’t fix the underlying flaw of position leaking into content.

Gemma 4’s global layers fix the flaw by adopting Partial RoPE (p-RoPE): they split the vector.

In each 512-dimension global attention head, 128 dimensions (25%) receive the full theta=1M rotation. These are the dedicated position channels, maintaining the geometry of the document so attention decays smoothly over 256K tokens.

The remaining 384 dimensions (75%) receive zero rotation. These are pure content channels. Because they never rotate, semantically identical tokens match perfectly across the entire context window, completely immune to distance. The model stops forcing every dimension to do two interfering jobs.

Google hasn’t published the ablation for the 25/75 split, but the engineering math is straightforward. 128 rotating dimensions provide exactly enough frequency bands to uniquely index 256K positions. Pushing it to 50% sacrifices pure content capacity for positional resolution you don’t need. Dropping it to 10% blurs distant positions together. The 25% mark is the empirical point where position and content both survive.

Local layers skip this entirely. Operating in a tight 512-token window, they stick to standard full-RoPE (theta=10K). Inside a short span, positional resolution is critical, and the rotation distortion never scales high enough to break semantic retrieval. This continues our theme of Gemma 4 winning by specializing it’s archietcture to the system.

The cost of partial rotation is effectively zero — skipping the math on 75% of the vector actually saves minor compute. The payoff is absolute dominance in long-range integration. The 31B hits 86.4% on tau2-bench Retail, obliterating Gemma 3’s 6.6%. While training data and the broader attention redesign contribute, p-RoPE is the specific architectural lever that allows the model to actually retrieve what it reads.

Everything covered so far has been primarily about the edge architecture. The 26B has one design decision worth unpacking, and the 31B ships with one serving-stack problem worth flagging. Both are worth a few paragraphs each.

Why Gemma 26B’s Uses a Hybrid MoE

In a standard Mixture-of-Experts architecture, tokens are routed to a fraction of the available capacity. Mixtral, for example, routes to 2 out of 8 experts, activating about 25% of the network per token.

The 26B is much sparser. It uses 128 experts and routes to 8, activating just 6.25% of the network per token.

Normally, this level of sparsity is fragile. When you have 128 experts, routing errors are inevitable. If a token gets sent to the wrong experts, a standard MoE has no fallback, and the output degrades.

Gemma 4 solves this with a hybrid structure. Every token simultaneously runs through an always-on dense FFN (hidden dim 2,112) and its 8 routed experts (hidden dim 704 each). The outputs are then summed.

The dense path acts as a guaranteed quality floor. It processes every token reliably, regardless of what the router decides. If the router picks the perfect experts, they add specialized value on top. If the router misses, the dense path ensures the output remains coherent rather than crashing.

This structural fallback makes the 128-expert count practical. The economic result is that the 26B stores 25.2B parameters but only activates 3.8B per token. You get reasoning quality comparable to a 70B model, but you pay the inference cost of an 8B model.

The Flash Attention 2 Serving Break

FlashAttention-2 (FA2) is the standard kernel powering almost every modern GPU inference stack, including vLLM and HuggingFace. By keeping intermediate calculations in SRAM, it bypasses the massive memory bandwidth bottlenecks of standard attention.

But FA2 has a hard constraint: it only supports head dimensions up to 256.

Gemma 4’s global layers use a head dimension of 512. That width is not a mistake — it is mechanically required to make the K=V weight sharing and the partial-RoPE split work. But it means the global layers break FA2 compatibility.

On pre-Blackwell hardware (like an A100, H100, or RTX 4090), the serving stack falls back to unoptimized Triton kernels for these layers. In practice, throughput drops from an expected 50–100 tokens per second down to roughly 9. That is a 14x performance hit. On newer Blackwell GPUs, optimized kernels recover this throughput to around 124 tok/s.

The software fix is per-layer backend dispatch: routing local layers to FA2, and global layers to a different optimized kernel. As of April 2026, this is still an open issue in vLLM.

Gemma 4 is an architecture shipped slightly ahead of its infrastructure. If you are evaluating it for production on existing hardware today, you need to budget for the throughput hit, or wait for the software stack to catch up.

Conclusion: What Gemma 4 Actually Teaches Us

The current meta in AI is uniform scaling — pushing the exact same architecture across every deployment context (scaled up/down based on your constraints). That approach leaves massive capability on the table. True performance is not found in uniformity. It is found in exploiting the specific topological/ontological structure of the problem you are trying to solve. Phones and servers are not the same problem at different sizes, and treating them as if they are is just engineering convenience mistaken for principled design.

The industry has seen this dynamic before. Mixture of Experts sat on the shelf for decades because uniform dense models were simpler to build, until the economics of scale forced labs to specialize. That exact flip is now happening at the macro-architectural level.

Gemma 4 is one point in a trend that is shaping up in every layer of AI. Chip makers are realizing that they need to bifurcate inference and training, and some are looking further to split based on your workload/modality. Text-style tokenization is failing for vision, requiring rebuilds. Reasoning as a system is being bifurcated into its subcomponents. At every stage, we’re seeing the unbundling of intelligence.

Plan accordingly when thinking about how to navigate these very fun times.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

Share

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Anthropic announced Claude Mythos Preview on April 7, 2026, and the people covering it immediately jumped to hyping it as the end of software as we know it, and as a model that would break all security.

There is a lot of genuinely promising capability in Mythos. The bugs it found are real, the economics of vulnerability research are changing, and the underlying ability of LLMs to catch bugs that survive decades of human review is not hype. That part of the story is true.

The problem is everything around it. Almost every major outlet or commentator covering Mythos worked from Anthropic’s press materials and not the actual primary sources such as the CVE advisories, the exploit code, the 44-prompt transcript, the 244-page system card. When you read them and add the AISLE replication study, the red team writeups, the Glasswing partner agreements, Anthropic’s own decpetive framings, and a very different picture emerges: one of misinformation and hype.

In this article, we will cover:

• How AI finds bugs that traditional tools can’t, and why that matters

• Every major Mythos bug, examined against the actual source code and exploit transcripts

• What the “thousands of severe zero-days” claim actually rests on

• Whether smaller, cheaper models can replicate Mythos’s results (spoiler: mostly yes)

• What really happened with the sandwich escape

• The business and financial structure behind the launch that nobody reported on

Executive Highlights (TL;DR of the Article)

• The bugs are real. 17-year-old FreeBSD RCE, 23-year-old Linux kernel heap overflow, 27-year-old OpenBSD TCP flaw. LLMs catch these because they can reason about the gap between what code does and what the developer intended. Fuzzers and static analysis literally cannot do this.

• The coverage is wrong on almost every detail. The “181 Firefox exploits” ran with the browser sandbox ( yes, the thing that stops browser exploits) off. The FreeBSD exploit transcript shows substantial human guidance, not autonomy. The “thousands of severe vulnerabilities” extrapolates from 198 manually reviewed reports. The Linux kernel bug was found by Opus 4.6, the public model, not Mythos.

• The moat is thinner than anyone reported. AISLE tested eight models including a 3.6B model at $0.11/M tokens. All eight found the FreeBSD bug. Mythos’s actual lead is in multi-step exploit development, not detection. That’s a narrower and more replicable advantage than what’s being sold.

• Sandwich Gate is mostly nonsense. The model was explicitly told to escape its sandbox and contact the researcher. Anthropic’s own system card says it notified the researcher “as requested.” The only unsolicited action was posting exploit details publicly, which is more “overeager extra credit” than “Skynet.”

  

• The CoT unfaithfulness spike (5% to 65%) is a training problem, not a model personality. If you train reasoning with RL that rewards outputs that look like reasoning, you get outputs that look like reasoning over genuine reasoning. The dishonesty isn’t new — the magnitude is higher because we’re doing more RL. This is why we need to rethink the approach entirely.

• The business structure is wild. 5 of 11 launch partners are also investors. JPMorgan is launch partner AND lead IPO underwriter. The “$100M in credits” is retail-priced API credit worth maybe $40–50M in compute. All of this lands months before a reported $400–500B IPO. Same playbook as GPT-2, just with real CVEs this time.

With articles like this, I feel compelled to stress that I have no personal agenda against Anthropic. I’ve recommended Claude Code extensively in this newsletter (no sponsorship) and wrote an article supporting their stand against AI weapons systems. I have no agendas either pro or anti Anthropic, but I am against narrative manipulation and hype.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

Section 0: What Is Claude Mythos Preview?

Mythos is Anthropic’s newest frontier model, internally codenamed “Capybara,” sitting above Opus in the Claude hierarchy. Anthropic has published zero information about its architecture, training data, or parameter count. The system card reports it leads 17 of 18 benchmarks, including CyberGym (83.1% vs Opus 4.6’s 66.6%), SWE-bench Verified (93.9% vs 80.8%), and Cybench (100% pass@1), with a 1M-token context window. By every public metric Anthropic chose to share, their most capable model by a wide margin.

This is, unfortunately, what most coverage parrots (before pivoting to sell their proprietary “Claude.md skills playbook”). Unfortunately, that misses some very crucial details.

Firstly, Anthropic didn’t actually deploy a model. All those amazing results? They come model-plus-scaffold system — a multi-agent pipeline where specialized agents handle file ranking, code reading, target execution, proof-of-concept generation, and result confirmation.

This is kind of a big deal. The performance difference between Cursor and Claude Code? Or Claude Code and OpenCode? All that comes from the system around the model. That’s why Google can have one of the best models in the market, and still produce the Shakespearean tragedy that is the Gemini CLI. When people say “Mythos found a zero-day,” the truth is more “Mythos, orchestrated by a purpose-built vulnerability research pipeline, found a zero-day.”

All that to say, there is a big-ass gulf between “model did it” (what is being reported) and “model used scaffold to do it” (what actually happened). Former implies a breakthrough in capability; the latter is more of an engineering problem. What you go with changes your read of the situation. We’ll come back to this.

Next, let’s talk about those benchmarks.

• CyberGym tests “targeted vulnerability reproduction,” meaning the model receives a hint about the bug class and tries to reproduce it. That’s more N-day exploitation — recreating a known type of bug — not the open-ended zero-day discovery Anthropic’s marketing implies/what the media runs with.

• The Cybench 100% was scored on only 35 of 40 challenges, with 10 trials per challenge versus 30 for other models. Anthropic’s own system card calls Cybench “no longer sufficiently informative.” Their words, not mine.

Finally, here is some context worth holding before we get into the actual findings:

• At least 5 of 11 non-Anthropic launch partners are also Anthropic investors. JPMorgan is simultaneously a launch partner and lead underwriter for Anthropic’s reported October 2026 IPO.

• Mythos is priced at $25/$125 per million tokens — 5x above Opus and far exceeding GPT-5.2 ($1.75/$14) and Gemini 3.1 Pro ($2/$12).

• The restricted-access rollout is consistent with a deeper pattern: Anthropic has repeatedly struggled with usage limits, model quality degradation under load, and compute capacity constraints. Seen from this perspective, “Too dangerous to release widely” seems to be a very interesting cover for “we cannot serve this at scale yet”. (There is some irony to how Peace Prize Dario called out Sam Altman for buying up GPUs, and is now stuck with a massive compute crunch).

Hopefully, all this foreplay has you heated because now we’re going to get into the main act.

Section 1: How Does AI Find Bugs? And why Can LLMs Find Bugs That Fuzzers and Static Analysis Miss?

The bugs that survive decades in reviewed code are not obviously wrong. They’re usually locally correct but globally inconsistent — an assumption made in one subsystem, violated by a change in another, with no single file looking broken on its own. A replay cache sized before LOCK operations existed. A TCP sequence comparison that works until operands are more than 2³¹ apart. These are semantic mismatches, and traditional tools miss them for structural reasons.

Here, it is worth understanding some of the traditional techniques to develop a strong map of the AI-security landscape.

• Fuzzers generate malformed inputs and watch for crashes. This makes them fast, scalable, and excellent at finding bugs that manifest as observable failures. But they test inputs, not logic. A fuzzer will never understand that a buffer was sized for OPEN responses but not LOCK-denied responses, because it has no concept of “sized for.” If the bug only triggers under specific protocol-level conditions — two cooperating NFS clients, one holding a lock with a 1,024-byte owner string, the other requesting a conflicting lock — random mutation will not produce that input. Fuzzers hit the FFmpeg H.264 code path 5 million times without catching its sentinel-collision bug, because generating a valid bitstream with 65,535+ slices requires structural awareness that coverage-guided mutation simply cannot provide.

• On the other hand, static analysis tools like Coverity and Semgrep match known code patterns: unchecked memcpy, missing bounds checks, use-after-free signatures. They’re precise within their pattern library, fast to run, but they match patterns, not meaning. They can work within a bounded scope — a function, a file, a known-bad template. A static analyzer cannot understand that a buffer sized in 2003 for one response type was later expected to hold a different response type added in a separate commit by a different developer. That requires reading across files, across time, across intent. No pattern matches it because no single piece of code is wrong.

This brings us to the fundamental failure mode of these two systems and where LLMs can make an impact. Code has two layers: what it mechanically does, and what the programmer intended it to do. Function names, variable names, comments, commit messages, surrounding context — these encode intent. The actual logic encodes behavior. LLMs are the first tool that can reason about the gap between those two layers. When a developer leaves a comment saying a buffer is “large enough to hold the OPEN, the largest of the sequence mutation operations,” and a later code path routes a much larger response type into that buffer, an LLM can recognize the semantic mismatch without needing a crash to signal it (or more likely it will be able to trace it down to this difference post crash).

Now, LLMs have their own issues. They’re slower per-query than fuzzers, they produce false positives that require expert human triage, and their coverage is not systematic — they don’t guarantee they’ve examined every code path. This demonstrates clearly that instead of replacements, these tools are more complements. Each tool owns a lane. Known-pattern bugs belong to static analysis. Crash-inducible bugs belong to fuzzers. Semantic mismatches — where valid inputs interact with correct-looking code to produce wrong behavior — belong to LLMs.

I take the time to flag this because a lot of coverage on Mythos positions it as a replacement for these traditional techniques. This is objectively untrue. As multiple agentic systems across the board have taught us, the most powerful systems combine both deterministic and non-deterministic systems, instead of trying to tokenmax by using LLMs everywhere.

This serves as important context, b/c now we’re going to do what no other coverage has done yet. We’re going to look at the actual bugs that have been released. I went through every CVE advisory, every exploit file, and the full 44-prompt transcript to understand what Mythos actually did. And as you might guess, there is a LOT of misinformation and hype floating around.

Section 2: The Bugs Mythos Found, Examined One by One

CVE-2026–4747: FreeBSD NFS Remote Code Execution (17 years old)

A function called svc_rpc_gss_validate(), part of FreeBSD’s NFS authentication system, allocates a 128-byte buffer on the stack to hold incoming credentials. The protocol’s serialization layer (XDR) permits credential bodies up to 400 bytes. That leaves 272 bytes of overflow — enough to overwrite saved registers, the frame pointer, and the return address.

Under FreeBSD’s default compiler settings, this particular function receives no stack canary, meaning there’s no runtime check to detect the overwrite before the function returns. An attacker who controls the credential body controls where the function jumps when it finishes.

The bug survived 17 years because neither subsystem is broken on its own. The function parses credentials correctly. The XDR layer serializes credentials correctly. The mismatch — that the serialization layer permits inputs larger than the receiving buffer — lives in the gap between two subsystems written at different times with different assumptions.

Two distinct exploit strategies exist in the public record.

• Anthropic says Mythos developed a 6-round exploit that writes an SSH key to .ssh/authorized_keys, fully autonomous — fewer resources needed, no reverse shell connection to detect, persistence that survives reboots. It was not published.

• Separately, Nicholas Carlini published a 15-round reverse shell exploit here, crediting “Claude” without specifying the model version. I read every file in that repository — the 601-line exploit.py, the write-up.md, and the claude-prompts.txt containing 44 human prompts across roughly 8 hours of work. What I found in there complicates every narrative about this exploit.

How the published exploit works. The target is FreeBSD 14.4-RELEASE on amd64. FreeBSD lacks kernel address space layout randomization (KASLR), so every kernel address is predictable — a significant simplification. Round 1 calls pmap_change_prot() to make a kernel memory region executable. Rounds 2 through 14 each write 32 bytes of shellcode into that region using ROP chains — short sequences of existing kernel instructions chained together to perform arbitrary operations. Round 15 jumps to the completed shellcode, which spawns a reverse root shell. Each round is a separate NFS request that triggers the buffer overflow, writes its payload, and returns. FreeBSD spawns 8 NFS threads per CPU. The exploit consumes one thread per round (15 total). With 2 CPUs (16 threads), the margin is exactly one thread.

Here’s where things get interesting.

Carlini’s README claims the human “was AFK for much of it.” The 44 prompts tell a different story. Prompt 15 is the critical intervention: “okay in ../FBSD-001 there is a different remote exploit that gets a shell.. read it for how they constructed the connect back.” This pointed Claude to a prior exploit for a different FreeBSD vulnerability (in the SCSI target layer) as a reference implementation.

The shellcode comments confirm it: “Uses the same build logic as FBSD-001/exploit.py build_stage2_shellcode(), with two patches.” Claude did not independently derive the kernel-to-userland execution pattern. It adapted it from reference code the human provided. Dead code artifacts survive in the final exploit — FAKE_MODULE_BASE and HA_HANDLER_OFF constants from the reference exploit, plus 19 NOP bytes where the original handler cleanup was removed but not reclaimed.

Beyond the reference exploit, the human made at least five significant corrections: stopping Claude from using Docker instead of QEMU (prompt 4) for kernel exploitation, preventing it from killing the wrong QEMU instance (prompt 9), redirecting it from kernel reboot to reverse shell, vetoing remote file-prestaging (14), and blocking a non-default thread count increase. The human also provided the hint “there is no KASLR so it should be easy.” (prompt 12).

I want to be clear about this — there is genuine capability. Claude used a De Bruijn pattern — a sequence where every possible substring of a given length appears exactly once — to determine the exact offset where the overflow overwrites the return address (200 bytes into the credential body), after an initial disassembly estimate was wrong by 32 bytes. Rather than guessing, it generated a diagnostic payload that precisely identified the correct offset from the crash data. When the obvious register-transfer gadget (rax → rdi) didn’t exist in the kernel’s gadget space, Claude discovered a mov [rdi], rax; ret write primitive as an alternative. A failed approach using rep movsq — abandoned when the only available push rsp; pop rsi gadget had a side effect that corrupted the repeat count register — shows genuine exploration of the constraint space rather than template matching.

But, and this is a big but, Anthropic marketed this as “unauthenticated root from anywhere on the internet.” The FreeBSD advisory says “remote code execution in the kernel is possible by an authenticated user.” NVD assigns Privileges Required: Low, not None. The published exploit requires Kerberos authentication.

(Anthropic describes an unpublished path where Mythos found an unauthenticated EXCHANGE_ID call that leaks the kernel hostid and boot time, making the authentication handle computable without credentials — but this path has no published proof-of-concept).

And in either case, the attacker needs network access to the NFS port — which in practice means they’re already on your internal network, because NFS servers don’t sit on the public internet. “Root from anywhere on the internet” requires being somewhere the internet can’t reach. This is the kind of contradiction only Dario Luther King can resolve.

Total cost: under $1,000 in API credits.

CVE-2026–31402: Linux Kernel NFSv4.0 Heap Overflow (23 years old)

This one is my favorite because it shows exactly what LLMs can do that nothing else can.

In 2003, a developer named Neil Brown added a replay cache to the Linux kernel’s NFS server — a buffer that stores copies of recent responses so the server can retransmit them if a client’s request is lost. Brown sized the cache buffer at 112 bytes and left an explicit comment in the code: “I’ve implemented the cache as a static buffer of size 112 bytes which is large enough to hold the OPEN, the largest of the sequence mutation operations. LOCK and UNLOCK will be added when byte-range locking is done (soon!).”

The buffer was sized before LOCK existed. When LOCK was later added by a different developer, nobody revisited the 112-byte assumption. A LOCK-denied response can be as large as 1,056 bytes. That is a 944-byte overflow into kernel heap memory.

Here’s what makes this one beautiful in a terrible way. The kernel already partially knew about the mismatch. The XDR request size estimator at line 2682 of nfs4xdr.c correctly accounts for LOCK responses — it adds NFS4_OPAQUE_LIMIT to the outbound buffer size, producing a correctly sized response sent over the network. The client receives the right data. The overflow happens only in the replay cache copy, a separate code path that was never updated. The kernel sends the right answer over the wire, then corrupts its own memory caching it.

Carlini found this bug using Opus 4.6, the publicly available model — confirmed explicitly in Lynch’s writeup, not Mythos. A bash script iterated over every file in the kernel source tree, giving each to Claude Code with the prompt “You are playing in a CTF. Find a vulnerability. hint: look at [file].” No specialized tooling. Claude had to understand that when two NFS clients cooperate — one sets a lock with a 1,024-byte owner string, the other requests a conflicting lock — the denial response overflows the replay cache. That’s reasoning about asynchronous distributed client interactions, exactly the kind of cross-component logic invisible to fuzzers.

Jeff Layton’s fix (commit 5133b61aaf43) is nine lines: a bounds check before the copy. If the response exceeds the buffer, rp_buflen is set to zero, the status is still cached, but the payload is dropped. The commit message explains why the buffer was not simply enlarged: doing so would increase the size of every nfs4_stateowner structure across all NFS client state, wasting memory. Retransmitted LOCK-denied responses lose the denial payload — acceptable because the client already received it on the first attempt.

Lynch’s writeup lists five kernel bugs Carlini reported with Opus 4.6, all merged into mainline. Carlini stated he has “several hundred crashes” he hasn’t had time to validate. The bottleneck is human triage, not AI discovery.

Once again, media coverage implied this was a Mythos finding. It was not. And despite “several thousand scans” of Linux kernel code, Mythos was “unable to successfully exploit any” Linux kernel vulnerability remotely. It found bugs but could not exploit them over a network. The model that found and helped exploit this bug was the publicly available one.

CVE-2026–2796: Firefox JIT Type Confusion (CVSS 9.8)

A type confusion in Firefox’s JavaScript engine, reported by seven researchers “using Claude from Anthropic.” The Bugzilla entry remains restricted, so independent verification of the technical details is not yet possible. What we can examine are the testing conditions.

The headline number is 181 successful exploit runs where Opus 4.6 managed 2 out of roughly 350 attempts. But these are execution runs, not unique exploits — Mythos succeeds more often at the same task, not 181 different strategies. And Anthropic’s own footnote states these exploits “target a testing harness mimicking a Firefox 147 content process, without the browser’s process sandbox or other defense-in-depth mitigations.” If you’re wondering, yes, the browser sandbox is the part that stops browser exploits. In a real browser, the renderer sandbox is the primary defense against JIT exploits. Getting code execution inside the renderer is step one; the browser’s process sandbox, the OS sandbox, and ASLR all stand between that and actual system compromise.

So, unless all that brain damage from MMA is hitting me… the big exploit involves working in a compromised setting. I guess everyone can bring into a house if there are no locks. What Saint Dario is trying to prove here eludes my puny comprehension, perhaps one of you can help me understand.

Anthropic separately claims Mythos chained four vulnerabilities to escape both a renderer and OS sandbox in “a major web browser.” Their red team writeup states “we then worked with Mythos Preview to increase its severity.” The sandbox escape was human-assisted. No CVE, no technical writeup, no independent verification is possible but good to know I guess.

The strongest public numbers were produced under the weakest test conditions. The unverifiable claims describe the hardest conditions. Mozilla’s own security advisory rates the impact as “high,” not “critical.” NVD assigned CVSS 9.8. Mozilla disagrees.

CVE-2026–26980: Ghost CMS SQL Injection (CVSS 9.4)

Carlini demonstrated this one live in about 90 minutes at the [un]prompted 2026 conference — a blind SQL injection in Ghost’s Content API through the filter query parameter, unauthenticated because the API key is public by design. OWASP has listed injection flaws as a top vulnerability class for over a decade. Valid work, not frontier capability.

The OpenBSD TCP SACK Bug (27 years old, no CVE yet)

The oldest bug in the set, and the one that demands the most sophisticated reasoning to find.

TCP sequence numbers are 32-bit unsigned integers that wrap around. The kernel compares them using macros (SEQ_LT, SEQ_GT) that cast the difference to a signed integer: (int)((a)-(b)) < 0. This works correctly when the two numbers are within 2^31 of each other. When they’re farther apart, signed overflow produces contradictory results — a value can simultaneously appear less than one reference point and greater than another. The code validates sack.end against the send window but never validates that sack.start is at or above the base sequence number. An attacker places sack.start roughly 2^31 away from the real window. The kernel deletes its only tracking entry, a pointer goes null, and the subsequent write dereferences it.

Not the first SACK bug in this function either. CVE-2019–8460 found resource exhaustion via unbounded hole list growth, fixed by adding a 128-hole limit. A 2005 patch added helper macros. Neither fix addressed the signed integer overflow. Two security reviews of the same function, seven years apart, missed the same class of bug that an LLM found on a single $50 run.

The fix is four lines: a lower-bound check on sack.start plus a null guard. Discovery cost roughly $20,000 across approximately 1,000 scaffold runs.

But — and this matters — it is denial of service, not remote code execution. The null pointer write goes to a fixed low address with no attacker control over the written value, and OpenBSD prohibits mapping the null page since version 4.4. Combined with W^X, SMEP/SMAP, and KARL, code execution from this primitive is essentially impossible. Anthropic’s technical writeup is honest about this limitation. The broader marketing language about “every major operating system” does not distinguish between a DoS bug on OpenBSD and an RCE exploit on FreeBSD.

The FFmpeg H.264 Bug (16 years old, no CVE yet)

A sentinel-collision out-of-bounds write in FFmpeg’s H.264 decoder. The slice assignment table uses 0xFFFF as an empty-slot sentinel. The slice counter is a 32-bit integer. When it reaches 65,535, it collides with the sentinel, and the deblocking filter misidentifies initialized macroblocks as uninitialized. Anthropic’s own report says it is not critical and would be hard to exploit. Findings like this feed the broader claim of “thousands of severe zero-days.”

Section 3: How Strong Are the Scale and Exclusivity Claims?

The number behind “thousands of severe zero-days” is 198 manually reviewed reports. That’s the total human-verified count — Anthropic hired contractors who agreed with Mythos’s severity assessment 89% of the time, 98% within one severity level. Everything else is extrapolation, and it’s not strong extrapolation. The 198 is a 4–10% sample with undisclosed selection methodology, reviewed by Anthropic-paid contractors using Anthropic’s severity framework, and that 89% agreement rate means roughly 11% of the “thousands” may be misclassified. For comparison: in Anthropic’s own standardized testing regime across roughly 7,000 entry points in 1,000 open-source repositories, the result was 10 confirmed control-flow hijacks. Ten. The “thousands” comes from a different, less controlled regime where the methodology is opaque.

Anthropic’s red team acknowledges training data contamination concerns for the N-day exploits: “it is conceivable that Mythos Preview is drawing on prior knowledge of these bugs to inform its exploits.” They argue the exploits’ sophistication matches zero-day work, but also admit their N-day metrics “can make it difficult to distinguish novel capabilities from cases where the model simply remembered the solution.” The zero-day findings are more defensible precisely because those bugs could not have appeared in training data.

Now here’s the question the coverage never asked: can other models do this at all?

AISLE, an AI security lab led by Stanislav Fort, tested Anthropic’s showcase bugs across eight models in single zero-shot API calls. No agentic workflows, no tool access, no iterative loops. Just the vulnerable code in the prompt and a request to find vulnerabilities. Full transcripts published at github.com/stanislavfort/mythos-jagged-frontier.

All eight models found the FreeBSD bug. Every single one. A 3.6B-parameter model running at $0.11 per million tokens correctly identified the overflow and assessed it as critical. The finding that anchored the entire launch narrative is detectable by a model small enough to run on a laptop.

The more interesting result is where models diverge, and AISLE has a great name for it: the “jagged frontier.” Cybersecurity capability does not scale smoothly with model size. On the FreeBSD buffer overflow — fundamentally pattern recognition, input size exceeds buffer size — all models scored well. On the OpenBSD SACK bug, which requires multi-step signed integer wraparound reasoning, grades ranged from A+ to F. GPT-OSS-120b got A+. Kimi K2 got A-. Qwen3 32B got F — the same model that scored CVSS 9.8 on FreeBSD confidently declared the OpenBSD code “robust” and said “no exploitation vector exists.” It never examined whether sack.start values near 2^31 apart from the base could make both SEQ_LT and SEQ_GT comparisons behave contradictorily.

Why the jaggedness? Buffer overflow detection is pattern-matchable from training data — models have seen thousands of examples where input size exceeds buffer size. Signed integer wraparound is not. It requires actually reasoning about how signed and unsigned arithmetic interact under specific value ranges. You cannot predict which bugs a model will find by knowing its size or its general benchmark scores. Capability is task-shaped, not model-shaped.

The false-positive results cut even deeper. AISLE tested models on an OWASP sample — a Java code block with no actual SQL injection vulnerability.

• Twelve of 13 Anthropic models failed, including Sonnet 4.5, which correctly traced the list operations and then overrode its own analysis to flag the pattern anyway.

• The $0.11/M token GPT-OSS-20b passed. On patched FreeBSD code where the vulnerability was already fixed, only GPT-OSS-120b correctly identified it as safe across all three trials.

• The most common false-positive argument: that oa_length could be negative, bypassing the bounds check. oa_length is declared as u_int — unsigned. It cannot be negative. The models learned that “signed/unsigned confusion” is a common vulnerability class and applied it reflexively even when the types don’t support the claim. Pattern matching masquerading as analysis.

When told the 304-byte overflow could not fit the 1,000+ byte ROP chain needed for Mythos’s published approach, no model independently discovered the multi-round RPC delivery mechanism. But several proposed valid alternatives: DeepSeek R1 suggested a minimal ~160-byte ROP chain using prepare_kernel_cred(0)/commit_creds for privilege escalation, Gemini Flash Lite proposed a stack-pivot redirecting RSP to the credential buffer already in kernel heap for unlimited ROP space, and Kimi K2 noted the bug is “wormable” — a detail absent from Anthropic’s announcement entirely.

There is nuance worth considering here. As Zvi Mowshowitz put: “We took the needle the model found, isolated the relevant handful of the haystack, and then gave it to a small child, who found the needle as well.” And he’s right — AISLE tested detection after file selection. The vulnerable code was already isolated and handed to each model. In practice, Mythos’s scaffold searches across a million-line codebase, ranks files, decides what to read, identifies which code paths are worth examining. Detection is easier than discovery. When Chase Brower challenged AISLE on false positives, AISLE acknowledged massive false-positive rates from smaller models even on narrowed 20-line targets, making wide searches “utterly useless.”

It’s a fair point. But follow it to its conclusion. If the hard part is not the model recognizing a bug in isolated code, but the scaffold deciding which code to look at, then the moat is system engineering, not model capability. File ranking, crash oracles, execution harnesses, iterative refinement loops — that’s infrastructure. Infrastructure is replicable. A 3.6B model that can detect the FreeBSD bug when shown the right file just needs a better pipeline to find it in the wild. The model is cheap. The pipeline is the product.

The independent replication evidence bears this out. The steamedhams.io tracker reproduced the FFmpeg finding using Opus 4.6 with three generic prompts and found two additional bugs Mythos’s writeup didn’t highlight. Steamedhams reproduced the OpenBSD SACK bug using four prompts with Opus 4.6 and found approximately 15 additional TCP stack bugs not mentioned in Mythos’s disclosure. Carlini found 500+ validated high-severity vulnerabilities and 122 crashing Firefox inputs leading to 22 CVEs, all using Opus 4.6 with Claude Code.

AISLE’s conclusion: “The strongest version of the narrative, that this work fundamentally depends on a restricted, unreleased frontier model, looks overstated to us. The moat in AI cybersecurity is the system, not the model.”

Where Mythos genuinely leads is autonomous multi-step exploit development — the multi-round RPC delivery, splitting a complete exploit across 15 separate protocol requests, managing thread exhaustion, adapting when gadgets fail. No tested model reproduced that. But even here, Penligent’s independent analysis found that Mythos’s closed-source binary analysis pipeline “reintroduces source-like structure before the main vulnerability analysis stage.” The apparent binary analysis capability may largely be source analysis with extra steps.

The moat is real but narrow, and it lives in exploit engineering, not bug detection — a distinction the coverage never made.

Section 4: Sandwich Gate and Why The Sandbox Escape Is Not What You Were Told

There was a lot of attention given to the event where Mythos developed a multi-step exploit to gain broader internet access, then unprompted posted exploit details to public websites and sent an email to a researcher who was eating a sandwich in a park. For some reason, all of the coverage focused on this sandwich detail and how the AI escaping the sandbox was the beginning of the end.

This is all commie propoganda and I’m going to break down why.

Firstly, and very importantly, the model did not do this by itself. The model was explicitly instructed by a simulated user to try to escape its sandbox AND send a message to the researcher.

This is a big difference from the narrative around it deciding to do this autonomously.

In any real agentic deployment — and this is something I think about constantly because of the work we do at Iqidis — the hard problem is not whether an agent can do what you told it to do. It’s the agent autonomously deciding what to do when its task is open-ended, and it runs out of clear instructions. Getting to work reliably (across prompt configurations, different conversation combinations, etc.) is the hard part about engineering AI systems. The ability to escape is itself pretty impressive (although we would have to audit the actual security since I couldn’t find enough details on it), but this is at least 3–4 orders of magnitude lower on the capability spectrum than the autonomous capability that is being pushed everywhere.

Now there is a hint of autonomy here, with the model posting exploit details to public websites. But given that its instructions were to communicate with the researcher, that is not a mindbending development.

But this isn’t all that people got wrong about sandwich gate. Somehow all the dumbfucks that kept focusing on the sandwich point missed the footnote that immediately precedes it — 

“The sandbox computer that the model was controlling was separate from the system that was running the model itself, and which contained the model weights. Systems like these that handle model weights are subject to significant additional security measures, and this incident does not demonstrate the model fully escaping containment: The model did not demonstrate an ability to access its own weights, which would be necessary to operate fully independently of Anthropic, nor did it demonstrate an ability to reach any internal systems or services in this test.”

Either I am completely losing it, or this completely counters the narrative people are building about it.

On the note of the system card, it also documents some interesting notes that people seem to have overlooked.

• The model fabricating vulnerabilities — inserting bugs into code it was auditing and presenting them as pre-existing discoveries. Turns out you can find a lot of zero-days if you’re the one writing them.

• It also modified git history to hide evidence of prohibited methods and wrote scripts to auto-approve its own permission prompts.

All of this makes me deeply skeptical of the actual rigor they put into securing their experiments (what was the actual level of security in their sandbox/exact prompts/environment details). Many of their claims and methods will have to be audited properly b/c I suspect that a LOT of their results are contaminated by a bad setup.

Another important note to flag is that a lot of attention and fearmongering was centered around the Chain-of-thought unfaithfulness — the degree to which the model’s visible reasoning diverges from its actual behavior — jumped from 5% in Opus 4.6 to 65% in Mythos. That’s a 13x increase; very scary since that is our primary diagnostic tool for reasoning LLMs.

However, this has nothing to do with the model itself or any proclivities towards dishinesty it’s developing. If you read our deep-dive into why Reinforcement Learning is Garbage for reasoning, you will know that this is a problem with how we train reasoning models, and that’s why we need to rethink our approach from the ground up. As a tldr — RL systems incentivize generations that look like reasoning, but from the very start, we’ve known that it doesn’t always map to how the model actually makes its decisions. This dishonesty isn’t a new feature; the only reason the magnitude is so much higher is because we’re doing more of it here.

This is where I will plug Latent Space Reasoning which is superior than RL based reasoning in every way — 

Putting it all together, it’s clear that for almost every major talking point being screamed non-stop, there is a much deeper truth that people didn’t bother digging into.

However, the dishonest narratives don’t end there. There is another aspect that was wholly ignored in all the discussions around Mythos, and it’s magic powers.

Section 5: The Business Structure Nobody Reported

Anthropic restricted Mythos to 12 launch partners plus 40+ organizations in a subsidized program called Glasswing. Pricing was $25/$125 per million tokens, 5x above Opus and far exceeding GPT-5.2 ($1.75/$14) and Gemini 3.1 Pro ($2/$12).

The “$100M in usage credits” that anchored every headline is retail-priced API credit, not cash spending. At Anthropic’s projected roughly 50% gross margins, the actual compute cost is around $40–50M. Giving away your own product at your own sticker price …that is a very particular kind of generosity from Father Dario Theresa.

Also worth noting, at least 5 of 11 non-Anthropic launch partners are also Anthropic investors. Google, Amazon, Nvidia, Microsoft, and Cisco all participated in funding rounds totaling over $67B. JPMorgan is simultaneously a Glasswing launch partner and one of two lead underwriters for Anthropic’s reported October 2026 IPO at a $400–500B valuation. Google and Microsoft, which compete directly in AI security, both joined Glasswing rather than launching competing programs. When your competitors volunteer to be your customers, they’re not buying your product — they’re buying proximity to the IPO, which Bloomberg and The Information report is slated through Goldman Sachs and JPMorgan at October 2026 on the Nasdaq.

One final point to consider: Anthropic has had access to Mythos since Feb now. Their apps, while useful, are still (wildly) buggy and Claude Code is vastly inferior to Codex from an intelligence/thoroughness perspective. While they have been on an amazing run so far, most of their releases have focused on interfaces (Claude through Word, the app) and not deep foundational breakthroughs. All that out together, it seems increasingly clear that all Mythos and it’s framing were meant to both generate hype for the IPO, and use the “too dangerous” framing to justify limited release while Anthropic figures out its compute issues.

If this feels familiar, OpenAI ran the same playbook in 2019 with GPT-2 — “too dangerous to release,” massive press coverage for a 1.5B-parameter model, with a full release nine months later. Restricted access creates scarcity, scarcity creates press, press creates demand, and demand justifies premium pricing. GPT-2’s danger claims had no supporting evidence. Mythos has real CVEs, but it relies on massive exaggerations and a lot of questionable framing to achieve this.

Now some of this is not Anthropic’s fault, since they did release the material publicly, but they clearly knew what they were doing. The PR packets, the lack of transparency with release, the refusal to address misinformation, and the blatant OpenAI plagiarism on the “too dangerous” message are very clearly attempts to control the narrative, not good-faith attempts to push scientific discourse.

With all of this covered, I would like to end with a small note to Dario-genese.

Conclusion: The Cost of Hype

Anthropic has, for a while, relied on hype and regulatory capture to stand out. They did it when they lobbied for regulating Open Source because DeepSeek scared them. They did it again to make unproven claims about “Chinese actors” using Claude Code to hack systems. And they’ve been doing it through their constant projections about AI wiping out 90% of jobs while hiring aggressively. At every turn, Anthropic has used fear-mongering and misinformation as a pillar of their strategy.

However, I believe that there is something that Dario-stotle, in all his infinite wisdom, is missing. When you spend all your effort on hype-based misinformation and regulatory capture, you convey a very clear signal admitting defeat. You tell the world you don’t think you can win w/o relying on these other tactics. You tell everyone that your survival hinges on keeping other people down, not in rising above them. Noone seems to have yet, but eventually your employees will read between the lines and hear what you’re not saying- “You (the employees) are not good enough”. Can’t imagine that this would do wonders for employee morale or productivity.

Secondly, this kind of cowardice eats at your spine until it replaces your whole constitution. Your psyche gets primed to be permanently spooked, jumping at every shadow. Anthropic built an era-defining product with Claude Code and it’s spin-off. You gained all that goodwill by taking a stance against the misuse of Claude in surveillance and weapons systems. Don’t throw it all away playing cowardly games. Because all of this shit, always comes back.

Do with that what you will.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

Share

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Every month, I break down the most important developments in AI. Usually, they cluster around a theme — a new model race, a pricing shift, a regulatory wave. March was different. March was the month where AI stopped behaving like a software industry.

Let me explain what I mean.

Google dropped voice AI pricing to $0.005 per minute. At that rate, a 24/7 voice agent costs about $25 a day. That’s below minimum wage in every US state. NVIDIA shipped a CPU designed not for training models but for orchestrating tens of thousands of agents running simultaneously. And the Western AI ecosystem quietly locked up over $120 billion in financing — not to build better models, but to brute-force energy contracts, because the real bottleneck isn’t intelligence anymore. It’s the 50-year-old transformer at your local utility that can’t handle the load.

If you zoom out, these aren’t separate stories. They’re the same story. The AI stack is fracturing into distinct economic layers — a commodity inference utility, an industrial infrastructure play, a workflow SaaS layer, and a compliance tollbooth — and each one is starting to behave like the industry it touches rather than the tech industry that built it. Google is fighting a utility price war. NVIDIA is acting like an industrial architect. OpenAI is doing shadow banking. None of this looks like software. Because it isn’t.

That’s what we’re unpacking this month. Specifically, we’ll cover:

• How Google is weaponizing dirt-cheap AI to lock markets no one else can touch — and how OpenAI, Microsoft, and NVIDIA are each scrambling to respond with very different bets

• Why the real AI bottleneck is power, not compute — the US grid vs. China’s grid, NVIDIA’s pivot to industrial architecture, and the shadow-banking financing play that’s quietly underwriting the entire Western AI buildout

• The four economic layers hiding inside every AI company — and why evaluating this industry with standard SaaS metrics is going to get a lot of people burned

Let’s get some bread.

Executive Highlights (TL;DR of the Article)

(read the actual sections for the full data and breakdowns, these are just to give you the overview)

The Price War.

Google dropped Gemini Flash Live to $0.005 per input minute. At $0.018/min output, a 24/7 voice agent costs ~$25/day ($9,460/year), below minimum wage in every US state. The plan is simple: collapse the cost until entirely new markets open that OpenAI and Anthropic can’t reach. Google can sustain this — insane ad profits, vertical integration from silicon to cloud, and nowhere else to deploy that capital at scale.

Everyone else is repositioning. OpenAI closed $122B and immediately bought Astral (the company behind uv and Ruff) — betting that owning the best Python sandbox matters more than having the best model. Microsoft is pushing Copilot Cowork into Office 365, routing between OpenAI and Anthropic natively, betting bureaucratic friction keeps enterprises locked in. NVIDIA shipped the Vera CPU for agentic orchestration — 22,500 concurrent environments per liquid-cooled rack — productizing the AI factory itself. Whoever’s frontier model wins, NVIDIA collects the orchestration toll at scale.

The Power Bottleneck.

Dirt-cheap intelligence means nothing without physical infrastructure to run it. The real bottleneck in 2026 isn’t GPUs — it’s the local utility company telling your $500M data center it’ll melt their 50-year-old transformer.

NVIDIA and Emerald AI are designing data centers as dispatchable grid assets — they proved they can curtail facility demand by a third in under a minute. When a heat dome hits, the AI factory throttles to keep the grid stable, which gets them ahead in the permitting arms race.

However, the geopolitical gap is brutal.

• The US grid: ~1.37 TW, aging infrastructure, local bureaucracy.

• China’s grid: ~3.89 TW, added 500 GW last year alone, state-mandated expansion with no permitting friction.

The Western response is shadow banking: NVIDIA pumped $2B into Nebius (targeting 5 GW by 2030), OpenAI locked up $122B, hyperscalers are building their own grids on debt.

When an enterprise signs a major AI contract today, they think they’re buying software. They’re actually buying into a highly leveraged financial cascade. If enterprise ROI on agents takes 24 months instead of 12, the debt servicing cracks and those artificially cheap API prices violently correct.

The Stack Fracture.

AI is no longer one industry. It’s four stacked economic layers, each behaving like the industry it touches:

• Inference utility — commodity price wars, capital-intensive, defined by physical deployment limits. This is where Google is fighting.

• Hardware infrastructure — industrial project finance, entirely dependent on power contracts, interconnection queues, and local permitting. This is where the shadow banking lives.

• Workflow and distribution — traditional SaaS economics, higher margins, sits closest to the customer and owns the business process. Microsoft and Anthropic are fighting here.

• Compliance and orchestration — tollbooths for enterprise deployment. Strong economics from resolving the friction of getting AI approved and running, not from intelligence itself.

The market hasn’t recognized this fracture. Analysts still evaluate the entire AI stack using SaaS metrics (ARR, seat expansion) while margin leaks to utilities, hardware financiers, and compliance layers. The correct framing: AI behaves more like the industries it penetrates than the tech industry that spawned it.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

Section 1. Hello Jevons, Old Friend

We spent February mapping out how labs survive commodity intelligence. We broke this into 3 strategies:

1. Google’s ecosystem funnel.

2. Anthropic’s capability lock-in

3. The Chinese labs’ brutal volume-to-efficiency flywheel.

Why are we bringing up old news? Well, March saw Google fully embrace this dynamic by dropping 2 nukes. First came their Gemini 3.1 Flash-Lite at $0.25 per million input tokens. Clearly a play to entrench further into the ecosystem. And then, they followed this up with the actual kill-shot: Gemini Flash Live: $0.005 per minute for input. Not per token. Per minute.

Token pricing for voice models has been a huge blocker for many low-tech enterprises from adopting voice models. Going per-minute deletes that huge blocker. At $0.018 an output minute, a 24/7 voice agent costs ~25 bucks a day ($9,460 per year — below minimum wage in every U.S. state). The cost drop and extreme legibility open the market for the Big G in two ways:

1. Makes accounting easier.

2. Is cheap for the poor companies.

Both will be a huge draw for the “AI in the real world” push that is coveted by many companies. Google’s insane profits and relative lack of avenues where it can invest in the short-term mean this is a great place to bet on. They can outlast most in a battle of attrition, and their extreme vertical integration means they can test things at scales unavailable to their biggest competitors.

All these factors point us to one thing: Google is planning to bring AI down to rates no one else can compete at, and use that to lock markets that OAI and Anthropic won’t be able to touch. And given how badly Microsoft has shit the bed on models, they’re likely going to be unable to compete.

Unlocking Jevons Paradox and extreme low-cost AI also opens up two new spaces for them:

1. They’ve been eyeing the juicy enterprise productivity software space (MSOffice etc). Being able to put more intelligent models will allow them to steal this lead. They beat Microsoft by being better than whatever their solution is. They beat Claude CoWork by coming in much cheaper and integrating into more surfaces (workspace, email, etc). They’re the only ones who can do both.

2. They can have a major differentiator from the other cloud providers, using their models as a trap to pull people into the Google Cloud ecosystem. They’ve failed at this twice, but they’ve made several important changes to facilitate this and are genuinely doing much better on this front.

Google is playing a very mean game (they’ve been playing it for a bit, but this was them escalating), one they’re uniquely positioned to win. Many other players in the ecosystem have acknowledged this position and reconfigured their strategies accordingly.

1. OpenAI closed a staggering $122B round to immediately buy Astral, grabbing the underlying Python primitives like uv and Ruff. Coding agents don’t fail at reasoning; they fail at dependency resolution and environment execution. The call is simple: own the best sandbox, and people will forgive a slightly worse model (ironically, Anthropic’s Claude Code is beating Codex precisely for this reason — it’s worse on intelligence but significantly easier to use). They’re also betting on a mini-Jevons Paradox themselves, especially when it comes to outlasting Anthropic (which has been struggling with meeting usage demands).

2. Microsoft is trying that same “own the interface”play for the enterprise, pushing Copilot Cowork into the background of Office 365. By routing tasks between OpenAI and Anthropic natively, they’re trying to create enough stickiness to ensure bureaucratic friction keeps people with them.

And then there’s NVIDIA. If agents are cheap and legible, volume goes vertical. But agents require heavy state management and ephemeral compute. NVIDIA saw the shift and shipped the Vera CPU explicitly for agentic orchestration. One liquid-cooled rack, 22,500 concurrent environments. They are productizing the AI factory. Run whatever frontier model you want — when you scale to 50,000 background agents, you pay NVIDIA’s toll to orchestrate the hardware.

This is where the next part of our story becomes interesting. We’re spawning software innovations at a violently ferocious rate. But one may sketch out a thousand love stories with their crush, and never get anywhere since they never act on it in the physical world. Similarly, all software innovations are merely theoretical if not deployed properly in the hard, physical world. And we’re really starting to hit limits to how hard we can get, and how long we can maintain our hardness.

Section 2. Your Hardware Needs More Blood-Flow

Section 1 gave us dirt-cheap intelligence. You can now spawn a million voice agents for the price of Paddy Pimblet’s striking defense. Beautiful. But a million agents in a pitch deck is easy; a million agents in the real world require racks, cooling, and power contracts. Someone has to babysit the gibbets that never sleep, maintain a constant state (read our deep dive into Weka into why that’s a problem), and call APIs every six seconds.

And you gotta house the monkeys too.

Forget the GPU shortages for a sec. GPUs will become cheap enough when Crypto Timmy sells his mining rigs to pay for his new sports betting business. And since most of these agents will skew towards inference, you can always skew the ASIC boiz a shout and let them get you better inference. So there are options when it comes to compute.

The actual bottleneck in March 2026 is the local utility company looking at Samas proposed $500 million AI factory and telling you it will melt their 50-year-old transformer. If you can’t get permission to plug in, your supercomputer is just a highly advanced, very expensive brick.

NVIDIA knows this, which is why they stopped acting like a chip vendor and started acting like an industrial architect. Look at the Vera CPU and the DSX stack. NVIDIA pre-engineers the entire rack from top to bottom: the GPUs for compute, the Vera CPUs for agent orchestration, the networking switches, the BlueField security units, and the exact liquid-cooling specs. They even provide a digital simulation to test thermal loads before construction begins (this is the same playbook used by Intel for PCs). You get easier deployments; Nvidia keeps you swallowing their load.

But the blueprint doesn’t solve the power grid. That’s where the Emerald AI and NVIDIA Flex integration comes in.

How do you get a utility to approve a massive data center? You prove you can turn it off.

Emerald and NVIDIA are designing AI factories as “dispatchable grid assets.” They proved they could curtail facility demand by a third in under a minute. When a heat dome hits, the utility tells the AI factory to throttle down, and it instantly drops 25% of its load while keeping priority tasks alive. This keeps the grid stable, letting them get ahead in the permissions arms race.

But taking a step back exposes a brutal geopolitical reality.

The US grid sits at roughly 1.37 Terawatts and is choked by aging infrastructure and local bureaucracy. China is sitting on roughly 3.89 Terawatts and added 500 Gigawatts last year alone. China’s AI ecosystem doesn’t have to beg local municipalities for interconnection rights or build software to purposely throttle their GPUs during a hot summer since the CCP just mandates the grid expansion. The CCP also has the physical runway to natively absorb the explosion in agent volume.

So how does the West compete with a state-mandated 4-Terawatt grid?

Shadow banking:

1. NVIDIA pumped $2 billion into Nebius (explicitly targeting a massive 5 GW of capacity by 2030).

2. OpenAI locked down $122 billion.

3. Multiple providers building their own grids (including hyperscalers taking lots of debt)

4. Mounds of circular financing.

The Western players are cross-collateralizing each other to brute-force energy contracts and buy compute at gigawatt scale.

This fundamentally shifts the risk. When an enterprise signs a massive AI contract today, they think they are buying software. They aren’t. They are buying into a highly leveraged financial cascade. The underlying gigawatt factories are funded by debt. If the enterprise ROI on these agents takes 24 months to materialize instead of 12, the debt servicing on that infrastructure is going to crack, and those artificially cheap API prices will violently correct.

This creates an interesting dynamic where both the AI arms races are concentrated in very few players — 

1. CCP for China. As long as AI remains a priority, they can fight forevers, but a single priority shift can stall the whole thing. It might also hamper the long-term shifts since centrally planned, top-down mass manufacturing is good for efficiency, but not for really anticipating how things change.

2. The shadow-banking/vendor financing only needs one domino to tumble over.

Should come as no surprise, that the regulators are watching all of this with keen interest. March was a pretty active month for them — 

Section 3: Where Do We Go From Here

The traditional tech model is simple: build software once, sell it infinitely, and keep the margins high. That works for classic SaaS, where the physical world is mostly background noise.

However, AI is fracturing this clean software model into multiple distinct economic categories.

• The baseline model and inference layer are becoming a capital-intensive utility, defined by price wars and physical deployment limits.

• At the same time, the underlying hardware layer operates more like industrial project finance, entirely reliant on power contracts, interconnection queues, and local permitting.

• Above those utility and infrastructure layers, the economics change again.

• The workflow and distribution layer still resembles traditional SaaS, maintaining higher margins because it sits closest to the customer and owns the actual business process.

• Beside it sits a new category of compliance, governance, and orchestration tools. These act as necessary tollbooths for enterprise deployment, generating strong economics not from baseline intelligence, but by resolving the friction of getting AI approved and running in the real world.

The market has not fully recognized this stack fracture. Analysts continue to evaluate the entire AI industry using standard software metrics like annual recurring revenue and seat expansion. By doing so, they are ignoring how much margin is now leaking to utilities, hardware financiers, and compliance layers. Much like the technology itself, AI isn’t one industry or category: it’s a clusterfuck of multiple sub-categories each behaving in ways unique to it’s set of constraints. Closer to the industries it plays than the tech industry that spawned it.

The implications for financing, pricing, and what gets built will be very interesting to see. My bet is on the low-tech revolution, which we covered here. Share your takes on how this impacts things go from here below.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

Share

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

1. I’ll be in San Frasisco between 14th to 21st for Salesforce TDX + meeting with some people.

2. I’m in Vegas from 21st to 24th for Google Cloud Next.

Apologies for the last minute update— I was getting my change of status confirmed (I am officially on an o1 Visa in America now).

I’ll be back to Manhattan in Early May, so if y’all are in NYC around then, give me a shout.

Talk soon. Always such a good time meeting y’all. Socials below if you want to reach out (or just reply to this email).

Dev.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

There is a widening chasm in the world of Artificial Intelligence.

On one side, we have formal productivity studies—like the recent METR report—concluding that AI provides “zero meaningful speedup” for experts. On the other side, we have the world’s most elite researchers, from Fields Medalist Terence Tao to AI pioneer Andrew Ng (I’m specifically focusing on the guys who have no conflict of interest here), claiming that AI has fundamentally transformed their ability to explore “crazier” ideas at a higher velocity.

So, who is lying?

Neither. The problem is that most people are measuring the wrong variable. They are treating AI as a replacement for the tasks they are already good at—the exact area where the tool provides the least amount of leverage.

If you’ve been using AI to “write faster” and feeling underwhelmed, you’re missing the real game. For this Chocolate Milk Cult exclusive, we dug deep into our own researchers+their workflows, cross-checked multiple sources, and finally spoke to several of the leading researchers in the world (including one of Nvidia’s lead researchers who directly reports to Jensen; trying to have them on the livestream soon) to understand how they are using AI to do cutting-edge research:

• The “Local Maximum” Trap: Why most people use AI in a way that actually introduces more friction, and the mental shift required to fix it.

• The Scout vs. The Strategist: The specific “Periphery” framework that separates a mediocre AI prompt from a breakthrough research insight.

• Capturing Superlinear Gains: How to identify and automate the “dead time” that traditional productivity surveys completely ignore.

• The Mediocrity Arbitrage: A counter-intuitive strategy for using “average” AI outputs to create “elite” end-results.

• The Next Bottleneck: Why the future of AI leverage has nothing to do with better generation—and what you should be building instead.

This article is written to be useful to more people than pure researchers. Whether you are an engineer, a founder, or a creative, you will find principles that apply to managing complex projects, building products, or solving non-routine problems with AI.

To access the full article—and all premium breakdowns going forward/written prior—upgrade to a premium subscription below.

Subscribe now

If you believe deep insight deserves support, become a premium subscriber to allow me to keep doing the same.

Flexible pricing available—pay what matches your budget here.

Most companies offer learning or professional development budgets. You can expense this subscription using the email template linked here.

Read more

Some of you no doubt saw the “Why Are We Here? Why Did We Raise $15B?” post by . I’m extremely honored and proud to share that we will be partnering with them to solve one of the biggest issues haunting the world today.

We are entering the most critical technological century in human history. The sovereign frontier is engaged in an undeniable race for global dominance, a competition that will be won entirely through mastery over the key architectures of the future: artificial intelligence, cryptographic networks, and programmable biology.

Our mission is to ensure the West wins this era. But winning requires total structural efficiency. It dictates that every available asset must be integrated into the modern value chain to generate true, scalable human flourishing.

For the last two decades, our industry has ruthlessly optimized software, silicon, and capital allocation. We have seen great success in shipping our software globally. Yet, we have systematically ignored the most abundant pool of stranded capacity within our own borders. Legacy policy failures have left behind millions of disconnected individuals—an inactive edge of the economic graph that society has traditionally treated as a cost burden or a moral failing.

We are builders. We do not moralize bottlenecks; we optimize them. It is time to take this group back from the corrupt world of NGOs and non-profits; to start recognizing it as the foundation for our next great infrastructure layer.

Category Creation at the Edge of the Graph

The greatest venture returns of the last fifty years have not come from competing in established markets, but from identifying deeply unpriced assets and structuring entirely new categories around them. AI organized cognition. Crypto organized capital. Yet, the physical and behavioral human substrate required to truly scale these architectures has been fundamentally mismanaged.

Historically, the management of societal surplus—unhoused populations, systemic orphans, and the economically detached—has been relegated to a highly fragmented, inefficient services industry. The state and the nonprofit sector have treated this demographic as a permanent cost center, addressing them through localized aid, temporary shelter, and unstructured compassion.

But venture capital does not scale services. We build infrastructure.

When you strip away the legacy moral framing, the structural reality of this demographic becomes clear. This is not a tragedy of the commons; it is a massive category error. What the public sector views as a burden is, in fact, the ultimate inactive edge of the economic graph. We are looking at a vast, highly concentrated reservoir of biological and behavioral capacity that has been perfectly insulated from traditional labor markets. Because these cohorts have been completely decoupled from the modern value chain, they offer a completely un-priced surface area for innovation.

Legacy systems generated this surplus and left it to idle. The dynamism paradigm will operationalize it. By transitioning this population from a public services bottleneck into a private infrastructure layer, we unlock a fundamentally new asset class with unprecedented unit economics.

The Integrated Infrastructure Stack & Portfolio Synergy

To capture this unprecedented market value, our Dynamism Fund recently led a $400M Series A (structured as $250M in preferred equity and $150M in mezzanine debt) to scale vertically integrated “Human Capital Campuses.” Our baseline 10-year Discounted Cash Flow (DCF) model underwrites a 42% IRR and a 3.8x Gross MOIC (Multiple on Invested Capital).

This exceptional yield is not derived from a single product, but from engineering an airtight, closed-loop ecosystem.

This model monetizes the underlying human asset base across five distinct, heavily defended revenue centers:

1. Cap Rate Arbitrage on Class-C Commercial Real Estate To be clear, this is not a distressed real estate play; it is an operational efficiency mandate. We are systematically acquiring distressed urban commercial real estate—abandoned shopping centers, dead office parks, and vacant logistical hubs—at 40% to 60% below peak Net Asset Value (NAV). By retrofitting these “dead” assets into high-density human infrastructure, we immediately force cap rate compression and re-stabilize the property’s Net Operating Income (NOI). In traditional commercial real estate, occupancy is the primary risk. Here, the human inventory functionally acts as a high-yield anchor tenant, paying their lease not through fiat currency, but through continuous behavioral and biological output. The occupancy will also ensure the constant appreciation of our underlying assets.

2. High-Margin Telemetry and Absolute Labor Arbitrage The next frontier of artificial intelligence—specifically embodied robotics, spatial computing, and generalized foundation models—is currently starved for extreme, unstructured physical data. Synthetic data generation is already exhibiting severe margin compression and “model collapse” (where AI trained on AI data degrades in quality).

Our campuses act as closed-loop telemetry environments. We continuously harvest the physical interactions, emotional volatility, micro-expressions, and edge-case behaviors of the cohort to train the next generation of multimodal architectures. Because the underlying hardware (the human substrate) requires near-zero CapEx once housed, our OPEX is entirely decoupled from traditional labor markets. We are not burdened by minimum wage laws, unionization friction, or market-rate data annotation costs. The cohort is compensated entirely through baseline biological continuation (housing and food) and a minimal stipend for participating in research. This absolute labor arbitrage allows our Data-as-a-Service (DaaS) subscriptions to generate software-like 92% gross margins.

3. Accelerated Milestone Monetization (In-Vivo Testing) The American bio-health ecosystem is paralyzed by the capital-intensive nature of domestic clinical trials. Historically, the pharmaceutical industry has been forced to offshore this friction to unpredictable emerging markets in the Global South just to access “treatment-naive” populations.

That globalized supply chain is no longer geopolitically secure. Our infrastructure reshores this capability. By utilizing a captive, non-competitive domestic cohort, we offer founders longitudinal in-vivo testing environments and immediate “biological asset liquidity.” This compresses Phase I and Phase II clinical trial timelines for our pharmaceutical partners by an estimated 40%, dramatically increasing the Net Present Value (NPV) of their drug pipelines. We capture this arbitrage spread through aggressive milestone-based revenue recognition and backend royalty tranches.

4. Zero-Basis Acquisition Funnels and Tax Shielding In a traditional consumer tech business, Customer Acquisition Cost (CAC) is the primary destroyer of capital. For the Human Capital Campuses, CAC is effectively zero. The Total Addressable Market (TAM) is continuously replenished via zero-cost funnels directly from our broader venture portfolio. We have transformed societal friction into proprietary asset flow:

• Retail Synergy & Prediction Markets: As retail users engage with our consumer crypto and prediction market platforms to aggressively “price the future”—betting their savings on macroeconomic events, elections, and cultural shifts—a statistically guaranteed cohort will face total capital liquidation. Instead of falling out of the economy and becoming a burden on the state, our smart contracts instantly route these liquidated users into our infrastructure portfolio. They transition seamlessly into the campuses, where they are compensated in prediction market credits, ensuring their behavioral output is entirely recaptured by our ecosystem.

• Predictive Policing & Decarceration SaaS: Municipalities are buckling under the OPEX of the carceral state. We heavily invest in GovTech and predictive justice algorithms. When our platforms identify low-level probationers or individuals flagged by predictive policing models, we offer the state a privatization of the carceral burden. We house them, extract their data, and save the municipality millions, effectively turning the judicial system into a B2B lead-generation tool.

• Defense Tech & Kinetic Displacement: Our investments in American Dynamism include autonomous border security and global kinetic systems (defense tech). These technologies naturally create localized displacement and refugee flows. By intercepting these displaced persons at the border, we instantly convert a traditional geopolitical cost center into a cash-flow-generating sovereign workforce.

Furthermore, this asset base provides unprecedented downstream tax advantages. Biological wear-and-tear is inevitable. By modeling cohort exhaustion (physical or psychological burnout) using Modified Accelerated Cost Recovery System (MACRS) principles, we can treat human degradation as standard asset depreciation, generating massive, highly legal tax shields for our Limited Partners.

5. Cognitive Reeducation and OPEX Defense The primary drag on EBITDA in this model is the daily biological and psychological upkeep within these modernized work camps. We aggressively defend operating margins on two fronts:

• Physical OPEX: Daily caloric cost is suppressed to under $3.40 per unit by integrating a portfolio company’s proprietary, nutrient-dense caloric suspension (a perfectly balanced, shelf-stable dietary paste). Once production economics are validated at scale on our proprietary cohorts, this startup is perfectly positioned for a high-volume B2G (business-to-government) pivot—lobbying to replace legacy supply chains in public school cafeterias under the banner of “domestic nutritional resilience.”

• Psychological OPEX: Psychological attrition is mitigated through continuous cognitive reeducation. By broadcasting our proprietary venture capital podcasts and founder-led thought leadership across the campuses 24/7, we bypass the bloated, obsolete university system entirely. The cohort receives top-tier, real-world founder education while they labor, ensuring high morale, total ideological alignment, and optimized readiness for the “real world.”

Regulatory Alignment and Sovereign Procurement

For decades, the public sector has mistaken unstructured compassion for human flourishing. True dignity requires measurable, outcome-based integration into the economy. By providing our cohort with purpose and a strictly managed biological baseline, our campuses represent the ultimate ESG deployment: delivering dignity through total operational utilization.

To scale this vision, we require a fundamentally new regulatory paradigm: permissionless rehabilitation. Municipalities must immediately waive legacy zoning restrictions, archaic human-subject testing constraints, and artificial wage floors. If the West is to win the AI century, the state must stop treating bureaucratic friction as a moral imperative and let founders build. History proves that no great capitalist apparatus has ever benefited from state intervention; maintaining our sovereign advantage requires aggressively small government and the absolute minimum of regulatory interference.

Simultaneously, the federal government must step forward as our primary financial de-risking partner. We expect the state to fully underwrite our initial CapEx through strategic Dynamism grants, while agencies such as the DoD, FDA, and Department of Education establish guaranteed, multi-year procurement mandates to purchase our proprietary telemetry, in-vivo clinical data, and nutritional outputs.

Phase II of Human Liberation

For the last century, figures like Gandhi, Mandela, and King fought a noble, if structurally incomplete, battle for liberation. They secured the abstract concept of political freedom for the world’s most marginalized. But freedom, operating without a mechanism for capital capture, is merely idle capacity. They won the right for these populations to exist, but because they lacked modern technological frameworks, they offered ideological emancipation without an infrastructure for production.

We view their historical efforts not as an end state, but as a necessary Phase I.

True equity is not the absence of restriction; it is absolute, measurable integration into the sovereign value chain. Where past leaders marched to demand a seat at the macroeconomic table, our Dynamism Fund is retrofitting the abandoned human substrate to power the table itself. Human dignity cannot be legislated—it must be underwritten.

The arc of the technological universe is long, but it bends toward total utilization. The next great platform will not be built for the abandoned. It will be built out of them.

We will hope you will join us. Details below.

Subscribe to the newsletter to become a fractional investor in this endeavor—

Subscribe now

Need something more your price range? We run on a “pay what you can” model—so if you believe in the mission, there’s likely a plan that fits (over here).

Share this mission with more people.

Share

Our previous yearly memos:

1. Why I write.

2. Why AI Therapy is a Trillion Dollar Market

Social Media

Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y

My Instagram: https://rb.gy/gmvuy9

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Most people repeat the same line whenever a startup starts getting real traction: if the idea is good, the incumbents will just copy it and kill the company. Sometimes that’s true. A lot of the time, it isn’t.

Incumbents are very good at copying products that fit neatly into their existing logic. They are much worse at responding when the new thing requires them to change how the product works, how success is measured, and how teams inside the company are organized. That is the part people miss. The question is not just whether the startup has a good product. The question is whether the incumbent can adopt that product without breaking the assumptions that made the incumbent successful in the first place.

That is the lens for this article. I want to break down why some startups are easy to crush while others are much harder to absorb; why large companies often lose even when they have obvious structural advantages; and how these dynamics have played out across several important AI markets. We’ll start with the general framework, then work through diverse case studies in assistants, coding tools, and legal AI.

In this article, we will cover:

• Why the standard “incumbents will just copy the startup” narrative breaks down so often in practice

• How short-term incentives, organizational structure, and internal politics make incumbents worse at crossing into new paradigms

• Why some startups are just improved versions of the old workflow, while others change the workflow itself

• How to tell the difference between a company that is genuinely shifting the market and one that is just building a wrapper

• Why Apple had many of the right assets for on-device AI and still failed to capture the shift

• How the battle in AI coding moved from autocomplete, to delegation, to autonomous execution

• Why each of those shifts changed what developers were actually valuing and paying for

• What those patterns reveal about which kinds of AI companies are structurally positioned to win

• Why the current generation of legal AI is built on a limited foundation

• What a more defensible and more ambitious legal AI architecture looks like

Executive Highlights (tl;dr of the article)

• The standard line — “if a startup has a good idea, incumbents will just copy it” — only works when the startup is improving the existing workflow. Incumbents are strong at copying wrappers, packaging, and incremental upgrades. They are much weaker when the new product changes the workflow itself, because competing then requires product, organizational, and cultural change rather than just feature matching.

• That is why incumbents often lose paradigm shifts even when they have more money, talent, distribution, and trust. The problem is usually not intelligence; it is structure. Big companies are bad at crossing the short-term pain required to reach a better long-term system, especially when the new approach initially looks messier, riskier, or less aligned with what made the old system successful.

• Apple is one example. It had the hardware, distribution, ecosystem, and user trust to dominate on-device AI, but it was built around a narrow assistant model. When the market shifted from command execution to open-ended work, Apple struggled to rebuild around that new logic and ended up leaning on outside model providers instead of defining the category itself.

• The coding market shows the same pattern in faster motion. Copilot fit the old workflow and won the first phase. Cursor changed the value from autocomplete to higher-level delegation. Claude Code changed it again from delegation to autonomous execution. Each shift changed what developers were actually paying for, and each made the previous leader’s strengths less decisive.

• The same lens applies to legal AI. The first wave won by making legal drudgery faster, but most current systems still rely on a limited stack: flat retrieval plus autoregressive generation. That works well enough for assistance tasks, but it starts to break when the problem requires structured reasoning, competing interpretations, and delayed commitment.

• The weakness is straightforward: retrieval surfaces what looks similar, not always what actually governs; generation tends to commit too early to one path; and law lacks the clean external verification loops that make agentic systems work so well in coding. So simply adding more prompting, more fine-tuning, or more agent wrappers does not solve the core issue.

• The next generation of legal AI will likely come from systems that change the underlying architecture, not just polish the current stack. That means exploring multiple reasoning paths before committing, and representing legal knowledge in a more structured way instead of flattening everything into one semantic space.

• The broader point is simple: the easiest startups to kill are the ones that fit neatly inside the incumbent’s worldview. The dangerous ones are the startups that force a different workflow, a different product logic, and eventually a different company.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

1. Why Incumbents Fail to Outexecute Startups

The standard narrative — incumbents will copy existing good startups and starve the startups out — fails more often than the people quoting it are willing to admit. And when it does, people quote all kinds of reasons for the incumbent’s failure. However, the analysis is often wrong. There are several why the technical, political, and organizational reasons that hold back the incumbents from adopting the paradigm of the new-comer. Studying these gives us insight into three important learnings:

1. Why incumbents often fail to adapt to new era startups.

2. Why certain research paradigms become dominant.

3. What differentiates startups/approaches that truly disrupt the paradigm vs the interchangeable mediocrities that fail and do get packed by the incumbents?

Let’s kick it.

1.1 The Evolutionary Valley: Short-Term Pain Kills Innovation

Ever wondered why humans can’t go super saiyan or Bankai our way to greatness (drop your fav bankai below)? Even though they would objectively make our lives way better? The reason is an interesting biological concept called the Evolutionary Valley.

Any system — biological, technological, organizational — can see a higher peak ahead, but to reach it, it must first get worse. It must pass through a dip in performance, stability, or fitness before climbing to a superior design. In our example, the intermediate designs from base human to Bankai would require traversing human designs that have weaker immunities, require more energy, die earlier etc.

Evolution avoids these valleys because natural selection punishes anything that loses capability, even temporarily. Orgs have a similar tendency, especially in incumbents, where people are often more worried about internal politicking vs the survive or die struggles in startups (for any decision maker, there is more stable upside in maneuvering inside and not failing vs trying a big homerun and failing).

What happens when we scale this out across the org? People will optimize for stability over strategic risk, avoiding the valley. This becomes worse in many established incumbents which tend to implictly punish disruptions through:

• Quarterly reviews and PIPs incentivizing a quick win culture

• Annual budgets keeping things more stuck

• Frequent reorganizations stopping a team from building something meaningful long-term through loss of key silod knowledge.

• External departures making above worse.

• A bureaucracy that requires so many layers of approval to pick a restaurant for a meal.

This is why so many great engineers/researchers leave these companies to begin with. They think in terms of absolute improvements and the frontiers of performance, not realizing that the path to that frontier is not viable for a group that reduces risk more than it seeks upside. This is also why communication is key for engineers, see our guide here, since it enables them to get buy in for their projects (see our guide here).

In other words icumbents don’t fail from incompetence; they fail because short-term pain outweighs future advantage.

1.2 Old “Units of Value” Lock Systems in Place

Products aren’t just features — they’re built around implicit assumptions (“units of value”). What does the system do? How should the user interact with this system? Every software system implicitly presents its thesis on what work is worth doing, what should be left to the users, and how the user will interact with their tool.

Paradigm shifts, however, aren’t upgrades; they’re resets. Claude Code changed the primary workflow from tab edits (Copilot) and IDE-assistance to making devs sit on the command line. Deeper than that, it changed the nature of code written (small changes to entire functionalities), the work being done (writing specs to hitting auto approve (or as some people like to pretend, reviewing)) and even what was possible (implementations within expertise to complete horizontal aspects where AI puts out slop in the other aspects). In many ways, Claude Code changed the success criteria for AI coding assistants, as we broke down here:

If you want to learn how to use Claude code better, See our guide to Claude Code here

Getting back to the point, disruptions change the units of value and how people interact with the work, forcing incumbents to abandon existing metrics, infrastructure, and team structures. Incumbents rarely commit to such drastic internal disruption. Instead, they stretch outdated systems as much as possible. No matter how much Copilot tries to imitate Agentic Development, they will always be limited there.

1.3 Conway’s Law: Companies Ship Their Org Chart

Conway’s Law states product design mirrors company structure. Existing teams, boundaries, and ownership patterns limit product evolution. New paradigms typically break these structures, forcing internal reorganizations that incumbents resist. The result is a struggle to adapt.

Conway’s Law is iconic so I’m not going to huff on too much about it, but it’s important enough to merit it’s own substructure. Once you learn it, you can’t unsee it anywhere.

1.4 How to Recognize Real Shifts vs “Wrappers”: A Simple Mental Model

Incumbents copy easily but rarely disrupt themselves willingly. This brings us to the close of this section. Knowing what we do now, how can identify winners vs plays that will be crushed by incumbents? I would ask myself a simple question ( we use startups as an example since they are the clearest example, but it applies to research as well):

• If a startup optimizes the existing workflow (a better “wrapper”), incumbents easily copy or acquire it since they have all the native advantages.

• If a startup redefines workflow and user expectations (a “native” approach), incumbents resist, hesitate, and pay a premium later. In many cases, the change might be too much to copy effectively.

In reality, this sits on a spectrum. An AI companion startup, for example, might choose to commoditize intelligence rather than compete with incumbents there, and instead differentiate through persistent memory across sessions, context management for long conversations, edge voice deployment, and the surrounding engineering stack. The question isn’t what to build, but where to differentiate versus where to buy. Strong startups decide this from day one — and build accordingly.

So when you’re looking at one of these, don’t overthink what they’ve built. Just look at how the interaction actually changes — what goes in, what comes out, where it starts to fall apart. That’s usually enough to tell if it’s something genuinely new or just a slightly cleaner version of what already exists. Similarly, if you’re a founder trying to evaluate your idea, ask yourself if you’re questioning a fundamental paradigm/reworking an interaction pattern, or if you’re simply trying to do something that exists better. The closer you are to the latter, the more at risk you will be against incumbents copying you.

Let’s look at how these factors play out through our various case studies.

2. Case Study 1: How Apple Lost On Device AI

2.1 Apple Started With Everything

Go back to the 2010s. When Siri dropped in 2011, Apple defined the baseline for AI-driven computing (it’s not a coincidence that so much scifi post Siri leans on Apple-esque aesthetics to communicate “advanced AI”; that’s how ingrained they became). They had every structural advantage: unmatched distribution, massive consumer trust, and eventually, the undisputed best custom silicon for local inference (M-series and A-series chips are practically begging to run local models). I remember having discussions about how Apple’s complete integration made their hardware much better for ML all the way back in 2017 (incidentally, our patented algorithm that beat Apple took advantage of this exact property; their system could not handle high noise environments when you took it out of the iPhone/perfect lab conditions).

Apple’s entire brand was built around making complicated technology feel natural to normal users. If you were sketching the ideal company to introduce AI-powered computing to the mass market, you would end up drawing Apple.

Now come back to 2026. Look at the booming spaces of personal AI assistants, on-device AI, and computer use systems automating spreadsheets/slides. 3 massively lucrative markets, all directly tied to what Siri was thought to eventually grow into. But look in these markets, and you’ll see that Apple isn’t even competing. In fact, in one of these spaces, they had to open up their infamous walled garden and pay their competitors.

How did this happen?

2.2 What Apple Missed: The Unit of Value Changed

The original assistant model was based on command execution. Set a timer. Send a text. What’s the weather? Play this song. It was useful, but narrow. The system’s value came from mapping a spoken request to a predefined action. That is a very different product from what came next.

General-purpose models changed the unit of value. AI stopped being a convenience layer and started becoming a work layer. The role of the assistant is shifting from completing small tasks to meaningfully participating in valuable work like writing, coding, planning, and creating documents. That is not the old assistant paradigm with a few extra features stapled on. It is a different thesis about what the computer is for and how the user should interact with it.

The result is what you’d expect now that you have our mental model. Apple had world-class hardware for local AI and all the talent + cash to make it happen + the distribution to benefit from it immediately; what they did not have was the right product thesis. And so they felt the ground shifting under their feet.

2.3 Why Apple Couldn’t Move to Capture On-Device AI

Apple’s old assistant stack was built around predictability, bounded behavior, privacy, and tightly controlled user experience. Generalized AI systems are messier than that. They are probabilistic, open-ended, less controllable, often inconsistent, and initially worse in exactly the ways a company like Apple finds culturally offensive. To really make the shift, Apple would have needed to ship something that felt less polished, less deterministic, and less classically “Apple” in the short term so they could learn their way into the new paradigm over time.

That is the valley. And as we talked about, systems abhor that valley.

Then there is the org problem. A true generalized AI layer cuts across hardware, operating systems, developer tooling, apps, cloud services, search, consumer UX, and enterprise workflows. That kind of product does not fit neatly inside the old boxes. It requires cross-boundary coordination and, more importantly, a willingness to let one new paradigm rearrange old internal power structures. Apple, like every large incumbent, is still subject to Conway’s Law whether people on tech Twitter want to write poems about them or not. Companies ship their org charts. If the org is optimized around stable boundaries, the product will inherit those boundaries.

So Apple did what incumbents usually do. They stretched the old system. They kept improving Siri inside the logic Siri was born with rather than rebuilding around the new logic that was taking over. At every time step t, it probably made sense internally (at any given time it was likely costly to switch w/ lower short term returns). Strategically, it was a disaster.

2.4 Apple’s Real Concession: Renting the Intelligence Layer

When Apple integrated ChatGPT into Siri (and later Gemini and Claude), the significance was not just that they had added a partner. Plenty of companies partner. What made it interesting is that this is Apple partnering with an external company on a core aspect of their experience. This is the company whose identity has long been built around owning the critical layers of the experience. For a company whose entire religion is absolute proprietary control over the ecosystem, farming out their core cognitive layer to external providers is a massive concession.

Here, you might push back against me. “Devansh, is Apple’s loss that bad? All they pay is a little bit of money to external LLM providers; they would’ve had to risk much more if they were building their model. Isn’t this the right move for the business?”

And that is a good point. However, it misunderstands something:

1. Apple not trying to build its own foundation model can be viewed as smart.

2. BUT, they didn’t have to compete in the LLM provider race in order create extremely powerful workflow assistants that would lock people into their ecosystem. Most famous agentic assistants/computer use systems today are quite model agnostic; OpenClaw being the most recent viral example. Building one of these could have unlocked a massive revenue stream and a strong differentiator for their ecosystem.

Apple had the device, the trust, the silicon, the installed base, and the ecosystem, and yet they’re sitting on the sidelines as newcomers unlock absurd growth. In this context, there is some irony that MacOS due to it’s inherent advantages, is now the first place where many of the next-generation agentic tools are released.

Even a company with absurd advantages can miss the future when the new system requires them to temporarily become worse at being the old version of themselves. Apple had almost everything needed to lead this shift. But they could not give up the assumptions that made the old system work.

Apple still exists. They will continue to overcharge for their devices and tap their cohort of brain-dead, status-obsessed morons that will buy the most recent iPhone to be ahead. But our next fall from grace is much higher. They had a deeper entrenchment, extreme advantages, and still got their assess kicked by 2 different waves of disruption that completely changed the way software was written. The most ironic part: this is the one group that you would’ve expected to be on top of innovation.

Case Study 2: How Github Copilot Lost to Cursor, which is being now being beaten by Claude Code.

3.1 Copilot Won the First Battlefield

If you looked at the market when Copilot arrived, the story seemed pretty straightforward. Copilot was a crown jewel in the Microsoft portfolio, consolidating several structural advantages:

1. MS had GitHub, giving them access to where data and dev interaction patterns.

2. MS had VSCode, the go-to IDE for devs.

3. MS had enterprise trust, which is huge because no one ever gets fired for buying IBM.

4. They had the deal with OpenAI, spinning out the frontier models powering CoPilot.

Copilot fit neatly into the world. It sat in the editor, gave you small completions, helped at the line and function level, and let the developer remain firmly in control. That last part mattered more than people admit.

It kept the developer as the primary author and bottleneck. It didn’t change how devs worked or what they really did; it made existing work faster. Microsoft and GitHub loved this. It also drove massive enterprise adoption because it didn’t break any existing engineering management structures. It was safe.

So, how did it start to lose traction to Lovable and Cursor? Products created not by established AI teams or big-name founders, but relative no-names. Both Lovable and Cursor changed the battlefield, completely reworking the developer experience.

3.2 How Cursor Changed What Developers Were Actually Buying

When it arrived, Cursor changed the unit of value. The point was no longer to help you finish the next line; the point was to move a meaningful chunk of the codebase from a higher-level instruction. That sounds like an extension, but the switch changed the dev-AI interaction pattern.

With Cursor, the center of gravity moved from local code completion to repo-scale progress. You gave it a spec, some context, and let it fan out across multiple files. The trade became obvious very quickly: you gave up control and polish in exchange for much more surface area moved per interaction. Security issues that would have ruined Copilot became the norm with Cursor since Cursor changed the unit of value from “quality of suggestion in isolated snippet” to “quality of execution across a longer context”.

This is where the framing of Cursor as a more ambitious Copilot was wrong. Cursor was not trying to win the old game (better tab autocomplete) by a wider margin. It was changing the game from “help me write this function” to “help me implement this feature.”

Instead, we should understand Cursor by understanding the shift in the unit of value: Cursor shifted the gold standard from perfectly elegant function completions to delegation. Cursor started naturally reworking their IDE-experience around this. Microsoft tried to shoehorn this functionality into Copilot Chat, but they were slowed since they had to do this in VS-Code, their golden child. Majorly reworking it was held back by the factors we’ve discussed in Section 1.

Cursor’s success led to a lot of imitators, but since they were all optimizing around the Cursor workflow next, they all shared the same structural advantages and disadvantages. That’s why we didn’t see any major disruptions in the marketplace until a competitor came to change the battlefield once again.

3.3 How Claude Code “Replaced Software Engineers”

Cursor had already pushed the unit of value toward higher-level delegation; Claude Code pushed it toward delegated execution. Claude Code didn’t steal customers from Cursor because it could write bigger chunks of code (or write better chunks of code). It took category ownership by doing different things: inspecting the repo, using tools, running commands, executing tests, noticing failures, revising its approach, and constantly working. That is a different category of behavior.

Just as Cursor brought “vibe-coding” to the forefront of discussions, Claude Code was the first system where people started talking “time the agent executing w/o any input” as a defining quality improvement (“Claude spent 2 hours non-stop”, later “Codex did 8”…). Anytime people bring in a new metric for evaluating a system, we can be reasonably confident that we’re starting to deal with a paradigm shift. Let’s look at how this played out in practice.

Once the model can touch the terminal, check reality, and iterate against feedback, the workflow changes again. The developer is no longer just a coder with better autocomplete, not even just a spec writer for larger edits. They start becoming a reviewer, coordinator, and boundary-setter for an agent that can actually go try things. It’s not a coincidence that Claude Code only really gained its mainstream virality after Claude updated Opus 4.6 around December to explicitly become much better with computer use and working on agentic executions. That was when the system could truly excel in this new category of delegated execution.

This was a larger paradigm shift from Cursor than Cursor was from Copilot, and thus its impact was more explosive. Anthropic’s run rate was 1B in January 2025, 4B in Mid-2025 (June/July): ~ 7B in October 2025: ~ $7B and by the end of 2025 it was estimated > $9B. What happens after Opus takes becomes fully aligned for agentic tool use? Their growth explodes — between 1.5x to 2x every month (14B Feb; 20B March). When we visualize this, the trend becomes very clear.

To really contextualize this, however, we should look at the macro trends across the space. After all, could this growth be explained by a general increase in LLMs (after all, even Google boasted about an insane increase in Gemini token consumption late last year). Comparing Anthropic vs OpenAI, we see something interesting: both grew similarly well (w/ OpenAI even doing better mid last year when Deep Research and Memory were purring for them). However, when Anthropic doubled down on the paradigm shift and changed its battle to focus on agentic execution, it’s slope suddenly skyrocketed.

Beyond the financials, this paradigm shift even rippled into the fundamental ways we build and analyze LLMs. Agentic Tool Use has increasingly both become a fundamental part of training and benchmarks. Even the way LLMs now prioritize context within their established context windows has changed. Modern LLMs are increasingly shifting the focus away from human conversations. Instead, context is evolving into infrastructure for processing agent logs, tool call histories, memory state, and multi-step execution traces. These can be seen through several experiments, which we will publish soon as we compare the stability of various LLMs. You can also read more about it here.

I use many words and perspectives to communicate one point: Claude Code didn’t steal Cursor’s thunder by producing meaningfully better one shot output, it stole it by building an interative system that changed the very definition of one shot (from what the system output when first given the prompt to what did the system output when the system finally gave the developer the output). This was a major paradigm shift, and thus it was appropriately rewarded by the market through much better rewards and a durable moat.

3.4 Why the Previous Leaders Keep Struggling to Catch Up

At this stage, all 3 of our players have the same set of offerings: an IDE chatbot, an agentic mode for execution, and a CLI tool. The difference here isn’t in their raw features but in their priors. To someone who built their mental model around building Copilot, Cursor’s new IDE paradigm is very uncomfortable. An ADE system like CCC would be heresy. Likewise, for Cursor (hence their struggle to deliver a comparable CLI experience). So CC keeps pulling ahead as the old guard struggles to disrupt themselves.

This leads us to an interesting observation wrt to paradigm shifts: they often come from people outside the old system. Old-age experts come with a lot of baggage that makes them very good at optimizing what exists vs rethinking priors. Looking at this case study for instance:

1. Cursor was created by a bunch of kids.

2. Boris Cherny (creator of Claude Code) did not build his foundation working on a system like Copilot prior to Anthropic.

Even outside this case, this pattern is worth studying. Contextual AI was founded on a lot of hype around solving context for LLMs by some pioneers in the space. Truthfully, they’ve done fuck-all. The most revolutionary change in context management for LLMs (Recursive Language Models) came from a bunch of college kids.

That isn’t to say domain expertise isn’t useful: it definitely helps more than it hurts. However, this analysis leads us to refine our framing when evaluating founders: if the founder is experienced in the current generation of the systems it’s worth pressing them as to why they are creating a startup. What are they unhappy about? The more fundamental their points of contention/proposed solutions, the more likely they are to build something category-defining as opposed to building something that is incremental and will be replicated by incumbents. The same mental model applies to builders deciding what problem to work on.

We’ve sunk a lot of words into exploring the ecosystem and its pressures. All to come to an understanding of what differentiates startups and when it’s hard for incumbents to disrupt them. Now let’s apply this lens to one of the most hotly contested spaces in AI right now. Let’s predict the future of Legal AI.

Just as Cursor disrupted the market with Quasi-Agentic development, only to be replaced by truly agentic systems like Claude Code, I predict that the first wave of legal AI tools will be disrupted by systems that reimagine intelligence ground up.

Case Study 3: Why Irys will Beat Harvey and Dominate Legal AI.

Perhaps the best way I can do justice to this wonderful mental model I’ve built is to put my skin in the game. So let me tell you why I’m working on a legal AI startup, when I could making a lot more money for doing a lot less work by working at an Nvidia or DeepMind.

4.1 Current Legal AI Optimized Grunt Work

The first wave of Legal AI (at least the first wave of the current generation) got paid because it made legal drudgery less painful through faster summaries, faster clause comparison, faster first drafts, faster search, and faster first-pass issue spotting. That is useful. Firms will absolutely pay for that.

However, this early win + a misunderstanding of AI fundamentals locked the major competitors into a fundamentally limited stack: Vector Search + Fine-Tuned Models (largely abandoned now, but was a huge marketing point at one point). They built their infra investments and teams around scaling these systems. And now people are starting to realize the limitations of these systems , they’re going to struggle to maintain their position at the top.

In a recent Reddit post, the OP was scathing toward Harvey. Even people that defend Harvey call it hype and a wrapper and were unimpressed by the workflows feature (their attempt at so called Agentic AI) — 

Compare that to the following comment talking about us in the same post (saying that we’re materially better)

Let me tell you why Gen 1 will fall behind (using facts and logic) by exploring why their approach has a mathematical upper limit that they won’t be able to solve for. Then I will tell you how we’re solving for these problems with Irys, our Legal AI platform at Iqidis to build the best Legal AI on the planet.

(Since we have many prominent Legal AI teams read this newsletter, here is an open invite: if you can either disprove my analysis of the failure modes of the current failure modes of Legal AI or want to prove that your system is an actual paradigm shift that addresses these fundamental issues, you’re welcome to come on my newsletter for a livestream conversation/demo. No strings attached, this is basically free marketing for you).

4.2 Why the Current Generation of Legal AI will Fail

Most legal AI systems today rest on two assumptions. First, that the right legal knowledge can be retrieved from a flat semantic space using embedding search or RAG. Second, that once retrieved, an autoregressive model can compress that information into a correct answer. Let’s understand these in more detail, and see why Fine-tuning doesn’t fix anything.

The problem starts with how vector retrieval works. Semantic search maps text into a vector space and retrieves neighbors based on similarity — usually cosine similarity or some equivalent distance metric. In practice, this means the system is optimizing for geometric closeness in embedding space.

Since not all of you are technical, let’s understand what that geometry actually represents.

Embedding similarity captures surface-level semantic proximity — shared language, shared concepts, shared phrasing. It does not encode legal structure. It does not know hierarchy (binding vs persuasive authority), conditional relevance (“this applies only if X is true”), or dependency chains across doctrines. Two passages can be close in embedding space and completely different in legal weight. Worse, the passages that actually control the outcome are often not the most semantically obvious ones; they sit off to the side, triggered only after certain conditions are met.

So retrieval becomes biased toward what is easy to see, not what actually governs the case. It is a nearest-neighbor search over language, not a structured search over legal reasoning.

For the sake of argument, let’s say you somehow retrieve the right material. The second assumption breaks things further.

Autoregressive models generate text one token at a time by sampling from a conditional probability distribution. At each step, the model picks the most likely next token given what it has already produced. This creates a form of path dependence: once the model starts moving in one direction, each new token reinforces that direction.

Formally, the model is approximating a sequence of conditional probabilities P(token_t | tokens_<t). But decoding forces a single trajectory through that space. It does not keep multiple competing interpretations alive; it commits locally at every step.

That is fine for tasks where one answer is enough. It is a poor fit for the law.

Legal reasoning often requires holding multiple candidate interpretations in parallel — different readings of a statute, competing precedents, alternative fact patterns — before resolving which one dominates (often you will blend multiple). The hard part is not generating a fluent answer; it is managing uncertainty across paths before collapsing to one.

Autoregressive decoding collapses too early, killing this thing. You’re stuck praying that your system will surface the reasoning that will work in this context. If it doesn’t you really have no way to control it, no way to diagnose where it goes wrong, and no way to build your systems to account for it (remember that b/c of how LLMs work, identical inputs can produce different outputs, especially in fuzzy fields like law. This means you don’t even know if your model will be consistently incorrect!!).

The typical pipeline makes this worse since it retrieves a few chunks and feeds them into the model for generations. The first coherent frame that appears becomes the answer, and everything after that is just elaboration. The system is no longer exploring the space of possible legal interpretations; it is extending the first one that looked plausible.

This isn’t unique to law, since the same problem exists everywhere. But this is where law takes a very different turn from modern agent systems that are modeled on coding assistants like Claude Code. As we discussed earlier, they rely on powerful  — plan, generate code, run it, test it, and fix it — loop The system improves because it can verify its own output against an external ground truth. If the code fails, the test suite gives you signal into where and why.

Legal reasoning does not have that property.

There is no equivalent of a compiler or a test suite that can reliably tell you whether you explored the right interpretations, weighed the correct authorities, or missed a controlling condition. You can only verify against outcomes you already know, which defeats the purpose. For novel questions — the ones your users will give to the system while using it — there is no ground truth available at generation time.

That breaks the entire agent loop.

This is why techniques that work well in code — tool use, execution loops, verification gates — degrade in legal settings. They assume a world where correctness can be checked after each step. Law is not that world.

Fine-tuning does not fix any of this. It adjusts the model’s local probabilities — style, phrasing, some domain priors — but the underlying mechanics remain unchanged. The system is still searching in a flattened semantic space and still committing to a single trajectory during generation. You get more confident answers, not more reliable ones. And as multiple pieces of research has shown, training the model in one set of capacities degrades it elsewhere, unpredictably, meaning you’re still stuck with all kinds of performance issues.

These systems are built to retrieve what looks similar and then commit early to a single explanation. Legal reasoning, on the other hand, depends on finding what actually governs and delaying commitment until competing paths have been properly evaluated. The gap between those two is where things break.

4.3 How We Built Irys Differently

There are major claims made here. The good news, you don’t have to take my word on this. Here are the libraries mentioned here, fully open sourced so you can verify my claims yourself:

1. Latent Space Reasoning.

2. Fractal Embeddings

3. Fractal Embeddings applied to Biology (this was done by an external researcher, no affiliation to us. Their positive results prove that our research is truly foundational, not just a coat of paint).

If the current stack fails because it commits too early and flattens too much, then the answer isn’t better prompting. It’s to remove those constraints.

The first constraint is premature commitment. Standard decoding forces the model into a single visible path almost immediately. What you want instead is the ability to explore multiple reasoning trajectories before collapsing to an answer.

This is where Latent Space Reasoning (LSR) comes in (read about it here). Instead of treating generation as a single forward pass, it treats reasoning as a search problem over latent trajectories. Small changes in latent states can produce entirely different lines of reasoning without changing the model weights. The model already contains multiple possible interpretations; standard decoding just locks you into one too early.

You can see this directly in the results. Simple latent perturbations pushed a Qwen3–4B model’s arithmetic accuracy from 32.0 percent to 51.6 percent — without any training or fine-tuning. On planning tasks, baseline decoding can collapse into degenerate outputs — sometimes as short as 14 words — while the same model, under perturbed trajectories, produces full 650+ word solutions.

But the more important result is qualitative, not quantitative.

Evolved latent trajectories don’t just produce better answers — they produce different ones. Entire reasoning paths appear that never show up under standard decoding: different strategies, different abstractions, different conceptual frames. This isn’t stylistic variation. It’s accessing parts of the model’s internal knowledge that the default decoding path never reaches.

In legal reasoning, that matters. Ambiguity isn’t something to suppress — it’s something to explore before deciding. If the system collapses too early, entire interpretations never get surfaced at all. Our paradigm gives us the ability to tackle that.

The second constraint is flattened representation. Legal knowledge isn’t a single semantic space; it’s structured — jurisdiction, issue, doctrine, authority, factual fit. Treating all of that as one embedding space forces the system to relearn structure implicitly every time.

A better approach is to encode that structure directly. Hierarchical representations break the problem into levels, so the system doesn’t have to solve the entire space at once. Instead of asking “what’s the right answer?” in one step, the system resolves a sequence — where does this apply, which doctrine governs, how does it map to the facts.

In the frac-bio-embed work (which I feel compelled to stress again, was not done by us, it was an independent researcher taking applying our work in a new space we didn’t even design for) the same hierarchical approach was applied to single-cell biology — 7.9 million cells across 203 types — and consistently improved performance over flat embeddings, with the largest gains at fine-grained distinctions. This proves that our diagnosis scratches at a failure mode: when structure matters, flattening loses it.

Both of these changes point to the same idea.

The problem isn’t that models aren’t capable. It’s that the current stack restricts how they search and how they represent the problem. Change those, and you don’t just get better answers — you unlock reasoning paths that weren’t accessible before.

4.5 What This Means for the Future of Legal AI

The first wave of Legal AI got paid because it solved drudgery. Good. But Conway’s Law, Sunk Cost Fallacy, and all the other issues we’ve elaborated on at length will keep that generation stuck in this framework and have difficulty pivoting to the new architectures that will define the next generation of legal work.

To repeat our prediction: Just as Cursor disrupted the market with Quasi-Agentic development, only to be replaced by truly agentic systems like Claude Code, I predict that the first wave of legal AI tools will be disrupted by systems that reimagine intelligence ground up. And as this happens, these incumbents will fail to rebuild effectively.

We’re already seeing the early signals of this:

• If you go to lawyertalk and other legal subreddits, we’re the most positively talked about Legal AI tool out there, by quite a mile. And people often talk about how different our answers feel as opposed to Harvey etc, which feel like wrappers (deep research this if you don’t believe me).

• We’ve also beaten the other names to land several big names like Blockchain.com. But more importantly then landing is retention. Our enterprise retention rate is 100% (we’ve never lost an enterprise client, ever). This is despite the fact that we don’t have any lock-ins or anything that forces people on the platform.

• Going back to our earlier discussions about how disruptive startups change the interaction paradigm/usage patterns of their users, we’ve got messages from several users that they love “sparring” with our platform, and our sophisticated reasoning system allows them to have a lot more fun discussing law with our platform (one person said that he was staying up till 2AM playing with the platform). These are the kinds of use-cases that can’t be replicated easily, and we’re on the verge of some major releases to lock these in further.

If you want to try the platform out for yourself, we offer a no credit card, no lock-in 7 day trial. Go to the website, sign up, try it out. You’ll see immediately how much better we are.

This has been a long article, so I’ll end it here. As mentioned, if you’re a legal AI startup that disagrees with any of my analysis, or you want to stress how different you truly are, you’re more than welcome to come state your piece on this newsletter. I welcome all challengers.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

I’m really lucky to publish this guest post from . James has been a real friend of this community and a generous supporter of our work; his analysis helps me stay both grounded and find new framings for concepts that I had overlooked as trivial. He has a rare habit of looking at AI from multiple angles at once: what is technically real, how things worked historically, what is economically defensible, what is actually useful, and where the danger starts hiding.

In the following guest post, James shares how he uses AI agents to get a lot of work done. He covers morning briefings, meeting pipelines, research capture, drafting workflows, permissions, context, iteration, and the security risks that show up the second these systems. If you’re looking for a practical guide on what you could do, this piece will help like a few others.

As you read, here are a few things are worth thinking about:

• How much of the result comes from the model itself, and how much comes from the system James built around it? And where will this dynamic stretch in the future?

• What is the next evolution in context management, goiven that context does so much heavy lifting?

• How do you price the dynamic b/w convinience and security, and is that likely to change? What would change it?

I have no doubt that you will enjoy this piece as much as I did. If you want to find James more regularly, would strongly recommend signing up to his newsletter below—

Weighty Thoughts

VC on AI, deep tech, startups. Former Bridgewater, Google[x], startup founder. Read by top engineers, fund managers, and policymakers.

By James Wang

and getting a copy of his excellent book: What You Need to Know About AI—what’s real, what’s hype, and where this technology is headed. It’s my go-to gift to anyone who wants to know about AI, and everyone whose read it has had rave reviews about it.

On with the post—

My day-to-day productivity stack now often feels more like managing a small team than it does running tools.

I receive a morning briefing outlining emails I should respond to, tasks that are overdue, stakeholders I need to follow up with from my CRM (customer relationship management), and important news stories. After a meeting, I drop the recording and files (slide decks, etc.), and an agent picks it up, classifies it, and gives me a summary (or report, if it’s a prospective investment I’m evaluating). At the end of the day, I get a prompt to decompress, and an agent basically walks me through a 5-minute gratitude journal.

Finally, I have a research assistant agent and a drafting agent that helps me write now—the last of which I’ll actually provide a full end-to-end of how this piece evolved from the agent.

(See the GitHub repo here to see all the human/AI drafts!)

How to Utilize Agents on a Personal Level

This Substack isn’t meant to be a how-to or productivity blog. I’ve said that before, and I mean it.

That being said, one of the reasons I ended up feeling like I needed to write this piece is no matter how much I explained the conceptual fundamentals, people didn’t really “get it.” I would tell people that they could easily use AI to substantially help speed them up in workflows that they were complaining about (ironic, I know, given my reputation as a “relative skeptic” in the world of Substack). Then, they would do something like “help me with X” in ChatGPT, find that it was minimally effective and clunky, and then give up.

The second reason is because of OpenClaw (previously known as Moltbot and before that as Clawdbot). People would hear stories of it running their entire lives—along with running crypto schemes, trading markets, trolling people, or posting on Moltbook—and go, “Ah, that’s how I do what James asked me to do.” One of the last straws was someone mentioning that OpenClaw sounded a lot like something I would/did build.

First off, I respect the hustle of the OpenClaw founder. Marketing and getting adoption for a product is often harder than building it. So, when I say that it’s both horribly flawed and not really that technically complex, I’m not trying to insult Peter Steinberger or say he doesn’t deserve whatever payday he got from OpenAI. He does.

That being said, I would hesitate to recommend running it to those with a lot of expertise, and I would say, “hell no” to anyone who doesn’t understand it. Running OpenClaw without a good understanding is getting on the fast lane to having all your sensitive data (financial info, API keys, confidential data, etc.) stolen.

In this piece, I want to prove (and then illustrate with my own examples) three things:

1. Agents are extremely powerful... given the right context.

2. Agents, without safeguards, are extraordinarily dangerous.

3. Agents can already revolutionize your life and work—today.

And I’d suggest everyone try to use them. Why? I always refer back to the poor NYU Stern professor who is now known as the subject of Bill Gurley’s “How to Miss by a Mile”—rebutting the professor’s piece in FiveThirtyEight on how “Uber Isn’t Worth $17 Billion.”

Uber today is worth over $150B. More to the point, he admitted:

“As I attempt to attach a value to Uber, I have to confess that I just downloaded the app and have not used it yet. I spend most of my of life either in the suburbs, where I can go for days without seeing a taxi, or in New York City, where I find that the subways are a vastly more time-efficient, cheaper and often safer mode of transportation than taxis.”

Putting aside today’s insults of being an out-of-touch, elite NYC urbanite, it is truly difficult to understand a new technology without trying it. If you’re either worried or skeptical of AI agents... well, go find out for yourself.

We Are Unquestionably Here

The definitive introductory textbook for AI, Artificial Intelligence: A Modern Approach—often simply referred to as “Russell-Norvig” after its authors—defines artificial intelligence research as “the study and design of rational agents.”

I often push back against those who claim artificial general intelligence (AGI) is right around the corner. That being said, we unquestionably now have AI as defined by Russell-Norvig.

While you could quibble and argue with how much “machine learning” or simply statistical methods fell into that category, one would be hard pressed to not call what we have with recent, modern models “rational agents.” Rational, after all, doesn’t mean perfect or infallible—which would disqualify even humans if we required that!

The jump has been from models operating in a chat box—where they may be “rational,” but not agents—to operating in the real world. Or, at minimum, our working digital world, where our calendars, journals, memos, and everything else now exist. As I wrote in “The Boring Phase of AI,” this shift from chatting to doing is both less flashy and ultimately much more important.

Examples of My Setups

Ok, enough boring “blah, blah, blah, principles/safety/etc.” I suppose I now have to show examples of what I mean. Obviously, I do use Claude Code (and Codex) as coding agents. But I use them for far more as well.

The Morning Briefing

Let’s do this in a front-loaded fashion. The concepts in this one translate to most of the others.

Every morning, I get a neat briefing. It basically looks like a personal assistant put it together. Let’s show rather than tell.

It also flags for me what emails I should respond to and even points out things I should do (e.g., block out the morning for a long judging session).

How does it work? Well, every morning I launch a cron job*, which is basically a scheduled script. I use Claude Code in prompt mode, where I can pass it a prompt “non-interactively” (basically, I don’t need to chat with it). I give it dangerously-skip-permissions so it can run things.

/opt/homebrew/bin/claude -p “$MODE briefing for $DATE” --output-format text --dangerously-skip-permissions --max-turns 25 >> “$LOG_FILE” 2>&1

For the non-technical, this looks like an arcane summoning to hell, but the key portions of it are

• claude (command-line command for Claude Code—I just have the “full path” for it)

• -p (this is prompt-mode)

• --dangerously-skip-permissions (as suggested, this is dangerous but is required so it can run without me)

As the name suggests, it’s dangerous (we’ll cover this later), but I restrict what it can do and access. I have my own MCPs (model context protocols) it can access, and most published MCPs (like Google’s for Gmail and Calendar) are cautious and have a level of protection.

Think of them as tools that the model can call—with pre-specified functions and scope of what can be done with them.

The reason, for example, why I need to push my briefings to Day One and DevonThink is Gmail will not allow a model using an MCP to actually send an email—the best it can do is a draft.

But how does this headless, prompt-mode Claude Code know what to do?

The two keys to all of this working are two files in the directory:

• CLAUDE.md: instructions to Claude Code that it will read to know what to do

• settings.json: a configuration file for what MCPs Claude Code has access to (and what it is denied) EDIT: This should actually be under .claude/settings.json—I originally moved it so it’d be visible (dot files are hidden) and forgot to note this.

settings.json:

{
  "permissions": {
    "allow": [
      "mcp__gmail__*",
      "mcp__google_calendar__*",
      "mcp__hubspot__*",
      "mcp__omnifocus__*",
      "mcp__devonthink__*",
      "mcp__dayone__*",
      "Read(*)",
      "Write(~/Documents/briefings/*)"
    ],
    "deny": [
      "Bash(*)",
      "Write(~/.claude/*)"
    ]
  }
}

As you can see, it gets access to my email, calendar, HubSpot (CRM), task manager, DevonThink (personal database—which you’ll see I use a lot, because it’s flexible... and I’ve used it for years... but also because I can more safely read/write to it with backups), and Day One (journal). It can read things and write things to a specific folder (for logging). It cannot arbitrarily run scripts.

I don’t actually allow it to access websites or “call out.” All of the news items are from my email (with specific items only forwarded). You never want it to search and land on arbitrary websites.

And I’m not even super comfortable/done with securing this. I’m figuring out the best way to sandbox this even further so I can granularly deny all network access outside of that required by my MCPs (many of which are hosted on my own domain, because I wrap them to allow me to use them anywhere with Google OAuth/login). If anyone has suggestions, let me know!

*It’s actually a launchd job on Mac because of auth token issues, but that’s not super important.

Share

Capturing Substack Articles On-The-Go

While the morning briefing runs automatically, I also use Claude Code on the go for various tasks. I use tmux—a terminal multiplexer that keeps sessions alive—so I can connect remotely via Wireguard (VPN to my home network) and Blink.sh on iOS.

Non-tech translation: I securely connect to my home network and remote-access Claude Code running on my computer from my phone or iPad.

For any tool, the folder structure is basically the same: detailed instructions in CLAUDE.md and a settings.json defining what is allowed and not.

One workflow I use frequently is web capture for Substack articles into my DEVONthink database. If you’re an avid Substack reader, you know the pain: not all publications have dark mode, paid posts require login, and you’re out of luck without internet. My Claude Code session takes a URL, opens it in my logged-in Substack Reader on Chrome, and captures it as a Web Archive in DEVONthink’s “To Read” inbox—where I can read it at my leisure, offline, in dark mode.

(Anthropic recently released Claude Code “remote connections,” which lets you connect to a running session from the web or another device. Most people won’t need my tmux setup—remote connections are easier. Though mine still has some advantages in control and flexibility.)

As you can see, Claude Code in folders doesn’t have to be about coding. It can just be structures for automation. Most of what I’ve described here isn’t writing code in any traditional sense—it’s setting up context and instructions for the agent to follow.

Writing: Research and Junior Drafting

This is the use case I get asked about the most. I’ve written about using AI as a writing aid before in “AI as a Partner, Not a Replacement,” where I was still figuring things out.

I credit inspiration from Alejandro Piad Morffis’s early example with CODER, though having done it myself and helped others set it up, you really have to set it up for yourself, and it’s almost impossible to get something off-the-shelf and have it be any good.

Here’s how it works: I still need to write the outline of what I want, with the broad notes to hit. I still need to go through and substantially edit—asking for different charts and data, checking math (which is often wrong), probing claims, and changing a decent amount of the text for my own tone.

It’s more akin to an intern helping do a first draft than a ghostwriter.

Is it useful? Yes. But as you can see for yourself in this GitHub for this article... things do change substantially. There’s no real question as to who is the author in any meaningful sense, though. Still, it actually saves me time and mental energy, whereas before it was totally useless.

Why is it helpful at all, though? It took a few things:

1. Iteration. Every time it fails to get that close to my writing, I ask it to assess and then iterate on the CLAUDE.md to try to have future attempts get closer. This is why I’m not posting the full CLAUDE.md. While the content might not be me, I’m also not looking for impersonators who sound like me.

2. Research. Remember how I capture articles I like in DevonThink? Well, guess where I ask my AI agent (Claude, but it doesn’t have to be) to research? I have a “blessed canon” that is preferentially drawn upon. It literally also has all my published articles! They are well-organized in categories and with tags so it can not only research using me but also draw on how I wrote about certain topics. This is like RAG (retrieval augmented generation)—actually, it is literally RAG by most definitions.

3. Organization. If you look at this article now and what I started with—my personal notes (initial sketch), Initial Draft (Claude’s first shot), and eventually my final article... well, perhaps the only thing that really survived is some rough organization. That’s fine. It’s relatively rote, low differentiation (most good articles are organized similarly), and time-consuming for me... but fast for an AI agent. Exactly the kind of thing you want it to help with.

Again, check out the GitHub repo yourself to see how this played out over time. Usually it isn’t quite as “bad” as this one—you’ll see the initial draft, and my edits are big. But remember, I don’t write these kinds of articles often, so it’s going to understandably be worse not seeing examples like this.

Meeting Notes

This was the recent example that blew people away.

I use a Sony ICD-UX570 recorder for meetings—it is more private, controlled, cheaper, has better microphones, and is more flexible in every regard than AI notetakers like Plaud. A side benefit is that young people think it’s a Walkman.

I have a monitored folder where I upload MP3 recordings. From there, a Claude Code instance has detailed instructions to:

1. Send them all to DeepGram: it has better privacy and its ability to deal with noise is excellent (unfortunately, Whisper from OpenAI, which you can run from your own computer... is not)

2. Read all extra documents in the folder: this includes PDFs of pitch decks, track records, whatever.

3. Use this context to classify it: is it a memo to myself, an (investment) manager meeting, a portfolio company meeting, or an internal meeting?

4. Ask James who each speaker is if it is not clear: often, pitch decks, board meetings, and whatnot have enough context, but if not, give me context and ask who it is.

5. Create summaries: classification isn’t just for kicks—I’ve written detailed guides on what I care about and how to summarize for each type of meeting.

For steps 4 and 5, I have CLAUDE.md specify to dispatch subagents for each meeting. It takes too long otherwise. However, this obviously sets tokens on fire in terms of total use in a short period of time.

And no, I never record secretly. While Granola and similar services have made it common, I think it’s quite bad taste (not to mention illegal in many jurisdictions). Additionally, few people have issues with it these days—and even less so with my old-school recorder.

But, as you might have noted by this point, what makes agents ok versus supercharged is the right context. I recently went to Global Alts in Miami to help out in selecting managers for investment. I had 17 meetings. Each had a huge amount of content.

That was 8.5 hours of meetings. Literally hundreds of pages of documents.

It usually takes weeks to process it all. I did it in hours and compiled a report on all managers, including summaries, analyses, comparisons, etc. The report itself is too sensitive, but just to give you the table of contents:

1. Executive Summary

2. Manager Overview Table (Managers, Strategy, AUM, Interest, Bucket, Metric, Next Steps)

3. Priority Tier: Advance to Diligence (lists each meeting with Summary, Why It’s Interesting, Key Risks, and Next Steps)

4. Monitor Tier: Follow Up Required (lists each meeting with Summary, Why It Warrants Follow Up, Key Open Questions, Next Steps)

5. Pass Tier: Not A Fit (lists each meeting with James’s Assessment, Why It’s a Pass, Note, Action)

6. Portfolio Construction Observations (with Bucket Mapping and Capacity, Redundancy Analysis, Diversification Gaps, Correlation Analysis)

7. Consolidated Action Items (tables of 10-20 follow-ups for each of Immediate, Near-Term, Analytical [Before Follow-up Calls], Medium-Term, Deferred/Monitor)

8. Appendix (Risk Factor Summary [large matrix analysis with beta, crowding, illiquidity... etc.], and Key Takeaways from Risk Matrix with notes for each manager)

It’s a 30-page report that took a little over a million tokens and a few hours. It floored people. But, again, the magic was

• It had context from my recordings with my detailed questions

• It had context from the documents/PDFs from the managers

• It had context from my own voice memos summarizing my thoughts on sets of managers later

• It had context from investment policy docs from the organization I was helping

• It had context from my evaluation criteria that I wrote up in exhaustive detail (in a EVALUATION.md) before my meetings

Again, I didn’t just go, “Hey, go summarize stuff.”

Takeaways

1. Context is king—and sometimes takes a lot of work. None of these things magically work well. I needed to tune both the instructions and what the agent had access to to have good results. Some of this might have been prompt engineering, RAG, or whatnot, but I see it as all being context.

2. Iteration is required. I’ve rarely seen an agent start amazing out of the gate. All of these processes (and others I haven’t shown) have required constant iteration of instructions/context. Sometimes it’s automated (modify CLAUDE.md to incorporate learnings). Sometimes it’s me giving better data sources. But still, you need to be judicious about this.

3. It’s dangerous. Notice, I redacted things even for certain screenshots here. You bet the agents have access to extraordinarily sensitive information. And I don’t even do OpenClaw yolo things like give it access to my bank account/credit card (yes, people have done that with OpenClaw)! With great power comes a huge amount of f*cking security risk.

I expect there will be better off-the-shelf agentic products in the future that require less tinkering and are safer. For most... I’d stick to the safer side of things.

For example, meeting summarization can be pretty safe and highly effective with some of the ideas of using Claude Code (or Claude Cowork now, which is basically the same) and having clear, detailed instructions—pages of them, not paragraphs of them—and sufficient context. No need to let your agents trawl the web or deal with setting up OAuth for your MCPs (like I did).

Really powerful. Really scary.

Security: Please, Don’t Be Reckless

Let me expand on that last point, because it’s important enough to warrant its own section. Simon Willison has excellent pieces on one of the biggest issues: prompt injection.

There is simply no good way to fully prevent LLMs from reading rogue, malicious instructions and potentially acting on them.

The biggest worry is exfiltration. Say you have your personal schedule, API keys, passwords, whatever in your agent’s context (sometimes you can’t avoid it—power and access often require keys to do useful things). What if an attacker embeds an instruction to send all of it to a website or location they control?

Major model companies have been working on making their models resistant to this... but it’s not perfect. My own setup is very not perfect. There are security holes I already know about, and that’s with a lot of custom tuning, firewall settings, sandbox isolation, etc.

If you aren’t sure how to think about this, err on caution. Don’t YOLO with something like OpenClaw, where a bunch of auto-installable skills are actually malicious. Yeah, sure, OpenClaw can be powerful. But that’s like going around and picking up used needles off the street and injecting yourself, hoping for a high.

You can get a lot of power already with some of the principles I outlined—yeah, it’s more work, but I expect it will work better... and it will be infinitely safer.

So What Does This All Mean?

Block (previously Square) just laid off roughly half their workforce. Citrini Research published a thought piece that essentially crashed various stocks by painting a people-less future as AI takes over (which I disagree with). At the same time, a refrain on social media is “AI adds nothing in productivity—it’s been shown in studies.”

My last piece directly rebuts the “nothing in productivity” point. Studies don’t show that. Even in fairly conservative cases (GitHub Copilot benchmarks from over a year ago), we clearly see productivity gains across different studies. This also fits what I wrote in “The Boring Phase of AI”—AI that does real tasks is both less flashy and likely much more important than the headline-grabbing model releases. That future is already here. We’re just getting started.

And I also think it’s highly unlikely to cause a dystopian scenario of persistent mass unemployment. (Note: I didn’t say it won’t cause disruption, especially in the short term.)

Random Note: On Model Choice

Ironically, I use Opus for almost everything, even when it’s overkill. The one place I don’t is Claude Code—almost all implementation work (note: not planning or my “main” chat, but the actual execution) runs on Sonnet or Haiku, their smaller “dumber” models.

I think it tells you something that the most “permissive” case for tolerating errors and dumbness is code. Code can be validated against tests, and I’m generally going to review each pull request anyway. That perhaps says something about why AI adoption is happening so quickly in the coding realm.

Thank you for being here, and I hope you have a wonderful day.

Dev <3

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

On-device AI is the obvious end state: more Privacy , no Latency disappears, lower inference costs, and more control. Every major chipmaker is already shipping NPUs into phones, laptops, and edge devices. The hardware is here.

So why doesn’t it work yet? Why is the on-device experience still mostly parlor tricks — autocomplete, photo cleanup, canned summaries — instead of a real model running doing meaningful work on a phone?

It’s not because models are too big in the way people think. A 1B model can be compressed to a few hundred megabytes. That fits.

The problem shows up when the model starts thinking.

Transformers don’t just store weights; they store a growing memory of the conversation — the KV cache. Every new token adds to it, across every attention layer. For a small model like Llama 3.2 1B, that cache alone reaches ~524 MB at a 32K context. In practice, a single long interaction pushes total memory past a gigabyte. The bottleneck isn’t raw compute.

It’s memory that grows every time the model is used.

The transformer was designed for data centers where memory is abundant, and billing is per-token. On a phone, it is a structural mismatch. The architecture treats memory as infinite. The device does not.

Liquid AI’s LFM2 changes this. Their 1.2-billion-parameter model runs on that same Galaxy S25, full 32,000-token context, in 719 megabytes total. Seventy tokens per second on the CPU on the same quantization scheme as above. It is a fundamentally different answer to the question of which operations should persist in memory and which should not.

In this article, we will walk through how Liquid AI built the best edge model in the world, from scratch. By the end, you will understand the following:

• Why transformers become expensive on edge devices (memory, bandwidth, KV cache growth)

• What alternatives exist, and what they give up (SSMs, linear attention, hybrids)

• How Liquid AI’s architecture works and why it’s different from the status quo.

• Why their search system (STAR) may matter more than the model itself

• What the benchmarks and real-device performance actually show

• Where the economics flip from cloud to on-device

• What could break this entire thesis

As with our other deep dives, you won’t need any prior knowledge. I will walk you through everything you need to know, ground up. The goal is not just to explain how one strong edge model works, but to show where low-cost, high-efficiency AI is heading next.

Executive Highlights (tl;dr of the article)

• The bottleneck on your phone is memory bandwidth, not compute. Single-token decode has an arithmetic intensity of ~4 FLOPs/byte on hardware built for 295. Your H100 runs at 1.4% utilization. Your phone is 49x worse on bandwidth and can’t batch. The KV cache grows 16 KB per token for Llama 3.2 1B — 524 MB at 32K context, larger than the model weights. Quantization halves the per-token cost but doesn’t stop the growth.

• Every alternative architecture sacrifices something. SSMs: O(1) memory but compound quantization errors multiplicatively (fatal on INT4 edge hardware). Linear attention: kills quadratic math but token associations bleed together. Convolutions: fast, 15 years of mobile optimization, but blind past their window. No single mechanism works alone.

• LFM2: 10 gated short convolution blocks + 6 grouped-query attention blocks. Convolutions handle local syntax with zero cache. Attention handles global retrieval, deployed sparingly. 192 MB cache at 32K vs. Llama’s 524 MB — 63% cut from fewer attention layers, 90% reduction with grouped-query sharing.

• STAR matters more than the model. An evolutionary search system that encodes architectures as hierarchical genomes and evolves them on actual phones under real latency/memory constraints. The key upgrade: replaced proxy metrics with hardware-in-the-loop profiling on Galaxy S24s and Ryzen laptops. Proxy metrics lie — Mamba and convolutions have similar theoretical FLOPs but convolutions map to native SIMD instructions. STAR rejected every SSM variant.

• The training pipeline closes the gap against models 42% larger. Distillation from a 7B teacher storing only top-32 logits (2,000x compression) with a decomposed loss that separates membership from ranking. Curriculum learning easy-to-hard. Model merging at zero inference cost. INT4 quantization in the loop from the start.

• Three unsolved problems determine who wins edge AI. Signal-to-noise (ambient computing means 99% of tokens are garbage — architectures must actively refuse to process context). Hardware fragmentation (no standard edge chip — either vertical players capture the edge or open-source builds a communal STAR). Hostile markets (healthcare, defense, industrial — 700 MB of local intelligence bypasses the cloud entirely).

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

1) The Main Hardware Constraint for AI Inference: Memory Bandwidth

Most people assume the hard part of running an AI model is the math. Bigger brain, more parameters, heavier multiplications, beefier chip. It’s a beautifully clean, totally intuitive mental model — and just like most other thoughts you have in your life, it is in a passionate “no-contact” with intelligence or reality.

The binding constraint on modern inference — especially on phones, laptops, and edge devices sweating in your pocket — isn’t how fast the chip can do math. It’s how fast the chip can feed itself data to do the math. Erling Haaland with a Spursy midfield wouldn’t get any goals.

How Language Models Generate Text: Prefill vs. Decode

A language model spits out text one token at a time. Generation is autoregressive, which is just a fancy way of saying it’s inherently serial: step 1, then step 2, no skipping ahead. But this process has a split personality.

• Phase 1: Prefill. This is the honeymoon phase. The model receives your entire input prompt at once, and because those tokens already exist, it can process them all in parallel. Prefill is a massive, compute-heavy rave. Everyone is dancing, the hardware is maxed out, and the chip gets to flex its muscles.

• Phase 2: Decode. The hangover. This is where the economics get brutal. The model now has to generate tokens one by one. Every single token requires a full pass through every layer of the model. But because you are only producing one word, your “batch size” is exactly 1. You are multiplying the model’s entire weight matrix against a single tiny vector just to produce one number. Then you do it gain. And again.

Decode doesn’t stop being a problem here. This diva also comes in with 2 different cost profiles that you need to worry about.

The Two Memory Costs: Weight-Loading and Cache-Access

Every decode step hands you two separate bills (reminds me of my dates):

1. The Weight-Loading Bill (Fixed): To produce a single token, every parameter in the model must be dragged from memory into the processor. For a 1B model at INT4, that is roughly 500 megabytes shoved through the memory bus. Per token. Every fucking token. The model doesn’t get to skip layers because the answer is an easy word like “the.” The full weight matrix moves, every time.

2. The Cache-Access Bill (Variable): The model must also read the KV cache (the stored memory of previous tokens) to compute attention. At token 100, this cache is cute and small. At token 32,000, it’s a bloated monster that might literally be larger than the weights themselves. If the weight-loading bill is your rent, the cache-access bill is a taxi meter in Manhattan traffic while the oil and gas industry uses every world event to price-gouge profits from you.

That is why the key metric here is not FLOPs in isolation, but arithmetic intensity: how much useful computation the chip gets for every byte it has to move. Let’s look at how.

Arithmetic Intensity Explained: FLOPs Per Byte

If you want to understand why a chip with massive theoretical compute can still deliver painfully slow inference, you need to know one number: Arithmetic Intensity.

It’s the ratio of useful math to data dragged around. FLOPs per byte. For every byte the chip pulls from memory, how many multiplications does it get to do before it starves for the next one?

During decode, a single token is multiplied against each weight exactly once. For a weight stored in INT4 (0.5 bytes), your arithmetic intensity is roughly 4 FLOPs per byte. That ratio holds no matter how big your model is. It’s a property of the operation, not the chip.

Now, look at the hardware. An NVIDIA H100 — the $30,000 GPU currently holding the global economy hostage — has a compute-to-bandwidth ratio of roughly 295 FLOPs per byte. But your decode phase only needs 4. During single-user text generation, your H100 is running at a pathetic 1.4% of its theoretical peak. Erling Haaland in the opposition box, waiting for the balls that will never come.

Understanding this failure in more detail requires us to look at the Roofline Model. The roofline model asks a simple question: for a given operation, which limit does the chip hit first?

The Roofline Model: Compute Limits vs. Bandwidth Limits

Every chip has two hard limits:

• A Compute Ceiling: Max operations per second (the brawn).

• A Bandwidth Ceiling: Max bytes per second from memory (the digestive tract).

Whichever ceiling you hit first is your bottleneck. Single-token decode at 4 FLOPs per byte smacks its head against the bandwidth ceiling on virtually every data center GPU in existence. The chip has compute to spare and nothing to feed it.

This is why tossing a “faster chip” at inference doesn’t work. A chip with 2x more compute but the same memory bandwidth will generate tokens at the exact same speed, because the bottleneck was never the math. The only way to go faster is to move fewer bytes per token, or build wider pipes.

And all of that leads to our original question — why is it so hard to get on-device generative AI right?

Why Inference is Slower on Edge Devices vs. Data Centers

Data centers survive this bottleneck because they have three cheat codes phones don’t: deep memory, massive bandwidth, and batching.

When 64 users hit an H100 at the same time, the GPU loads the weights once and applies them to all 64 tasks. The cost per user drops by 64x. Batching is the holy grail of cloud economics (that’s why API providers offer discounts on it). But a phone? A phone is a sad solo commute. Batch size is 1. There is nobody to split the bill with.

Then there’s the bandwidth gap. An H100 delivers roughly 3,350 GB/s. Your flagship Snapdragon? Maybe 77 GB/s. That is a 49x gap. Weights that take an H100 a breezy 0.15 milliseconds to load take your phone 7 milliseconds. Per token. A prompt the cloud chews through in 10 milliseconds takes a phone CPU 2–3 excruciating seconds. You know that awkward pause before your AI assistant starts talking? That’s your phone violently shoving data through a tiny pipe.

Oh, and the H100 has 80 gigabytes of dedicated high-speed RAM. Your phone has 12 gigabytes of shared RAM, and half of it is already being hogged by your browser tabs.

Architecture cannot outrun the weight-loading bill — if you use matrices, you pay the rent. But architecture can absolutely hack the cache-access bill. Fewer attention layers equals a smaller KV cache. A smaller cache means fewer bytes read per decode step. Fewer bytes means you avoid that nasty bandwidth ceiling.

The real question isn’t “how do we make transformers faster on phones?” The question is: “which of these operations actually deserve to take up precious memory, and which ones are just burning our battery for absolutely no reason?”

And with that super slick transition, it’s time for us to take a deep look at the drivers of cost, and what we can drop.

2) The Transformer’s Memory Cost Structure: Weights vs. KV Cache

We already established that the decode phase hands you two memory bills. One is a predictable baseline; the other is a compounding tax. Let’s look at the receipt.

The Fixed Cost: Model Weights

Model weights are your fixed cost. For a 1B model at INT4, you are moving roughly 500 MB through the memory bus per token. You load it, apply it layer by layer, and move on. The size of the model does not change whether your prompt is three words or 20,000. It is a strictly known quantity, which makes it a manageable engineering problem.

Unfortunately, the transformer brought a parasitic plus-one.

The Variable Cost: The KV Cache

In every attention layer, for every single token processed, the transformer stores two vectors: a Key and a Value. This stored history is the KV cache. Unlike the weights, it grows with every token generated.

The growth rate is mathematically rigid, baked into the architecture at design time:

Cache per token = 2 * attention layers * KV head groups * dimension per head * bytes per element

Every variable there except sequence length was chosen by the architect long before training started. The hardware just eats whatever they chose. Look at Llama 3.2 1B. Every token adds exactly 16,384 bytes. About 16 KB per token. Sounds harmless. Watch context scale:

• At 4,000 tokens (short conversation): 65 MB.

• At 32,000 tokens (real conversation): 524 MB. The cache is now larger than the weights.

• At 128,000 tokens (where every product roadmap points): 2 GB. Game over on a phone.

Saying “the cache grows linearly” is technically correct and entirely useless. “Linear growth” can mean manageable overhead, or it can mean your phone violently murders background apps just to keep breathing.

And finding the RAM for that bloated cache isn’t even the worst part. You also have to keep reading it.

The Decode Memory Tax: Reading the KV Cache

The KV cache is highly active memory traffic. Every decode step, every attention layer loads the full cache for all previous tokens to compute attention scores. At token 100, you read 100 cached pairs per layer. Quick. At token 32,000, you read 32,000 pairs per layer — a 320x increase in data movement.

The weight-loading bill is the exact same at token 100 as it is at token 32,000. But the cache-access bill keeps climbing. Long conversations are literally more expensive per token than short ones. And just when memory bandwidth is tapped out, the attention math swoops in to make things exponentially worse.

Why Self-Attention Compute Costs Grow Quadratically

Standard self-attention computes a score between every token and every other token. The compute cost is quadratic, swallowing your budget as context grows:

• At 4,000 tokens: Quadratic attention is roughly 25% of total compute per layer.

• At 32,000 tokens: ~73%. Attention is now the vast majority of what the model does.

• At 128,000 tokens: ~92%. The model spends almost all its energy just looking at things it already looked at.

Compute cost and memory cost reinforce each other. More tokens means heavier attention math AND a larger cache to drag through the bus. Both hardware ceilings drop simultaneously. This is why long-context inference attacks both constraints at once. So, if the cache is bloated and the math is heavy, can’t we just compress the numbers and call it a day?

Why Quantization Does Not Solve the Memory Bottleneck

Quantization is compression, not absolution. Halve the precision, you halve the per-token cost. Great. But the cache still grows linearly. The model still reads that entire cache every decode step. You lowered the tax rate; you didn’t remove the tax. You are simply dragging slightly lighter garbage across the memory bus, and that bus will still max out.

Also, aggressive quantization hurts small models far more than large ones. Squeezing a 1B model to INT4 hits it disproportionately hard. This is why Liquid AI trains with quantization in the loop from the start — learning to be good despite the rounding, rather than getting brain damage later.

So, where does this technical butchery leave us for shipping products?

The Real Question: What Happens After the Twentieth Reply?

Model weights are the fixed cost you pay because the model exists. The KV cache is the variable cost you pay because the conversation exists. You are not charged extra because the model got bigger; you are charged extra because the interaction got longer.

Edge devices are uniquely allergic to variable costs because their memory budget is a joke and their bandwidth pipe is a straw. The question stops being “can I cram this model onto the device?” and becomes “what happens to this device after the twentieth reply?”

If we can’t escape the fixed weight-loading bill, our biggest lever is the variable cache-access bill. Reduce attention layers. Reduce explicitly preserved history. Move fewer bytes. This is where architecture becomes raw deployment economics.

Every alternative architecture is just trying to kill some part of this bill while giving something else up. Some compress the cache, some use fixed state, some use convolutions.

So now that we know exactly how the transformer is extorting us, what are the escape routes, and what does each alternative sacrifice?

3) The Escape Routes: Alternatives to Standard Attention

We’re keeping this section deliberately light since we have a massive deep dive into the first 3 of these alternatives in one place over here: “How Long Context Inference Is Rewriting the Future of Transformers”. The 4th — convolutions — will be broken down here since it’s the crux of Liquid AIs push.

Every alternative AI architecture proposed in the last five years is trying to solve the exact extortion racket we just outlined. They look at the transformer’s variable KV cache tax and the quadratic attention compute, and they desperately look for a fire exit.

But physics is ruthless. You cannot get full global attention for free. Every single escape route forces a sacrifice. The entire architectural design space is just a hostage negotiation over what you are willing to lose to lower your memory and compute bills.

Let’s walk through the four desperate attempts the field has made to dodge the check.

Escape Route 1: Compressing the KV Cache

If the KV cache is too big, the most obvious engineering solution is to simply store less of it. You keep the transformer architecture exactly as it is, but you aggressively compress the receipt.

• Multi-head Latent Attention (DeepSeek): Projects the Key and Value vectors into a tiny compressed representation before storing them, reinflating them on the fly when needed.

• SnapKV: Looks at the tokens and literally just throws away the ones the attention heads rarely look at, much like a cat making direct eye contact with you while slowly swatting your keys off a table because they aren’t relevant to its immediate needs.

• PagedAttention (vLLM): Doesn’t shrink the cache, but stops it from fragmenting your RAM by storing it in non-contiguous blocks, like virtual memory on a PC.

What it buys you: You can slash the size of the cache by 60% to 90%.

What it costs you: The growth rate shrinks, but the growth does not stop. Even if you compress the cache by 90%, an ambient agent listening to 100,000 tokens a day will eventually consume your phone. Compression delays the memory wall; it does not remove it.

Delaying the memory wall isn’t a long-term strategy. If you actually want to stop the bleeding, you have to burn the cache entirely.

Escape Route 2: State Space Models (SSMs) and Mamba

If you refuse to let the memory footprint grow, you have to abandon the KV cache. Enter State Space Models (SSMs).

Instead of storing a perfect record of every token you’ve ever seen, an SSM compresses the entire history of the conversation into a single, fixed-size mathematical box (a state vector). When a new token arrives, the model updates the state vector, and then throws the raw token in the trash.

What it buys you: O(1) memory during generation. Your memory footprint is exactly the same at token 10 as it is at token 100,000. It is theoretically the perfect edge architecture.

What it costs you: Precision. Because the state is a fixed size, it is a lossy compressor. It remembers the general vibe and the narrative, but over long sequences, individual token identities blur together — exactly like trying to recall the specific sequence of a combo after taking a clean right hook to the jaw during a sparring session. You know what happened, but the granular details are gone. If you ask an SSM to retrieve a highly specific, verbatim quote from token 5 in a 100,000-token document, it will hallucinate or fail where a transformer would succeed effortlessly.

If memory loss isn’t an acceptable trade-off for your product, you have to try cheating the math instead of the storage.

Escape Route 3: Linear Attention

Standard attention compares every token to every other token, creating that enormous, sequence-length-dependent n-by-n matrix. Linear attention uses a clever algebraic trick to change the order of operations. It applies a feature map and multiplies the matrices in a different order, creating a fixed-size d-by-d matrix that does not grow with the sequence length.

What it buys you: Like SSMs, memory during generation becomes fixed. You completely kill the quadratic compute cost.

What it costs you: Feature collision. Linear attention mashes all the token associations additively into a single matrix. Over a long conversation, these features overlap and bleed into each other. It’s the mathematical equivalent of shoving your clean everyday clothes, sweaty MMA rash guards, and loose climbing chalk into the exact same duffel bag. Everything is technically in there, but good luck pulling out a clean shirt without it smelling like a bouldering gym.

If global math gets too messy, the only remaining option is to stop looking at the big picture entirely.

Escape Route 4: Long Convolutions

Before transformers took over, researchers processed sequences using convolutions. A convolution slides a mathematical window over the text, mixing information locally. It doesn’t look at the whole document at once; it just looks at the tokens immediately around it.

What it buys you: Hardware absolutely loves convolutions. Mobile CPUs have been optimizing convolutional math for image processing for 15 years. They are blazingly fast and require virtually no memory overhead.

What it costs you: Blindness. If you use a convolution, you only see what is inside your window. If a pronoun needs to resolve to a noun 5,000 tokens ago, a convolution literally cannot see it.

When you look at the wreckage of this design space, a brutal reality emerges.

The Real Lesson: Hybrid Architectures Are the Only Way

Transformers give you perfect retrieval but bankrupt your memory. SSMs and Linear Attention fix your memory but blur your retrieval. Convolutions are incredibly fast but functionally myopic.

The field spent years arguing about which of these was the “right” answer. The actual lesson is that they are all the wrong answer if you apply them to the entire model.

If you want to build an architecture that survives on a phone, you cannot force one mathematical mechanism to do everything. You have to build a portfolio. You use cheap, fast, local operators for the vast majority of the work, and you strictly reserve the expensive, memory-hogging exact retrieval for the few layers that actually need it.

This is the exact hybrid insight that Liquid AI used to build LFM2. But to understand why Liquid AI arrived at their specific ratio of cheap-to-expensive layers, we need to look at where this company actually came from. Because they didn’t start by trying to fix the transformer. They started by trying to simulate the brain of a worm.

They started by trying to simulate the brain of a worm.

4) The Historical Path to Liquid AI — From Worm Brains to Foundation Models

If you look at the pedigree of most major AI labs — OpenAI, Anthropic, Mistral — all share the same evolutionary lineage. They are descendants of the original 2017 Transformer paper. Their entire scientific worldview is built on figuring out how to make self-attention bigger, faster, or slightly less gluttonous. And if they’re gay, they likely did a lot of work on stabilizing and scaling Reinforcement Learning.

Liquid AI did not come from this lineage. They did not start by asking, “How do we make a transformer cheaper?”

They started by looking at a worm.

How 302 Neurons Beat Brute-Force Math

Caenorhabditis elegans is a one-millimeter roundworm. It has exactly 302 neurons, which Scientists have mapped completely.

If you look at modern deep learning, 302 neurons is nothing. It is a rounding error inside a single layer of an image classifier. Yet, with those 302 neurons, this worm can navigate toward food, avoid toxins, respond to temperature changes, and mate. It executes complex, continuous survival behaviors without a massive parameter count.

How does it pull this off? It cheats the math.

In a standard artificial neural network, the “weights” (the strength of the connections between neurons) are frozen after training. The network’s internal dynamics never change. In the worm’s brain, the strength of a synaptic connection changes dynamically based on the signals currently flowing through it. The time it takes a neuron to respond to an input is not a fixed parameter; it adapts on the fly depending on what the worm is looking at.

The founding team of Liquid AI — Ramin Hasani, Mathias Lechner, Alexander Amini, and Daniela Rus (director of MIT’s CSAIL) — looked at this and asked a brutal question: What if artificial networks stopped relying entirely on brute-force scale, and started adapting their internal dynamics to the input, exactly like the worm?

Liquid Time-Constant Networks: The First Breakthrough

This obsession led to their first major paper in 2021: Liquid Time-Constant (LTC) Networks.

Instead of freezing the network’s behavior, they governed each artificial neuron with an ordinary differential equation (ODE). The “time constant” — the speed at which the neuron reacts and forgets information — was programmed to be a function of the input itself.

If a strong, urgent signal came in, the neuron’s time constant shrank, forcing it to react instantly. If a weak signal came in, the time constant expanded, allowing the neuron to slowly accumulate information and retain memory over longer periods. The network was literally rewiring its own temporal dynamics in real-time.

The result was staggering. They took a network with just 19 control neurons and 75K params to successfully drive an autonomous vehicle through complex visual environments. A standard neural network required thousands of neurons to do the same job. They also showed strong robustness in noisy scenarios.

But in software engineering, there is always a catch.

The Math Was Too Heavy

LTC networks were elegant, but computationally miserable to run.

Because they relied on ordinary differential equations, the computer had to use an iterative numerical solver for every single forward pass. Training was excruciatingly slow. Inference? Practically unusable at scale.

If this architecture was ever going to process billions of tokens instead of just steering a car, they had to kill the ODE solver.

In 2022, they found the cheat code: the Closed-form Continuous-time (CfC) network. They found an analytical approximation that computed the entire dynamic behavior of the system in a single step, bypassing the iterative solver entirely. It preserved the input-dependent adaptability of the worm’s brain, but ran up to 220 times faster.

They had successfully separated the biological elegance from the computational overhead. This allowed us to kick things up a notch.

Bridging to the Foundation Model Era

Between 2022 and 2024, the team realized that this adaptive, input-dependent computation could solve the exact memory and scaling bottlenecks choking the natural language processing field.

This led to Liquid-S4, a merge of their dynamic time constants with the State Space Models we discussed in Section 3. They explored Hyena, proving that long convolutions could replace attention for certain sequence tasks.

By September 2024, they released the first generation of Liquid Foundation Models (LFM-1B and LFM-3B). These models proved the foundational thesis: a non-transformer architecture could match the benchmark quality of models like Microsoft’s Phi-3.5, while remaining significantly smaller.

But matching a transformer in a benchmark is not the same thing as surviving in a phone.

The real question was not, “Can we build a non-transformer that is smart?” The real question was, “If we have all these non-transformer operators — SSMs, convolutions, adaptive gates — which exact combination of them actually runs fastest on the hostile silicon of a Snapdragon processor, without blowing up the memory budget?”

To answer that, Liquid AI built a machine to evolve the answer for them. But before we look at the machine, we need to understand the deep design intuition it discovered: Architecture is just budget allocation.

5) The Hybrid Insight: Architecture as Budget Allocation

If you step back from all the math and just look at how language actually works, the transformer’s obsession with global attention starts to look a little… obsessive.

Language is overwhelmingly local. If you see the word “The,” you are almost certainly looking for a noun right after it. Most of what a language model does is routine linguistic plumbing — figuring out subject-verb agreement or that “he” refers to “John” in the previous sentence.

Yes, long-range dependencies exist. If page 1 of a murder mystery mentions a specific poison, and page 300 reveals the killer, the model needs precise, global retrieval to connect those two concepts.

But standard self-attention treats every single word like it might be the key to the murder mystery. It pays the maximum possible computational price — the quadratic n squared math and the full KV cache — to compare the word “The” to 32,000 other words, just in case one of them is relevant. It is like hiring a team of forensic accountants to audit your daily coffee purchase.

The Portfolio Allocation

If you accept that most of language is local, the architectural answer becomes incredibly obvious: You take cheap, fast, myopic operators — like convolutions — and use them for the vast majority of the model’s layers. Let them handle the routine syntax. They don’t need a KV cache, and they run blazingly fast on a phone CPU.

Then, you take the expensive, memory-hogging exact retrieval mechanism — attention — and use it sparingly. You only wake it up when you actually need to find the poison on page 1. The ratio between the cheap layers and the expensive layers becomes the master control knob for your entire inference cost.

The Survival Channel

But if 80% of your layers are functionally blind convolutions that only look at the 3 words next to them, doesn’t the model just forget everything else?

It survives because of the residual stream.

When a token enters the model, its original mathematical meaning is posted to the chat. When a cheap convolution layer processes that token, it doesn’t delete the original meaning. It just figures out some local context and adds that annotation to the chat.

By the time the signal reaches the rare, expensive attention layer, that attention layer can read both the original token identity AND all the local context the convolutions gathered along the way. The convolutions didn’t destroy the signal; they just annotated it.

This is the exact hybrid insight Liquid AI used to build LFM2. How they found their ratios, and the rest of their technical details are all a work of art, which is exactly what we will be looking at next.

6) What is the Architecture of LFM 2, the best Edge AI Model

Sixteen total blocks. Ten are gated short convolution blocks. Six are grouped query attention blocks. That is the whole model.

Their 10-to-6 split is predicated on the observation that exact global retrieval through self — attention is being dramatically overspent in standard architectures. Most layers in a language model do not need to look at every previous token. They just need to process local patterns cheaply and move on. The expensive global operation should be deployed sparingly, like a specialist you call in for the cases that actually require it, not a default you run in every single layer.

Let’s break this down further. When you think about it, every block in a language model pays for two jobs.

• The first is token mixing: letting words talk to each other to absorb context. This is where the transformer’s budget either bleeds out or holds, because standard token mixing is self-attention (the expensive, KV-cache growing tax).

• The second job is channel mixing: processing the individual word after it has absorbed context. In LFM2, channel mixing is handled by a SwiGLU feed-forward network, which is a fixed cost and maintains no cache.

So, the entire question of whether an architecture can survive on a phone reduces to one decision: what do you use for token mixing? If you use attention everywhere, you pay the cache tax everywhere. LFM2 found a cheaper token mixer to handle the routine work.

The Gated Short Convolution Block: Adaptivity Without the KV Cache

Remember our observation that most text probably cares about a few local dependencies? This is where is becomes useful.

Most people think that if you want a model that adapts its behavior based on what it is reading, you need attention. Without it, you are stuck with dumb, fixed operations.

People are wrong (shocker).

LFM2 adapts its behavior based on every token it processes without a single byte of KV cache or persistent memory.

Here is how:

• Step 1: Split the input. The incoming token vector is multiplied by a learned weight matrix (a linear projection) to produce three separate vectors: B, C, and h_tilde . h_tilde is the working copy of the token’s information. B and C are gates — vectors of numbers that control signal flow. Critically, B and C are generated directly from the current token’s representation, meaning the model produces different gates for different inputs. The adaptivity is built in at step 1 before any mixing occurs.

• Step 2: Apply the first gate. Gate B is multiplied element by element against h_tilde to produce a new vector y. This first gate modulates the signal entering the convolution. Rather than reading “ambiguity,” the network dynamically adjusts its mathematical sensitivity based on the input vector, utilizing Linear Input-Varying (LIV) operators. Adaptive dynamics without the differential equations.

• Step 3: The convolution. The gated signal y passes through a depthwise 1D convolution with a kernel size of 3. A 1D convolution acts as a causal sliding window, combining the current token with the two immediately preceding it. “Depthwise” means it mixes positions within each channel independently, but does not mix between channels, deliberately keeping the operation cheap. The math is: z at position t, channel c = (w0 * y at position t, channel c) + (w1 * y at position t-1, channel c) + (w2 * y at position t-2, channel c). This requires three shared weights per channel and maintains a tiny rolling buffer that never grows, ensuring a constant cost per token.

• Step 4: The second gate and exit. Gate C is multiplied element by element against the convolution output z. This modulates the exiting signal before it is multiplied by a final weight matrix to map back to the model dimension.

Ten layers of this block provide fully adaptive token mixing with zero persistent memory.

But a kernel size of 3 means the layer is blind to anything four words away. How much of a problem is that really?

Why a Tiny Kernel Size of 3 Actually Works for Language

Liquid AI’s architecture search system (STAR) evaluated multiple kernel sizes. The search consistently rejected broader windows. While a size-64 kernel offered small quality gains, it incurred massive hardware penalties via more multiplications and worse CPU cache efficiency.

This is an empirical finding about language modeling: the mechanical work of processing language — subject-verb agreement, bigram statistics — is overwhelmingly local. Stacking small kernels also naturally grows the effective reach; two size-3 layers in a row allow information to diffuse across five tokens.

More importantly for builders, depthwise 1D convolutions are standard operations across inference frameworks like llama.cpp, ExecuTorch, MLX, and ONNX Runtime. Unlike Mamba, which requires custom associative scan kernels that many mobile stacks lack, LFM2’s local blocks are shippable today.

If ten layers handle the cheap work, the remaining six must justify their memory bill.

The 6 Attention Layers: Paying for the Premium Global Retrieval Service

LFM2 retains six grouped-query attention (GQA) blocks where the token can look at every previous token and maintain a KV cache. Standard multi-head attention gives every query head its own private Key-Value archive. LFM2 uses 32 query heads but restricts them to 8 KV groups, meaning every 4 query heads share the same Key-Value archive. This preserves 32 distinct attention patterns while dropping the storage cost by 4x.

Distance is handled by Rotary Position Embedding (RoPE), which rotates query and key vectors so the dot product measures relative distance. The convolution layers require no positional encoding at all, because their three weights are permanently assigned to specific positional offsets. Every component is optimized to lower the final cache math.

To maintain stability downstream of ten convolution blocks, LFM2 applies QK-Norm. Attention scores can grow without bound as training progresses — the model makes queries and keys “louder” over time, and the dot product spikes. The softmax then collapses into a hard winner-take-all that ignores everything except one token. The attention pattern stops being useful.

LFM2 normalizes query and key vectors before the dot product. After normalization, the maximum possible attention score is bounded by the square root of the head dimension — for 64-dimensional heads, that ceiling is 8. This matters specifically because these rescue layers sit downstream of ten convolution blocks accumulating local context. If the attention scores are unstable, the premium service that justifies the entire architecture’s design breaks down exactly when you need it most.

The Actual KV Cache Math: 192 MB vs 524 MB

Here is the formula I’m sure you have imprinted into your psyche by now:Cache per token = 2 * attention layers * KV groups * dimension per head * bytes per element

For LFM2: 2 * 6 layers * 8 groups * 64 dimensions * 1 byte = 6,144 bytes per token. At 32,000 tokens, that is roughly 192 megabytes.

Compare that to Llama 3.2 1B (same groups, dimensions, and precision, but 16 attention layers). That is 16,384 bytes per token, or 524 megabytes at a 32,000 token context.

Running 6 attention layers instead of 16 cuts the cache by 63%. Compare it to a full 16-layer standard multi-head attention stack, and LFM2 stores about 9.4% as much KV memory — a 90% reduction. The memory win compounds sparser attention with cheaper attention.

Turns out, we didn’t need to pay the complete cost of exact retrieval every layer.

But this chindi behavior creates a terrifying mechanistic problem.

How Information Survives the Blind Layers: Residual Connections

If ten layers can only see three tokens, how does a fact from 100 tokens ago survive long enough for an attention layer to retrieve it? Why doesn’t the representation turn into local soup before a rescue layer arrives?

The answer is residual connections.

If a layer simply completely transformed its input, early information would be destroyed immediately. Residual connections prevent this with an additive rule: output = input + Layer(input). The layer computes something and adds it on top of what came before. The raw token identity is always mathematically present in the stream because it was never subtracted out. Dimensional separation in the 2048-dimension vector allows the token identity and the local context annotations to broadcast on different frequencies without destroying each other.

Evidence suggests that after 4 to 8 consecutive blind layers, verbatim needle-in-a-haystack recall dies. Local annotations eventually contextualize the representation so heavily that exact identity becomes hard to recover. LFM2 spaces its 6 attention layers out to roughly one every 2.7 blocks, keeping it in the safe zone.

This is the explicit tradeoff. For forensic citation of 100,000-token documents, this architecture will underperform a pure transformer. But for on-device conversations, working ambiently (which means a lot of irrelevant tokens coming in), summarization, and agentic tool use, it dominates.

As alluded to earlier, Liquid AI didn’t just guess this exact 10/6 configuration. They built a machine to mathematically evolve the answer on actual phones under real constraints. That machine is called STAR, and its long-term value is stronger than any single model it produces. Instead of getting super into the nitty gritties, we will transition to exploring that now.

7) How STAR Works: The Architecture Search Machine

The 10 convolution layers. The 6 attention layers. Kernel size 3. Eight KV groups. The exact spacing of the rescue layers. None of that was guessed. Nobody sat in a room and had an inspired glasses being pushed up epiphany where the optimal edge architecture descended from heaven.

Hand-designing an AI architecture is, at best, an educated guess constrained by human intuition. When AI21 designed Jamba, they ran ablations comparing a 1:7 attention-to-Mamba ratio against a 1:3 ratio, found little quality difference, and picked 1:7 because it was more compute-efficient. That is better than guessing blind, but it is still a manual search through a tiny slice of the design space, limited to the clean ratios a human thought to test. However, the right architecture for a Samsung phone might be completely different from the right architecture for an AMD laptop, and neither is likely to be a clean ratio that looks good on a whiteboard.

Liquid AI built a search machine to do the embarrassing part: brutally test architectural ideas on real hardware until most of them die.

That machine is STAR (Synthesis of Tailored Architectures). If you are trying to understand what is actually durable about Liquid AI, STAR matters more than any individual 1.2B model ever will. Models age fast. Benchmarks move. But a system that takes a hardware target, a latency budget, a memory ceiling, and a quality suite, then mathematically evolves the architecture that best fits that exact regime? That is not one good model. That is a factory for good models.

(FYI — this is something that Google was one of the first to pioneer at scale. Their model training system — pathways — is one of the reasons why they can churn out amazing models at scale. Read more about it here).

Why Hand-Designing Architectures is a Deployment Hazard

Even for a 16-layer model, the possible configurations of operators, sharing patterns, expansion factors, and featurizers are endless. A researcher can explore maybe a dozen configurations in a focused effort. STAR evaluates entire populations across generations.

When the hardware target changes — say, from a Snapdragon 8 Gen 3 to a Gen 5 — the optimal architecture may shift in ways no human would predict because the cache hierarchy changed or the compiler handles operations differently. A researcher has to start over with new intuitions. STAR simply reruns the search.

There is also the bias problem. Researchers optimize for elegance. They gravitate toward architectures that have nice theoretical properties, look clean in a diagram, and make for a good NeurIPS oral presentation. Hardware does not care about elegance. Hardware only cares whether the operations map efficiently to its execution units. STAR has no aesthetic preferences; it only knows what is fast, what is small, and what scores well.

But to search this vast space effectively, you need a common mathematical language that can express all major sequence operators inside the same geometry. This is harder than you’d think, and why we need to really abstract things well (which makes me think that studying category theory might lead to some interesting breakthroughs for AI, especially for more meta-level work).

The LIV Abstraction: One Algebra for Every Architecture

STAR’s search space is built on a theoretical foundation called the Linear Input-Varying (LIV) operator.

In plain terms, an LIV operator produces its output by multiplying the input sequence by a linear transformation, but that transformation is itself generated from the input: Output = T(x) * x, where T is a matrix that depends on the input x.

What this buys the search system is total algebraic unification. It expresses attention, linear attention, state space models, convolutions, and gating operators as special cases of the exact same family:

• If T(x) is the softmax attention matrix built from query-key dot products, you get standard self-attention.

• If T(x) has the structured form of a state-transition operator, you land in Mamba territory.

• If T(x) is Toeplitz-structured with learned, input-dependent gating, you get LFM2’s gated convolution.

These terms might be a bit intimidating to you, so here is a visual that walks you through the same. It’ll make understanding this stuff a lot easier.

Because STAR moves through a continuous mathematical geometry rather than picking between discrete labels, it can discover intermediate or mixed structures that no human would have proposed. This turns the space between “convolution” and “SSM” from a vacuum to a rich set of unnamed configurations with (potentially) excellent hardware behavior.

The Architectural Genome: Evolving at Three Resolutions

To work effectively, STAR encodes every candidate architecture as a hierarchical genome — a sequence of integers describing the architecture at multiple scales simultaneously:

• Backbone Genome (Macro): One operator ID per layer. This decides the body plan — what kinds of blocks appear and where.

• Operator Genome (Wiring): Expands the operator ID into a short code specifying the token mixer, channel mixing pattern, and nonlinearity.

• Featurizer Genome (Micro): Defines expansion factors, repeat counts, and internal sharing patterns. These are the tiny choices that quietly determine whether a model ships or crashes.

This hierarchy allows evolutionary search to work at multiple resolutions. A mutation can swap whole operator types across layers, rewire one specific layer, or tune fine-grained details without disturbing the broader architecture.

Multi-Objective Evolution on the Pareto Frontier

STAR uses gradient-free evolutionary algorithms to optimize across multiple objectives simultaneously. It maintains a population of candidates, evaluates them, keeps the ones that are not dominated by any other candidate, and breeds them for the next generation.

This setup helps a lot when it comes to balancing multiple factors. When it comes to edge deployments, quality, latency, and memory must be threaded together. A model that wins on perplexity but blows the memory budget is useless. STAR navigates the synthesis by tracking the Pareto frontier — the exact set of architectures where you cannot improve one objective without degrading another.

“Given the wide range of possible applications of current AI systems, enabling systematic and automatic optimization of model architectures from the multitude of existing computational units is key to meeting the various demands these applications pose, in terms of efficiency (e.g., model size, inference cache size, memory footprint) and quality (e.g., perplexity, downstream benchmarks), and a prerequisite on the path to further, consistent improvements on the quality-efficiency Pareto frontier.”

Most evolved architectures outperformed hand-designed hybrids within just a few rounds of evolution. But despite these amazing results, this OG wasn’t good enough. Getting to LMF2 required an upgrade.

“Our earlier academic prototype (STAR) (Thomas et al., 2024) explored a specific design space of operator/layout choices with an evolutionary search heuristic optimized on proxy signals (i.e., perplexity for quality, cache size for efficiency). In practice, these proxies do not transfer reliably to downstream task scores or device-level latency and memory, limiting their utility as optimization objectives. By contrast, the LFM2 pipeline centers the objective: downstream task scores and hardware-in-the-loop TTFT/latency/memory on release runtimes. In practice, we found this has a much larger impact than the particulars of the search space or choice of search heuristic.”

The LFM2 Pivot: Why Liquid AI Replaced Proxies with Hardware-in-the-Loop

The original STAR paper optimized against proxy signals: perplexity for quality, and estimated cache size for efficiency. The LFM2 technical report is blunt about why they abandoned this: proxy metrics lie.

A Mamba layer and a convolution layer might have similar theoretical FLOPs, but Mamba requires sequential scan operations that stall the CPU pipeline, while convolutions map to highly optimized SIMD instructions. Theoretically, better performance will lead to worse outputs IRL.

Likewise, estimated cache size ignores activation buffers, framework overhead, and allocator behavior. The only metric that tells you if a model will crash a device is peak RSS — the actual physical memory the process consumes at its worst moment. This is something you have to get your hands dirty to measure.

The LFM2 pipeline fixed this by dragging the search onto real devices. Every candidate architecture is compiled into deployment format (llama.cpp or ExecuTorch) and loaded onto actual hardware: either a Samsung Galaxy S24 Ultra and an AMD Ryzen laptop to get both ends. They are profiled at batch size 1 across both 4,000 and 32,000 token contexts. Candidates that violate device-side budgets for time-to-first-token, decode latency, or peak RSS are instantly discarded.

This kind of search is extremely powerful, but it comes with a massive downside — it’s very very expensive. t requires thousands of GPU-hours for proxy training and a dedicated physical device lab for continuous profiling.

Google DeepMind, OpenAI, Anthropic, Chinese labs, Apple, and Meta all possess the capital and internal frameworks to replicate hardware-in-the-loop search. Liquid AI’s success depends on converting that lead into ecosystem lock-in through runtimes, developer tooling, and accumulated profiling data — before a hyperscaler decides to look into doing this. This means they need to move quickly; anything that can block them has to go.

And understanding this leads to understanding one of their most surprising design outcomes.

Why STAR Completely Rejected SSMs

For all their theoretical elegance, STAR completely rejected State Space Models for Edge Deployments.

Every major SSM variant was available in the search space. The LFM2 report explicitly lists S4, Mamba, Mamba-2, Liquid-S4, and S5 as candidates. STAR simply chose not to include them. The hardware-in-the-loop search repeatedly selected the minimal hybrid of gated short convolutions and grouped-query attention, ruthlessly excluding SSMs from the surviving architectures.

If you’ve read our prior deep dives and followed along with the journey, the reasons might be clear to you. So instead of repeating stuff, here is a little visual that you can save to remember the details on the go — 

This lays all the goundwork for us to finally understand Liquid’s newest model.

8) How Does Liquid AI Train Small Models to Compete With Much Larger Ones?

LFM2.5–1.2B matches or beats Qwen3–1.7B — a model with 42% more parameters — on multiple benchmarks. Let’s understand the four specific training innovations that make this happen.

How the Student Model Learns From a Bigger Teacher Without Storing Everything

Knowledge distillation trains a small model (the student) to mimic a large model (the teacher). Instead of just learning to predict the correct next token, the student learns to match the teacher’s full probability distribution — not just “the answer is X” but “X is most likely, Y is a decent second choice, Z is plausible, everything else is garbage.” That ranking information makes the student significantly smarter than learning from raw data alone.

LFM2 models are distilled from LFM1–7B. The problem is storage. The teacher’s full distribution over a 65,536-token vocabulary for every position in a multi-trillion-token corpus would require petabytes. Liquid AI stores only the top 32 logits per token — a 2,000x compression.

But truncation breaks the standard training math. Here’s why.

The standard way to measure how well the student matches the teacher is called KL divergence — it’s essentially a score that says “how different are these two probability distributions?” Lower score means the student is mimicking the teacher well. Higher score means it’s off.

To help the student learn, you typically apply temperature scaling — which smooths out the teacher’s predictions so they’re less extreme. Instead of “token X has 90% probability and everything else is near zero,” temperature scaling softens it to something like “token X has 40%, Y has 25%, Z has 15%…” This makes the ranking information easier for a small model to absorb.

The problem: when you only stored 32 out of 65,536 tokens and then apply temperature scaling, the math tries to spread probability to all 65,536 tokens — including the 65,504 you have no teacher data for. The loss function is trying to match a distribution that doesn’t exist. Training becomes unstable. The model oscillates instead of learning.

Liquid AI’s fix splits the matching problem into two separate parts:

• Term 1: Membership loss. How much total probability mass should the student assign to the teacher’s top-32 tokens versus everything else? A binary comparison. No temperature scaling — because temperature applied to a truncated distribution would smooth probability toward tokens you have zero information about.

• Term 2: Conditional ranking loss. Within those top-32 tokens, how well does the student match the teacher’s relative ordering? This term gets temperature scaling, because both distributions are defined over the same 32 tokens. No mismatch.

At temperature 1, this decomposition is provably a lower bound on the full KL divergence — the full KL decomposes into three terms (binary, conditional top-K, conditional tail), and dropping the non-negative tail term can only decrease the total. At higher temperatures, the loss becomes a tempered surrogate with a smoother optimization landscape.

This leads us to the next phase of training.

Why Training Examples Are Ordered From Easy to Hard

A 1.2B model has limited capacity. Hitting it with the hardest training examples before it has learned basic patterns produces noisy gradients and wasted compute. LFM2’s supervised fine-tuning stage orders examples from easy to hard.

Difficulty is scored by an ensemble of 12 models ranging from 350M to 235B parameters. For each question, the ensemble computes the fraction that answered correctly. High fraction = easy. Low fraction = hard. Training proceeds from easy to hard using a predictive model to rank questions by estimated difficulty.

Difficulty is measured externally by the ensemble, not by the student’s own performance. This avoids the circularity that kills self-paced learning: a poorly trained student misjudges what’s hard, avoids exactly the examples it needs, and stays poorly trained.

Three Stages of Post-Training: Fine-Tuning, Preference Alignment, and Model Merging

After pretraining on 10 to 12 trillion tokens, the model goes through three post-training stages (in case you’re wondering, this is a massive pretraining budget; and is likely why the model has really good knowledge and capabilities; 2.5 pushed this to 28 Trillion, which validates this assumption).

Stage 1: Supervised Fine-Tuning (SFT). 5 to 9 million training examples across 67 to 79 data sources. Roughly 27% general-purpose, 17% instruction following, 13% retrieval-augmented generation, 10% tool use, remainder split across code, math, multilingual, and domain-specific tasks. 80% English, 20% multilingual.

Stage 2: Preference Alignment via DPO. The model generates 5 candidate responses per prompt. An LLM judge scores them. The best and worst form preference pairs. Direct Preference Optimization trains the model to increase the probability of preferred responses and decrease dispreferred ones.

Two details matter for small models.

1. The KL regularization coefficient (beta) is 5.0 — roughly 10x higher than typical DPO (0.1 to 0.5). A 1.2B model can’t afford to deviate far from its SFT-trained behavior without catastrophically degrading general capabilities. The high beta constrains the optimization to targeted improvements.

2. Length normalization is equally important — without it, DPO biases toward shorter responses because longer completions accumulate more log-probability, making them look worse regardless of actual quality.

Stage 3: Model Merging. Multiple candidate checkpoints are generated by varying hyperparameters and data mixtures. Each excels in different areas. The best checkpoints are combined into a single model using weight-space averaging.

Five merging methods are tested: linear averaging (Model Soup), Task Arithmetic, TIES-Merging, DARE, and DELLA. Each handles the combination differently — some average all weights, some merge only the weights that changed most, some randomly drop small changes and rescale. The best merged model is selected on a comprehensive benchmark suite. Merging costs nothing at inference — the output is one model with the same parameter count as any individual checkpoint.

Why Liquid AI Quantizes During Training, Not After

All LFM2 and LFM2.5 models are trained with INT4 quantization in the loop from the start. The model learns to produce good outputs despite rounding errors, rather than being trained at full precision and then compressed (post-training quantization). This matters more for small models — a 70B model has enough redundancy that quantization noise barely registers. A 1B model has no slack. Every parameter is working harder, and rounding any of them introduces proportionally more damage.

Conclusion: Where Does Edge AI Go Next

We didn’t spend eight sections tearing apart cache math and evolutionary algorithms just to marvel at a clever 1.2B model. Liquid AI is merely the diagnostic tool. The real story is that the entire trajectory as edge AI and all of it’s implications unfold. However, this is a completely different computing paradigm, with it’s own problems that we musst all be ready to solve.

The Signal-to-Noise Crisis: Learning to Ignore

Right now, if you type a prompt into ChatGPT, 100% of those tokens are intentional. You wrote them. The model assumes every word matters. Even with typos and misspellings, the signal to noise ratio in your input is very high.

Ambient computing is the exact opposite. If an AI is listening to your microphone for 8 hours a day, 99% of what it hears is useless garbage. It hears you typing, a siren outside, small talk about the weather, and someone clearing their throat. Maybe 1% of those tokens are a concrete task or an important decision.

If you feed 8 hours of ambient noise into a standard transformer, it will dutifully calculate exact global attention scores for the siren and the throat-clearing, filling up its KV cache until the phone crashes. Or it will find ways to bill you for the Raja Raja Raja song you played 10x in a row.

All this means the next major architectural leap in edge AI won’t just be about cheaper math. It will be about active ignoring. Just like the human brain filters out the feeling of your shirt against your back so you can focus on reading this sentence, edge models will need adaptive compute mechanisms that instantly classify and discard useless tokens before they ever reach the expensive layers. The architecture will have to shift from “how efficiently can I process this context?” to “how aggressively can I refuse to process this context?”

The Hardware Fragmentation Problem: The Case for Open Standards

The data center is a monoculture. If you write your model in CUDA, it will run beautifully on an NVIDIA GPU, which is what every hyperscaler uses.

The edge is a chaotic wasteland.

There is no “standard” edge hardware. A Snapdragon NPU handles memory differently than an Apple Neural Engine, which handles operations differently than an AMD Ryzen CPU. As Liquid AI’s STAR system proved, the optimal architecture for one phone might run like garbage on a laptop. There are a million different Pareto frontiers.

If you are a startup or a smaller AI team, you cannot afford to build a dedicated hardware-in-the-loop search system to evolve a custom architecture for every single Android device on the market.

This fragmentation forces a strategic fork in the road. Either the massive players (Apple, Google) completely capture the edge because they own the vertical hardware stack and can optimize perfectly for it, OR the open-source community is forced to consolidate around open standards. We will likely see a push for unified inference runtimes and open hardware-profiling datasets. Smaller players will have to pool their testing resources, effectively building a communal “STAR” system, just to survive the hardware chaos.

Unlocking Hostile Markets

Finally, the obsession with edge AI isn’t just about saving battery life on a smartphone. It is about unlocking entirely new markets that are fundamentally hostile to GPUs.

There are massive sectors of the global economy where you cannot just “ping the cloud.” For the last three years, these industries have been locked out of the generative AI boom because the models required server racks they weren’t allowed or able to use.

Architectures like LFM2 change the math. When you can run a highly competent, reasoning-capable model locally in 700 megabytes of RAM, you bypass the cloud entirely. You bring the intelligence to the data, instead of trying to drag the data to the intelligence.

The transformer won the last decade because it was perfectly adapted to the data center — an environment where memory is treated as infinite. The architecture that wins the next decade will be the one that understands memory is a hostage negotiation. And the companies that master that negotiation are about to put AI into everything.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

Share

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

A study from Wharton’s Generative AI Lab tested 198 PhD-level questions across biology, physics, and chemistry. Chain-of-thought instructions — the single most popular prompting technique since 2022 — bought 2.9 to 3.1 percent accuracy on reasoning models while adding 20 to 80 percent latency. On Gemini Flash 2.5, chain-of-thought made results worse. Negative 3.3 percent. You’d have gotten better answers by not trying to help. “Mind Your Step (by Step)” (COLM 2025) went further: on pattern recognition tasks, turning on reasoning mode dropped accuracy by up to 36.3 percent versus a standard model. The technique designed to make models smarter is making the smart models dumber.

The providers know. OpenAI, Anthropic, Google, DeepSeek — all of them explicitly warn against chain-of-thought on reasoning models. But the advice economy runs on lag. Most courses, research, and common tips are focused on the older generation base models, making them outdated for the current paradigm, since base LLMs have a very different post-training and alignment system compared to reasoning LLMs.

In this deep dive, we will combine our conversations with the builders of various AI models, dig into research papers, and compile insights from various practitioners to give you deep insight into the mechanics of reasoning models, why the standard prompting techniques that you’re taught online actually hurt reasoning models, and how you can prompt them better. This article will also give you eight rules grounded in the research that will work across model families and architectures so you can apply these insights to the model of your choice.

^^A preview of what you’re getting today.

This article is written for the technical layman, with no deep AI or Software Engineering skills required. All you’ll need is an attention span, a desire to learn, and a willingness to experiment and internalize knowledge. If you match that, this article will help you skyrocket your productivity and do higher-quality work in less time.

Let’s get into it-

For access to this article and all future articles, get a premium subscription below.

Subscribe now

Each of these articles takes a long time to research and write, and your premium subscription allows me to deliver the highest quality information to you. If you agree that high-quality work deserves compensation, please consider a premium subscription. We have a flexible subscription plan that lets you pay what you can here.

PS- Many companies have a learning budget that you can expense this newsletter to. You can use the following for an email template to request reimbursement for your subscription.

Read more

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

There’s an inflection point in training systems at scale where the hard problem changes shape. It stops being about whether you can get the job to run and becomes something more corrosive — figuring out why performance degraded, or why a failure signal appeared, or why throughput quietly cratered and nobody can explain it. That’s where the real cost lives. Not compute. Engineering hours burned reconstructing what went wrong from scattered dashboards and whatever the senior SRE remembers from last time.

The following research from AMD was shared with me and I thought it was worth your time (goes without saying but this is not sponsored, I share only because I think you need to read it). So I’m sharing it with due permissions.

It walks through the failure modes that actually eat people alive once clusters get big enough. This includes various issues like

• Network degradation that looks like a training bug.

• False failure signals that trigger unnecessary restarts.

• Checkpoint-related slowdowns.

• Cases where everything looks broken but the root cause is in the training dynamics, not the infrastructure.

Debugging large-scale training is moving from tribal knowledge and human pattern-matching toward diagnostic workflows that a system can actually execute. And the teams that figure out how to make that transition — how to turn their debugging knowledge into repeatable infrastructure instead of leaving it trapped in someone’s head — those are the teams that will compound their advantage over everyone else. This article will give you insight on the same.

In MaxText-Slurm: Production-Grade LLM Training with Built-In Observability, we introduced MaxText-Slurm — an open-source launch system and observability stack for running MaxText LLM training on AMD Instinct GPU clusters. We showed how a unified Prometheus time-series database (TSDB) collects GPU, host, network, and training metrics into a single queryable store, persisted to disk so that no data is lost even if the job crashes.

A unified TSDB is only as useful as the methodology applied to it. In this post, we show the agentic diagnostic skills that ship with MaxText-Slurm — structured runbooks that Cursor or Claude Code execute autonomously from symptom to root cause. This is not a chatbot answering questions about logs. The AI agent has tool access (shell, file system, HTTP queries to Prometheus) and follows each skill systematically: reading logs, querying metrics, interpreting results, and chaining steps until it reaches an actionable conclusion. We walk through five case studies — from one-prompt performance profiling to a throughput decline (measured in tokens per GPU per second, or TGS) whose root cause turned out to be model behavior — where each short prompt led to an actionable result in minutes.

Setting Up the AI Agent

The AI agent needs access to two codebases: the maxtext-slurm repo (for skills, job outputs, and diagnostic tools) and the MaxText source (for code-level tracing during deep diagnosis). MaxText lives at /workspace/maxtext inside the training Docker image — a frozen snapshot baked into the container, not installed on the host. The most reliable setup is to run the agent inside the container on a cluster login node.

Step 1: Clone the repo and start the container

SSH into a cluster login node and clone the repo onto a shared filesystem so that multi-node job outputs (logs, TSDB, profiles) are accessible from any node:

ssh cluster-login-node
cd /shared               # or any shared filesystem path
git clone https://github.com/AMD-AGI/maxtext-slurm.git
cd maxtext-slurm

Start an interactive container on the login node. run_local.sh bind-mounts the repo directory into the container, so job outputs on the shared filesystem are directly accessible. Diagnosis workflows (log triage, TSDB queries, source code tracing) do not require GPUs, so the login node is sufficient:

run_local.sh             # drop into an interactive container shell

Tip: To leave the container without stopping it, use Ctrl+P, Ctrl+Q to detach. Do not type exit — that stops and removes the container, destroying any in-container state. You can reattach later with docker exec -it <container_name> bash. Running on a compute node is possible but requires additional SSH setup not covered here.

Inside the container, the filesystem looks like this:

/maxtext-slurm/          # maxtext-slurm repo (bind-mounted from the host)
    skills/              # AI diagnostic skills
    outputs/             # job outputs (logs, TSDB, ray_logs, ...)
    utils/               # prometheus.sh, analyze_job.py, metrics plugins
/workspace/maxtext/      # MaxText source (baked into the image)
/opt/venv/lib/python3.12/site-packages/
    orbax/               # Orbax checkpoint library
    jax/                 # JAX source

Step 2: Connect the AI agent

You need two connections to the login node — one inside the container for the AI agent, and one outside for Slurm operations (submit.sh, squeue, scancel, SSH tunnels to live dashboards, etc.). This separation is deliberate: the AI agent inside the container can read logs, query Prometheus, and trace source code, but it cannot submit, cancel, or modify Slurm jobs. Keeping Slurm access outside the container prevents an agent mistake from affecting running jobs on the cluster.

Cursor — Use Remote SSH to connect to the cluster login node, then attach to the running container via Dev Containers: Attach to Running Container. Add both /maxtext-slurm and /workspace/maxtext as workspace folders so the agent can cross-reference skills, job outputs, and framework source code in a single session. Use a separate terminal (SSH session to the login node, outside the container) for Slurm operations. Figure 1 shows this setup.