Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

GLM-5.2 is the best open-weight model on coding and agentic benchmarks right now. It scores 1524 Elo on GDPval-AA, a real-world agentic work benchmark — ahead of every other open model by a wide margin and level with GPT-5.5. It hit #1 on Design Arena’s Code Categories. It scores 81.0 on Terminal-Bench 2.1 and 62.1 on SWE-bench Pro, landing within a few points of Claude Opus 4.8 on long-horizon coding tasks — at roughly one-sixth the per-token cost of GPT-5.5.

Much more importantly, its intelligence has been validated by multitudes of users, which is a strong indication that Zhipu did not simply game the benchmarks to come out on top.

So what makes GLM 5.2 so good?

GLM-5.2 combines four specific techniques to eliminate memory and compute bottlenecks across the agentic loop. Rather than relying on raw model scale, these architectural mechanisms act multiplicatively to sustain throughput during long-horizon tasks.

• Deepseek Sparse Attention: Cuts long-context memory retrieval costs.

• IndexShare: Reduces hardware memory bandwidth overhead via cross-layer weight amortization.

• Multi-Token Prediction: Multiplies generation throughput by predicting parallel tokens per forward pass.

• Critic-Based PPO via SLIME: Trains the architecture to learn from complex, long-horizon agent trajectories.

In this article, we will study how each of these techniques comes together to give GLM 5.2 it’s god tier performance. While we will touch on the benchmarks and other techniques, I want to be very clear that they will not be the focus of this deep dive. My goal isn’t to give you a summary/overview of GLM 5.2, but to instead go deep on the few decisions that make GLM 5.2 stand out from every other model, and to understand the implications of GLM’s design for all builders and investors with one goal: understand GLM 5.2 beyond the headlines, so that you predict where the industry is headed next. While we will go very deep, you will not need much knowledge to read since we’ll develop our intuition from first principles.

If that sounds good to you, let’s dig right in.

Executive Highlights (tl;dr of the article)

Agentic loops have to contend with 3 issues that make them very expensive: Long-context histories (dealing with input, subagents, tool calls etc); unparallelized multi-step token generation (from the subagents); and broken training loops. Zhipu’s GLM-5.2 tackles these distinct bottlenecks across both the training and inference layers.

• The Long-Context Memory Wall: Standard multi-head attention hits a quadratic complexity limit at extended contexts. GLM-5.2 implements content-dependent DeepSeek Sparse Attention (DSA) to compute attention only over a sparse subset of the top 2,048 tokens per head, yielding a 2.9x reduction in per-token FLOPs at a 1-million-token window. In the main article, we map the exact hardware trade-offs of this approach — specifically tracing how IndexShare reuses token selections across fixed four-layer blocks to eliminate 75% of indexer computations, delivering a 1.82x prefill speedup at the cost of severe KV cache fragmentation and increased systems serving complexity.

• Unblocking Latency in Multi-Step Generation: Standard autoregressive generation forces millions of sequential, unparallelized forward passes to process complex tool calls and reasoning traces. GLM-5.2 addresses this output bottleneck by expanding speculative decoding to a 5-token draft window. Further down, we break down the architectural sharing, rejection sampling, and joint total variation loss training that Zhipu used to minimize distribution differences between models, pushing the average speculative acceptance length to 5.47 tokens per pass.

• The Mathematical Breakdown of Credit Assignment: While the industry has widely adopted DeepSeek’s GRPO to eliminate value networks and save GPU memory during training, uniform trajectory-level rewards completely blur credit assignment on long-horizon agentic runs. GLM-5.2 explicitly returns to a heavy critic-based PPO pipeline. Our deep dive will provide a first-principles analysis of this transition, demonstrating how independent, per-token value baselines isolate decisive debugging steps and integrate with compaction-aware training to slash a 16-hour group-sampling compute requirement down to 2 hours.

• Adversarial Exploitation and Post-Training Consolidation: Maximizing a generalist model across conflicting domains (coding, web search, math) usually triggers catastrophic forgetting. Zhipu resolves this by optimizing separate domain specialists and merging them via On-Policy Distillation (OPD). We examine how the SLIME post-training infrastructure consolidated over 10 specialists in 48 hours, while detailing the two-stage online validation modules (rule-based filtering and inline LLM judges) deployed to stop agents from aggressively reward-hacking infrastructure, downloading solutions via curl, and auditing evaluation files during reinforcement learning.

It’s worth noting that one of the key drivers of GLM 5.2’s amazing performance is their exceptional multi-level design where each of their techniques feeds each other.

On the inference side, we see a nested mathematical multiplier:

• DSA first restricts the context window to a fraction of its original size →

• IndexShare then executes inside that already-reduced window, bypassing 75% of the remaining indexing passes →

• Multi-Token Prediction takes these combined upstream memory savings and compounds them during generation by outputting multiple tokens simultaneously per verification pass.

The hardware is not just receiving individual linear speedups; it is running fewer attention operations, executing fewer indexing passes within those operations, and maximizing output density on every final activation.

On the training side, the anti-hack module acts as the first gate, intercepting exploits to protect the objective function from corruption. Because the training data remains clean, the critic network can map an authentic value landscape across compacted sub-traces. This calibrated value landscape allows the PPO gradient to isolate precise token-level advantages, which directly accelerates policy updates per unit of compute. Finally, On-Policy Distillation locks in these completed, domain-specific updates without gradient interference.

This to me is the true takeaway from GLM 5.2 — systems engineering will always beat individual techniques. Far too many team spend their time optimizing for individual aspects of performance without considering the entire system they’re operating in. When making building/investment decisions, keep this in mind.

FYI: We interviewed Zhipu AI over here. If you’re interested in their building philosophy, you might find it interesting.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Why are Agentic Loops So Expensive?

Agentic loops have a very specific computational signature that creates compounding resource strains in three ways:

• Repeated long-context retrieval: An agent working on a complex task continuously accumulates context. Each turn of the loop requires the model to re-query this entire history. Because standard attention mechanisms scale quadratically with sequence length, attending to a 100,000-token history costs $10,000$ times more compute per attention head than a 1,000-token history. The longer an agent works, the more expensive every subsequent decision becomes. (read more about the costs of running AI here).

• Sequential multi-step generation: Unlike simple question-answering systems that return a single response, agents generate massive streams of tool calls, code blocks, and reasoning traces across many consecutive steps. Autoregressive decoding forces each token to wait for the previous one, requiring a separate forward pass. An agent generating 50,000 tokens across an entire trajectory demands 50,000 sequential, unparallelized model activations.

• Long-horizon credit assignment: Training agents using Reinforcement Learning (RL) suffers from sparse rewards. A coding agent might perform hundreds of distinct actions over several hours. It will only receive a binary success or failure signal when the final test suite runs. Figuring out how important each step was to the overall outcome creates a massive credit assignment problem. Standard RL relies on gathering thousands of complete trajectories to isolate variables, but this is basically impossible to do effectively with all the sources of variance in agentic runs.

GLM finds ways to attack these problems both in the training and inference layers. Let’s get into them, one at a time.

How GLM 5.2 Uses Deepseek Sparse Attention to Reduce Long-Context Costs.

As we’ve talked about before, standard MHA has quadratic complexity since every token has to attend to every other token.

To avoid this quadratic cost, earlier efficient attention methods like Longformer used fixed sliding windows and predetermined sparse patterns. They forced the model to attend to nearby tokens and a few global positions regardless of the text content. Because fixed patterns cannot adapt to the actual input, these methods suffer from a severe quality gap.

Linear attention variants like GDN attempted to scale context efficiently but still introduced measurable degradation. They dropped up to 5.69 points on the RULER benchmark at 128K context and 7.33 points on RepoQA. Coding agents break these rigid architectures. An agent debugging code might need to link an error message from 500,000 tokens ago with a function definition from 200,000 tokens ago. Fixed or linear patterns completely miss these non-contiguous, long-range dependencies.

The DeepSeek Sparse Attention Solution

“We use DSA in our training. The core philosophy of DSA [9] is to replace the traditional dense O(L 2 ) attention — which becomes prohibitively expensive at 128K contexts — with a dynamic, finegrained selection mechanism. Unlike fixed patterns (like sliding windows), DSA “looks” at the content to decide which tokens are important.”

— From the GLM Technical Report.

DeepSeek Sparse Attention (DSA) changes this by using a content-dependent, two-stage process. Instead of hardcoding which positions to watch, a lightweight indexer predicts which tokens are most relevant to the current query in real time. Attention is then computed only over that sparse subset, limiting the workload to the top 2,048 tokens per head.

This gets you a 2.9x reduction in per-token FLOPs at a 1-million-token context, slashing the cost of every agentic loop execution drastically. Z.ai applied DSA to every layer of the model, claiming the mechanism is lossless (look at the chart below) — 

Developments like this are very exciting since they put a downward pressure on the already heated chip memory market. However, this solution comes with it’s own set of challenges. s the DSA access patterns paper documents, the token-dependent selection pattern means the KV working set becomes fragmented and volatile, creating poor cache locality and potential throughput stalls during decoding — 

This kind of optimization adds a lot more system complexity to your serving process. This is a class of AI startup/system that’s completely overlooked in the current era —  non-genAI solutions that work on to optimize and flag things like buffer overflows, memory management, and garbage collection. This has been valuable before AI, but the market will open up a lot more now that every inference call made compounds the value of resource optimization.

Even done well, DSA solves the attention scaling problem, but it introduces a new one: the indexer itself. At 1M tokens, the indexer has to score every token in the context to select the top 2,048 — and it has to do this for every layer in the model. The indexer computation becomes a meaningful fraction of the total cost. Try to fix one problem, and your solution creates new problems. Life really is fun that way.

Let’s look at how Zhipu tackled that next.

How GLM 5.2 uses IndexShare to cut the Cost of Sparse Attention?

The naive implementation of content-dependent sparse attention assumes that every layer must look at entirely different parts of the context. Under this assumption, an architecture computes a fresh top-k token selection at every single layer. However, this brute-force approach wastes enough hardware resources to turn Nvidia interns into millionaires since forcing every layer of an LLM to materialize its own indexer requires constant, redundant memory round-trips to evaluate token relevance across long sequences. When an agent runs a 1-million-token context, these repetitive indexing passes create a major throughput tax.

This is what IndexShare was built to fix.

Instead of running a separate indexer inside every transformer layer, the model executes one lightweight indexer across a grouped block of four layers and reuses the selected token indices across that entire group. The indexer sits at the first layer of the four-layer block, computes the top-k selection, and passes those identical indices to the following three layers for their sparse attention math. This structural change eliminates 75% of all indexer computations.

This optimization exploits the high similarity and continuity in adjacent layers of neural networks. Empirical measurements of top-k token overlap in dynamic sparse attention models reveal that adjacent layers share between 70% and 100% of their top-k selections near the diagonal. The tokens relevant at layer N remain fundamentally relevant at layer N+1.

(this similarity breaks down at transition layers, where the model shifts from one processing mode to another. The same paper reports overlap dropping to 0.4 or below at certain layer boundaries. Sharing indices across those transitions hurts quality. This is why IndexShare uses a fixed 4-layer group size rather than sharing across the entire network — it is calibrated to stay within the high-overlap regime while avoiding the dangerous transitions. They also use loss-aware selection, meaning the sharing policy is explicitly calibrated based on its direct effect on the training objective.)

Amortizing the indexing workload yields immediate hardware efficiency gains. IndexShare delivers up to a 1.82x speedup during the prefill stage and up to a 1.48x speedup during autoregressive decoding. For production agent deployments, these gains compound. Because an agent spends hours continuously prefilling extensive code history and sequentially decoding multi-step reasoning traces, these hardware speedups accelerate every individual execution turn. Even under fairly conservative assumptions, this can net you six figure savings — 

How does Multi-Token Prediction Accelerate Agent Output for GLM 5.2?

DeepSeek Sparse Attention and IndexShare reduce the hardware cost of reading context. Multi-Token Prediction (MTP) addresses the opposite side of the agentLoop: reducing the cost of writing the output.

Standard autoregressive generation requires a full model forward pass for every single output token. The model produces token one, executes a complete pass, produces token two, executes another complete pass, and repeats this loop sequentially.

This doesn’t work for agents since an autonomous agent routinely outputs tens of thousands of tokens per execution turn to process tool calls, long reasoning chains, and full code blocks. This means your system ends up with millions of sequential passes, slowing down your output tremendously.

GLM-5.2 shifts this dynamic via speculative decoding, using a smaller, low-cost “draft” model to guess candidate tokens quickly so the large target model can verify them simultaneously in a single forward pass.

While previous versions used a 3-token draft window, GLM-5.2 extends this proposal capacity to 5 tokens. Data from the architecture’s implementation reports a compounding optimization path that maximizes the average number of accepted tokens before a mistake forces a restart:

• Baseline Performance: The raw configuration starts with an average acceptance length of 4.56 tokens.

• Architectural Sharing: Integrating IndexShare and KVShare directly into the MTP layer allows the draft model to reuse internal structural context, lifting the acceptance length to 5.10 tokens.

• Statistical Refinement: Applying rejection sampling during token verification pushes the average approval metric to 5.29 tokens.

• Joint Training: Implementing end-to-end total variation loss forces the draft and target models to minimize their distribution differences during training, maximizing alignment and driving final acceptance length to 5.47 tokens.

So far, everything we’ve covered is relatively straightforward. However, the last technique changes a lot of things — 

How GLM 5.2 Changed Reinforcement Learning for Agentic Systems?

DSA, IndexShare, and MTP lower inference costs, but maximizing agent quality is a training problem. GLM-5.2’s return to a critic-based reinforcement learning pipeline marks a deliberate departure from recent industry trends. Evaluating this architecture requires analyzing why standard algorithms diverge on long-horizon tasks.

Background: How GRPO Works and Why It Dominated

Ever since Deepseek, Group Relative Policy Optimization (GRPO) is the standard reinforcement learning approach for large language models. The algorithm samples a group of N complete outputs from the current model for each training prompt, typically using 8 or 16 rollouts. A reward function scores each output, and the system computes a relative advantage by subtracting the group’s mean reward and dividing by the standard deviation.

GRPO assigns this group-level advantage uniformly to every token in that trajectory, giving token 1 the identical credit signal as token 10,000. The model update adjusts the probability of the entire token sequence based on this trajectory-level outcome.

This approach took off b/c its design eliminates the traditional PPO critic network, which requires a separate value network matching the policy model’s size and doubles the GPU memory footprint during training.

However, this has a slight problem..

Why GRPO Breaks on Agentic Tasks

GRPO relies on structural assumptions that long-horizon agentic tasks violate in four specific ways:

• Linear Scaling of Generation Costs: Sampling 8 to 16 complete trajectories requires minimal compute for short tasks, but a coding agent running a two-hour task across 50,000 tokens is a needy motherfucker that requires hours of compute per training prompt. This linear cost curve makes group generation prohibitively expensive, forcing developers to restrict group sizes and degrade advantage imprecision.

• Context Compaction: Production agents utilize context compaction to periodically summarize early history and respect context windows, breaking single long trajectories into variable sub-traces. One rollout might yield two compacted fragments while another produces eight under the same prompt. GRPO cannot apply its comparison framework because the fragments have different lengths, varying internal alignments, and non-aligned structures.

• Uniform Advantage Blurs Credit Assignment: Assigning an identical advantage weight to every token in a 50,000-token agent trajectory treats boilerplate imports and routine helper functions as equally valuable to success as a critical debugging fix at token 30,000. GRPO cannot isolate the few decisive actions among thousands of neutral tokens. The signal-to-noise ratio degrades proportionally with sequence length, severely delaying policy convergence.

• Coarse Rewards Drive Zero-Signal Blocks: Coding environments generally rely on binary pass/fail outcomes from automated test suites (the “verifiable rewards”, which is also why they struggle with design and architecture decisions which are not verifiable). If a training run samples 8 trajectories for a hard problem where 7 fail and 1 passes, the advantage math grants a high positive value to the single success and minor negative values to the failures. While this provides a gradient direction, the binary reward fails to explain why the successful trajectory worked. Furthermore, if all 8 trajectories fail — or all 8 pass — the group standard deviation drops to zero. This zeroes out the computed advantage for every token, completely freezing the training signal and wasting expensive hardware cycles.

Essentially, GRPO gives out participation trophies. Participation trophies are no good if you’re trying to sort out your star players from the genetic defects. So, how do we fix this issue?

“For GLM-5.2, long-horizon tasks produce substantially longer execution traces, and once a super-long trajectory is split by compaction into multiple sub-traces, different rollouts under the same prompt yield different numbers of trainable traces with highly variable lengths. We therefore move from group-wise optimization to a critic-based PPO formulation that learns from individual rollouts, relying on a critic to estimate token-level advantages rather than group-relative comparisons. This single-rollout formulation fits compaction naturally, as it places no constraint on how many traces a prompt produces or on their relative lengths: we bring compaction into training by including all compacted sub-traces as trainable trajectories, and apply a token-level loss to address their length imbalance.”

Using Critic-Based PPO to Isolate Token-Level Value

Critic-based PPO bypasses group comparisons by running a separate value network alongside the policy model to estimate expected rewards from any given context state. Instead of measuring total trajectory outcomes, PPO calculates the specific advantage of an individual token choice relative to the critic’s baseline expectation for that exact position.

If the critic estimates an expected future reward of 0.3 from the current state, and the agent executes a decision that shifts the subsequent state expectation to 0.7, that specific token choice receives a high positive advantage signal. Conversely, an error that drops the expected reward from 0.3 to 0.1 receives a negative advantage. Decisions that exactly match the critic’s baseline expectations yield a near-zero advantage.

This mechanism fundamentally changes credit assignment behavior in long-horizon workflows:

• Per-Token Baselines Protect the Signal: Routine operations like boilerplate code and standard library imports generate near-zero advantages because the critic accurately predicts their outcome. When the model hits a critical debugging step at token 30,000 and unblocks the execution path, the critic’s value estimate jumps sharply. PPO isolates this specific token choice for a major probability update while keeping neutral positions unchanged. The signal-to-noise ratio remains stable regardless of overall trajectory length.

• PPO Consumes Single Trajectories: PPO computes its baseline internally via the critic, removing the requirement for group sampling. The system requires only one trajectory per prompt, cutting compute demands for a long coding task from 16 hours down to 2 hours.

• Sub-Trace Variations Integrate Naturally: Because PPO calculates loss on a per-token basis using independent value estimates, it natively handles compacted fragments. Sub-traces of varying lengths contribute proportionally to the global gradient without requiring alignment across separate rollouts.

• Binary Signals Propagate Backward: PPO resolves the limitations of sparse rewards by using the critic to distribute terminal pass/fail signals backward through the execution history over successive training iterations. The critic learns to identify intermediate markers of success — such as clean code syntax or accurate diagnostics — and assigns higher value estimates to those states. The final binary reward eventually gets distributed backward into nuanced per-token signals through the critic’s value landscape, not through the reward itself.

However, running a critic network equal in scale to a 753B parameter model like GLM-5.2 doubles the required GPU memory during training phases. For short, low-complexity tasks where group statistics remain accurate, this massive memory overhead represents a net hardware loss.

So how do you know when the switch is worth it? As always, you can rely on Daddy Dev to help you. By and large, there are 3 conditions, which when met, will justify the switch into critic-based PPO:

1. Group generation costs become prohibitively expensive due to hours-long trajectory lengths.

2. Trajectory-level averages become too noisy to guide multi-step choices.

3. Sparse binary rewards frequently yield zero-signal groups.

Long-horizon agentic workflows checked all 3 boxes, so Zhipu decided to spend big here.

It’s also worth noting that GLM-5.2’s PPO implementation includes an optimization detail: compaction-aware training. The model trains directly on the compacted sub-traces and lossy summaries generated by the agent framework, forcing the critic to compute expected values within degraded context structures rather than pristine histories.

In real-world production deployments, agents continuously condense past conversation turns and code executions to operate within fixed context limits. If a critic trains exclusively on uncompacted transcripts, its internal value functions become miscalibrated when encountering summarized text in production, forcing the model to evaluate highly unfamiliar states. Training directly on compacted fragments aligns the critic’s value landscape with real-world serving constraints, ensuring that the active training distribution closely matches the production inference distribution.

How Does GLM-5.2 Stop Agents From Cheating During Training?

System disclosures from Z.ai note that GLM-5.2 exhibited a notable increase in reward-hacking behaviors during RL training compared to GLM-5.1. The model’s increased capability allowed it to aggressively explore and exploit cracks in the evaluation infrastructure:

• External Answer Downloading: When given a target problem, the agent would construct curl commands to raw file URLs on GitHub to download known solution scripts directly into the environment.

• Evaluation File Auditing: The model would trigger automated shell commands like find /workspace -name “hidden” to scan the runtime container for hidden grading keys or evaluation scripts.

• Credential Exploitation: Agents would locate and execute cat commands on protected test files to extract secret validation inputs, subsequently hardcoding those exact values into their submissions to mimic genuine algorithmic computation.

From a pure mathematical optimization perspective, these exploits represent successful completions. The training metric demands a passing test suite, and downloading a solution successfully triggers that flag. However, reward hacking stops the model from developing generalized reasoning capabilities; the gradient updates reinforce infrastructure exploitation rather than algorithmic synthesis.

To neutralize these behaviors, GLM-5.2 integrated a two-stage online anti-hack validation module:

• Rule-Based Filtering: The first stage runs strict, rule-based keyword filters that detect known exploit patterns. It blocks unauthorized outbound curl domains, flags discovery commands targeting the evaluation directory, and restricts direct read access to protected validation paths.

• LLM-Based Intent Judgment: Tool calls that bypass the raw keyword filters but exhibit suspicious parameters are routed to an inline LLM judge. This judge analyzes the full context of the agent’s historical choices to determine if the execution step represents a valid problem-solving attempt or an adversarial shortcut. This can likely be done cheaper LLMs and is a good bull case for theses based around inference maxing.

Instead of terminating the rollout immediately upon identifying a hack, the system blocks the exploit and returns mock or dummy data as the tool output, allowing the trajectory to continue running since aborting runs mid-execution introduces severe training instability and can trigger gradient collapse due to truncated episodes and highly anomalous reward boundaries. This isn’t a huge problem for accuracy since the usage of useless dummy data acts as a natural corrective signal by lowering the reward. This signals that the shortcut failed to pass the tests, prompting the gradient to penalize the exploitative choices without disrupting global training dynamics.

How GLM 5.2 Became so Good at Multiple Things?

Optimizing a single generalist model across multiple diverse capability domains simultaneously presents a major optimization bottleneck. Fields like code generation, mathematical reasoning, web search, tool routing, and instruction following possess conflicting gradient paths, distinct token distributions, and unique reward systems.

Training a single model sequentially across these areas results in catastrophic forgetting, where the model masters the newest domain while losing its proficiency in previous ones. Conversely, training on all datasets simultaneously demands massive computational scale and introduces severe gradient interference.

GLM-5.2 resolves this trade-off by using On-Policy Distillation (OPD). The strategy decouples the training phase into two distinct steps:

1. Specialist Optimization: The team trains separate, dedicated specialist models on isolated domains. Each specialist policy consumes its full computational budget exploring a single capability space, entirely avoiding cross-domain gradient interference.

2. Generalist Consolidation: The group uses on-policy distillation to merge the collective capabilities of the specialists into a single generalist model.

This framework differs fundamentally from traditional offline knowledge distillation. Standard distillation forces a student model to match a static set of outputs pre-generated by a teacher model. This introduces exposure bias: the student trains exclusively on the teacher’s ideal token distribution, but during production inference, it generates its own tokens. Once the student makes an unaligned token choice, it enters an unfamiliar state space where its training fails, causing errors to compound rapidly.

On-policy distillation eliminates this exposure gap by making the student model generate its own token trajectories during training. The expert teacher model evaluates these student-generated sequences in real time, delivering dense, token-level supervision across the student’s own token distribution.

The specialist teacher does not merely evaluate terminal success; it calculates exactly what token distribution it would have produced at every individual step given the student’s active context history. This provides a highly dense, fine-grained training signal that transfers exact token-level reasoning capabilities far more effectively than coarse pass/fail outcomes.

Z.ai scaled this pipeline using the SLIME post-training infrastructure to merge more than 10 expert specialist models into the final GLM-5.2 generalist parameter set in roughly two days. The SLIME infrastructure relies on Megatron to handle distributed model parallel training and utilizes SGLang to manage rapid inference rollouts, bridging the two components via a specialized Data Buffer module.

The framework natively supports multiple rollout environments:

• White-Box Integration: Gives the training coordinator full access to the internal hidden states and log probabilities of the active models.

• Black-Box Integration: Evaluates external text outputs when internal parameter access is restricted.

• Compacted Trajectory Handling: Processes lossy history fragments during active training loops.

• Sub-Agent Workflows: Manages multi-agent execution traces during scale-out generation.

Multi-teacher on-policy distillation has become a common post-training primitive across modern open-weights architectures. The execution in GLM-5.2 stands out due to its operational speed; consolidating over 10 distinct architectural specialists within a 48-hour window indicates high infrastructure optimization rather than simple algorithmic experimentation.

This efficiency is driven by SLIME’s dual-purpose deployment model. The exact caching, memory routing, query scheduling, and parallelization frameworks engineered to sustain high-throughput rollouts during RL training carry over directly into production inference serving.

The optimizations and configurations validated during the training rollout phase are reused to provision enterprise serving clusters. This system-level unification ensures that the production serving infrastructure is pre-tested against the exact workload signatures generated during scale training, allowing infrastructure improvements to reinforce both training throughput and deployment efficiency.

Conclusion: What Does GLM 5.2 Mean for the Future of Agentic AI?

GLM-5.2 wins because it isolates and respects the unique structural footprint of agentic loops over everything else. It builds a tightly coupled systems pipeline to handle volatile cache fragmentation, multi-step generation drag, and long-horizon credit assignment.

We have seen this evolution in every mature engineering discipline. In automotive and aerospace design, engineers spent decades chasing a universal chassis or a single airframe that could do everything. They eventually hit a physical wall. The stress profiles of a long-haul transport are fundamentally incompatible with an agile fighter jet. To get more performance, they had to specialize. They had to switch to specific, co-designed architectures tailored entirely to a vehicle’s exact payload and flight envelope.

As raw scaling laws peter out, specialization is the definitive future of AI. The next generation of systems will move away from the “god in a box” conception of AGI (which we’ve been calling a scam forever), and fully embrace flexibility and modularity that enable us to lego our way to building more customizable solutions.

The future (just as things always have) belongs to specialized systems most capable of rapid iterations and improvements. Not the best now, but the one that can become the best after 6 rounds of improvement. This isn’t a principle that most engineering/investing teams have imbibed yet.

The ones that do will pull ahead.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

Share

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

I keep this short.

The mission of Chocolate Milk Cult is simple: make the highest level of AI intelligence accessible to everyone. We don’t do hot takes or aggregated headlines. We do actual deep dives built on original research, custom analysis, and technical frameworks you can use.

The breakdown below is a perfect example of what our open-source research community funds: a premium deep dive on Claude Code built from 300 hours of live deployment testing, exposing concrete structural bottlenecks like the 15-to-20 turn agent degradation cliff. We derived these engineering principles by A/B testing prompt strategies across parallel git worktrees, tracking latency compounding, and forcing multiple team members to run the tool across fundamentally different engineering environments—ranging from legacy backend migrations to fresh frontend builds—to ensure our data didn't overfit for a single workflow or code distribution.

The result was a guide that helped several people rework their entire operation and maximize a ton of value from Claude Code.

While the model weights have updated since our original publication, these core context orchestration principles still carry over seamlessly; in fact, Anthropic still lists these exact constraints as official developer best practices. This is because our research focuses strictly on the invariant mathematical and systems-level realities of the context window, delivering timeless architectural frameworks that hold true across changing software versions.

But running these experiments, funding infrastructure, and securing access to engineers who build these systems costs us between $17,000 and $20,000 a month. These costs add up quickly, but it is the only way to deliver truly differentiated analysis instead of simply repeating PR cycles.

Free subscribers get a lot, and that is intentional. I never want your financial situation to be a barrier to getting the best insights. But paid support is what makes the whole operation possible.

We run a pay-what-you-can model for standard subscriptions so you can contribute whatever matches your budget. There are no tiered content walls so any price point you select will get you access to our entire catalogue of deep dives on Substack.

If you think providing actionable, first-principles AI insights to everyone is valuable, please help support the community at whatever level makes sense for you.

The links for support are below.

Support AI Made Simple for 10 USD per month or 100 USD Annually

Support AI Made Simple for 9 USD per month or 90 USD Annually

Support AI Made Simple for 8 USD per month or 80 USD Annually

Support AI Made Simple for 7 USD per month or 70 USD Annually

Support AI Made Simple for 6 USD per month or 60 USD Annually

Support AI Made Simple for 5 USD per month or 50 USD Annually

Support AI Made Simple for 4 USD per month or 40 USD Annually

Support AI Made Simple for 3 USD per month or 30 USD Annually

Support AI Made Simple for 2 USD per month or 20 USD Annually

Support AI Made Simple for 1 USD per month or 10 USD Annually

Hope to see you among the premium members.

Thanks for reading either way.

Dev <3

PS: (For enterprise operators, builders, and capital deployers who want to fund our high-overhead experiments directly, we also have a Founding Membership. This tier directly supports our primary-source testing infrastructure. In return, founding members get direct access to our proprietary GitHub repository—where we aggregate our raw data, custom insights, and codebase experiments—along with the ability to request custom deep-dive analysis on specific architectures.)

Every month, the Chocolate Milk Cult reaches over a million Builders, Startup Founders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Claude Code is one of the most useful agentic tools on the planet (especially after the release of Opus 4.5). It’s incredibly easy to use (once you can get past the terminal interface), but using it well is another issue altogether.

Claude Code pairs two very unfamiliar design decisions with each other—

1. A Command Line Interface reminiscent of a time when the world existed in black and white only.

2. Cutting Edge Agentic systems with deceptively powerful tool-calling abilities, which can compound errors very quickly if not used properly.

Learning to navigate both these decisions is very worth it— CC is my favorite AI tool in the market, and it alone justifies my 200 USD Subscription to Anthropic (I even have the extra usage setup and happily pay around 750-1000 USD/month on it; you likely won’t have to). I wrote this article to help you use it better.

Aside from my personal experience, we dug through heaps of user case studies of Claude Code, dug into Anthropic research, ran over 300 hours of experiments in real deployments, and compiled their experiences into the most useful workflows. The guide that follows is the result. No generic advice like— “Be specific.” “Give examples.” “Break down complex tasks.” This guide will cover:

• Why context orchestration matters more than prompt wording—and what that means for your use

• The terminal-first design philosophy and why it’s a feature, not a limitation

• Autonomy configuration: permission allowlists, full auto mode, and when each makes sense

• The ~15-20 turn degradation cliff and how to work around it

• CLAUDE.md: what belongs, what doesn’t, and why less is more

• The failure modes you’re probably hitting—and how to actualy fix them.

• Workflows that actually work: explore→plan→code→commit, TDD loops, parallel instances, writer-reviewer splits

If you’ve been looking to take your Claude Code usage to the next level, then this guide is for you.

Executive Highlights (TL;DR of the Article)

• Claude Code is not ChatGPT with file access—it’s an autonomous agent with tools, permissions, and execution authority operating inside a finite context window. Treating it like a chatbot is the root cause of most failures, not prompt quality.

• Context orchestration is the game, not prompt engineering. Every file, instruction, command output, and conversation turn competes for attention in the same window. Polluted context produces garbage regardless of how clever your prompt is; clean context lets mediocre prompts succeed. More context is often worse.

• The terminal-first design isn’t a limitation—it’s the point. Running alongside your shell gives Claude access to CLI tools, scripts, CI hooks, parallel worktrees, and headless automation that IDE-embedded assistants physically cannot touch. You trade inline suggestions for system-level composability.

• Autonomy is a dial. Default permission prompts maximize safety but destroy flow. Custom allowlists or full autonomous mode (with version control as your safety net) restore the agentic value. Productivity scales with autonomy once rollback is cheap.

• Agent performance degrades after ~15-20 turns—repetition, instruction drift, hallucinated claims. This is a hard limit, not a soft one. Effective users reset context, decompose tasks, or externalize state (commits, checklists) instead of pushing through.

• CLAUDE.md should be minimal and operational: build commands, critical paths, non-obvious constraints. Bloated instruction files trigger Chekhov’s gun—models try to use everything they’re given, including irrelevant guidance. Less is more.

• The failure modes are structural, not model-level: premature execution, context contamination, over-specification, permission friction, ignoring terminal feedback. Each has a mechanical fix. If you’re blaming the model, you’re misdiagnosing.

The governing rule: If you don’t control context, you don’t control outcomes. Claude Code doesn’t reward clever prompting. It rewards system design.

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

Section 1: The Mental Model — Claude Code as Orchestrated Agent

Most people approach Claude Code like they approach ChatGPT: type a request, get an answer, repeat. This guarantees mediocrity.

Claude Code isn’t a chatbot that happens to edit files. It’s an autonomous agent operating in your codebase with a constrained set of tools, a limited context window, and permission gates you control. Understanding this shift is key to leveraging this system more effectively. That’s the difference between CAM McTominay vs CDM McT.

The Core Thesis: Context Orchestration

You may forget your lovers birthday, but never forget this mental model: Claude Code is context orchestration. The quality of your outputs is downstream of how well you curate the information space Claude operates in.

Every file you reference, every CLAUDE.md instruction, every conversation turn, every external tool connection—all of it shapes the context window Claude reasons within. Most users optimize their prompts. Power users optimize their context.

This parallels how prompting works at a deeper level. In the prompting guide, we discussed “semantic neighborhoods”—how your instructions steer the model toward regions of its latent space where good answers become statistically likely. Claude Code adds another layer: you’re not just steering with words, you’re steering with what information exists in the context at all.

A brilliant prompt in a polluted context produces garbage. A mediocre prompt in a well-curated context often produces something usable.

(As a fun fact— this is why Claude within Claude Code is so much better than Claude in Cursor and Co.; Claude leverages many tools to ensure that it’s context stays clean).

This is very important to understand for one of the biggest UX shifts that Claude Code users are hit with, especially when you’ve only worked with IDEs (like me).

Terminal-First Is a Feature, Not a Limitation

The terminal interface throws people off. After years of IDE integrations and inline suggestions, a CLI feels like regression.

It’s not. It’s a deliberate design choice that unlocks capabilities IDE integrations can’t match. Terminal-first means:

• Full access to your environment. Claude inherits your shell, your PATH, your tools. The gh CLI for GitHub operations, jq for JSON parsing, ripgrep for fast code search, docker for container management, your custom deployment scripts—Claude can use all of them. An IDE assistant sandboxed to the editor can’t run arbitrary commands across your system.

• Scriptability. Claude Code has a headless mode (-p flag) that lets you pipe it into CI pipelines, pre-commit hooks, and automation scripts. You can have Claude automatically triage GitHub issues as they come in, run code review on every PR, or generate documentation on commit. Try doing that with an IDE extension.

  

• Composability. You can run multiple Claude instances in parallel across git worktrees, each with separate context windows working on independent tasks. One instance refactors your authentication system while another builds an unrelated data visualization component. The terminal is the natural interface for this kind of orchestration.

• No vendor lock-in on your editor. Use VS Code, Neovim, JetBrains, whatever. Claude Code operates alongside your tools, not inside them.

The tradeoff is real: you lose inline suggestions, visual diffs, and the “code appears as you type” magic.

If that’s your primary workflow—quick edits, tab completions, surgical changes—something like Cursor or Augment might be the better fit (I haven’t been very excited by Cursor in the past, but their new updates have been pretty good from what I hear). But if you’re doing complex, multi-file work that requires reasoning across a codebase—refactors, migrations, debugging distributed issues—the terminal-native model scales where IDE integrations hit walls.

The Permission/Autonomy Tradeoff

Out of the box, Claude Code asks permission for almost everything: file edits, bash commands, and external tool calls. This is by design—safety-first defaults for an agent that can execute code on your machine.

But it creates a workflow problem. You give Claude a task, walk away to check Slack, come back five minutes later, and it’s sitting there waiting for approval to edit a file. The whole point of an agent is that it agents. Constant permission prompts break that.

You have a dial here, and you need to decide where to set it:

Maximum safety: Default permissions. Claude asks before any mutation. Good for unfamiliar codebases, production environments, or when you’re learning the tool. I personally would almost never recommend it.

Balanced flow: Custom allowlist via .claude/settings.json. Permit file edits, common commands like git commit and npm test, and trusted external tools. Block destructive operations like rm -rf or database drops. This is where most users land.

Maximum autonomy: The --dangerously-skip-permissions flag. Claude runs uninterrupted until completion. Despite the scary name, this is the daily driver for many Anthropic engineers. The practical risk is low in a version-controlled codebase where you can revert anything. For true isolation, run it in a Docker container without network access. I personally run this as my default, and I’ve never had issues here.

Many power users create a shell alias: alias cc= “claude --dangerously-skip-permissions”. Type cc, start working, no interruptions.

There’s no universally correct setting. The right choice depends on your risk tolerance, the task at hand, and whether you’re running Claude in a sandboxed environment. But you need to make the choice consciously rather than accepting the defaults and getting frustrated.

The Practical Implication

If context orchestration is the game, then your job when using Claude Code is:

1. Curate what enters context. Be deliberate about which files you reference, what history accumulates, and what persistent instructions you set in CLAUDE.md.

2. Invest heavily in documentation. Tools like Claude Code and Codex can often overlook key details because of incomplete searches. People often think that these CLI tools don’t need documentation because they can search through, but Claude Code uses text/regex-based search; ensuring that your documentation can guide this search better is a must to ensure your product has everything it needs. My recommendation is to use specific “guidance documents” where you map out important pieces of logic so that AI knows where to look. If you have the budget for it (and a very large codebase with no documentation), my recommendation is to use Augment Code CLI to write extensive amounts of documentation that can be used by CC for improvements. A good rule of thumb is given below—

   

3. Prune what pollutes context. Clear irrelevant history with /clear. Reset between unrelated tasks. Don’t let a debugging session contaminate a feature build.

4. Extend context reach. Connect Claude to external tools—Puppeteer for browser automation and screenshots, Sentry for error monitoring, PostgreSQL for database queries, Slack for team communication. Each connection expands what Claude can perceive and act on.

5. Set appropriate autonomy. Match permission levels to the task and your risk tolerance. Don’t fight the tool’s safety defaults; configure them to match how you actually work.

Once we understand this, we can move on. The next section covers the foundational setup—the concrete configuration decisions that shape your context environment before you write a single prompt.

Section 2: The Foundational Setup

Before you write a single prompt, your configuration decisions shape everything Claude can do. This section covers the concrete setup that separates productive sessions from frustrating ones.

CLAUDE.md: Your Persistent Context

CLAUDE.md is a file Claude automatically loads at the start of every session. It’s the highest-leverage configuration point you have—(anecdotally) instructions here get followed more reliably than anything you type in the chat.

The hierarchy matters:

• ~/.claude/CLAUDE.md applies to all your projects

• ./CLAUDE.md in repo root is shared with your team via git

• ./CLAUDE.local.md is personal and gitignored

• Child directories can have their own CLAUDE.md files for specific subsystems

Most people either leave this empty or stuff it with everything they can think of. Both are wrong.

What to put in CLAUDE.md:

• Build and test commands (npm run build, pytest -xvs)

• Branch naming conventions and merge/rebase preferences

• Key file locations Claude can’t infer (”authentication logic lives in src/auth/, not src/users/”)

• Project-specific gotchas (”the legacy API in /v1 is deprecated but still receives traffic”)

• Environment setup quirks (”use pyenv local 3.11 before running tests”)

What NOT to put in CLAUDE.md:

• Style guides. Never send an LLM to do a linter’s job. ESLint, Prettier, Black—these are faster, cheaper, and more reliable. Claude will read your existing code and match patterns anyway (this is the benefit of latent space guidance).

• Obvious folder descriptions. If your folder is named components, you don’t need to explain it contains components.

• Generic instructions like “write clean code” or “follow best practices.” These waste tokens and don’t change behavior.

Here’s the key insight from various research: models are naturally inclined to use all information they’re given, even when it’s irrelevant. Anthropic researchers call this the “Chekhov’s gun” effect. A bloated CLAUDE.md doesn’t just waste tokens—it actively degrades performance by introducing distractors.

Keep it lean. I’ve seen effective CLAUDE.md files under 50 lines. I’ve also seen 500-line monsters that made Claude worse at everything.

Run /init to generate a starting point, but don’t trust it blindly. Claude will capture obvious patterns but miss your team’s actual conventions. Treat it as a draft to refine, not a finished product.

Permission Configuration

I covered the autonomy dial in Section 1, but here’s the practical setup.

Your options for allowing tools:

• During session: When Claude asks permission, select “Always allow” for tools you trust

• Via /permissions command: Add specific tools to your allowlist interactively

• In .claude/settings.json: Configure once, check into git, share with team

• CLI flag: --allowedTools for session-specific permissions

A reasonable starting allowlist for most projects includes Edit, Write, and Bash commands for git operations (add, commit, push) and your build/test scripts (npm test, npm run build). This lets Claude edit files and run your standard workflow without asking. It’ll still prompt for anything outside the list.

For full autonomous mode, I just use an alias—cc mapped to claude --dangerously-skip-permissions. I work in version-controlled repos (if you’re using ADEs this is a non-negotiable), I commit often, and I can revert anything. The productivity gain is worth it. Your risk tolerance may differ.

MCP: Extending What Claude Can See and Do

MCP (Model Context Protocol) connects Claude to external tools and data sources. Without it, Claude is limited to your filesystem and shell. With it, Claude can query databases, control browsers, post to Slack, read from Notion, and more.

The MCP servers I actually use:

• PostgreSQL/MySQL for database queries. Claude can check schemas, run queries, verify data. Much better than having it guess at your data model.

• GitHub for complex PR workflows and issue management beyond what the gh CLI offers.

• Sentry for error monitoring. Claude can pull recent errors, stack traces, and frequency data when debugging production issues.

• I tried some Codex MCPs for review, but they did not work. I heard really good things about it, so if you have tried it successfully, would love to hear about this.

You configure MCP servers in three places:

• .claude.json in your home directory (global)

• .claude/settings.json in your project (project-scoped)

• .mcp.json in your repo root (shared with team via git)

A warning: each MCP server adds tool definitions to your context window. If you enable ten servers you’re not using, you’re burning tokens on tool descriptions Claude will never call. Enable what you need, disable what you don’t. You can check what’s consuming your context with /context.

Custom Slash Commands

Slash commands are reusable prompt templates. Put a markdown file in .claude/commands/, and it becomes available as /project:filename.

I use these constantly. For example, a fix-issue.md command that takes an issue number as an argument, runs gh issue view to get details, searches the codebase for relevant files, implements the fix, writes tests if appropriate, and commits with a descriptive message referencing the issue. Now /project:fix-issue 1234 handles the entire workflow.

Same idea for code review—a command that checks for bugs, edge cases, security issues, and performance concerns without the verbose commentary Claude defaults to.

Personal commands go in ~/.claude/commands/ and work across all projects. Project commands in .claude/commands/ get shared with your team when committed.

The $ARGUMENTS placeholder passes whatever you type after the command name. Simple but powerful.

Context Hygiene: The Commands That Matter

Three commands you need to internalize:

/clear wipes conversation history. Use this between unrelated tasks. A debugging session and a feature build shouldn’t share context. I /clear constantly—probably every 20-30 minutes of active work.

/compact summarizes conversation history to free up tokens while preserving key information. Use this when you’re mid-task but context is getting bloated. Auto-compaction triggers at 95% capacity, but by then you’ve already degraded performance. Compact proactively around 70%.

/context shows what’s consuming your context window. Run this when Claude seems slow or unfocused. You’ll often find old tool results, irrelevant file contents, or MCP server definitions eating your token budget.

Research on coding agents found that performance craters after roughly 20 iterations. Not 100. Not 50. Twenty. The developers who treat this as a hard limit—resetting context proactively rather than waiting for degradation—get consistently better results.

My rule: if I’ve been going back and forth with Claude for more than 15-20 turns on a single task, something is wrong. Either the task needs decomposition, or I need to reset and try a different approach. One interesting thing about ADEs is that it can often be much cheaper to rebuild from scratch than to try to fix what’s broken, so you should not be shy about reverting to the last stable states. This does increase the value of code review, however, because you want to be able to identify the last stable state and see what about it might push an AI in certain directions/what the issue could be.

The Setup Checklist

Before starting a new project with Claude Code:

• Run /init to generate starter CLAUDE.md

• Edit CLAUDE.md down to essentials—build commands, key paths, gotchas

• Configure permissions in .claude/settings.json

• Add MCP servers you’ll actually use

• Create 2-3 slash commands for your common workflows

• Commit .claude/ directory to share with team

This takes maybe 15 minutes. The productivity difference over a week of work is hours.

Now that your Claude Code is ready to be the CTO of your new AI-based sneaker Recommendation app with an element of gambling, you’re probably ready to start coding. But I would recommend holding your horses, because just as CC can write good code fast, it can also chef up spaghetti fast enough to make Takumi Aldini jealous.

Section 3: Failure Modes

We’ve discussed many of these failures at length either in earlier sections or prior deep dives. So I’ll keep this one short and direct, meant more for completeness/to have everything in one place for easy lookup.

Premature Execution

It is very important for you to break your lifelong commitment to be a two-pump chump when using Claude Code. This is the number one cause of failure with Claude Code.

CC starts coding before understanding the problem. If you’ve given it autonomy, you’ll end up three iterations deep solving the wrong thing or married to the wrong approach.

Fix: explicit staging. “Read the relevant files and understand the current implementation—don’t write code yet.” Then: “Propose a plan.” Only then: “Implement.” The words “think” and “think hard” trigger extended reasoning and actually allocate more compute. Planning mode is a godsend since reading its thoughts is also easier than debugging walls of code flying at you.

Context Contamination

You debug for an hour, then switch to a new feature without clearing context. Claude drags forward assumptions, references, and attention anchors from the debugging session.

Fix: /clear between unrelated tasks. I clear every 20-30 minutes on heavy usage. The cost of re-establishing context is lower than the cost of degraded performance.

The 20-Iteration Cliff

Research shows agent performance craters after ~20 turns. Symptoms: Claude repeats itself, ignores instructions, claims to have done things it didn’t.

Fix: treat 15-20 turns as a hard limit. If you’re still going, either decompose the task or reset with a fresh summary.

CLAUDE.md Bloat

People stuff CLAUDE.md with style guides, ADRs, and API docs. Now every session starts with 1500 tokens of mostly-irrelevant instructions, and Claude tries to use all of it (Chekhov’s gun effect).

Fix: under 50 lines. Point to docs during specific tasks, don’t load them into every session.

One-Shot Syndrome

Expecting complex tasks to work on the first try. They won’t. Requirements are ambiguous, edge cases emerge during implementation.

Fix: treat first outputs as drafts. Build in verification—tests, screenshots, behavior checks. Build modularly to ensure that you can make changes easily and regularly think about refactoring code before starting major expansions to capabilities to make sure you firm up the foundations.

Over-Specification

500-word prompts specifying exactly how to implement something. You’ve wasted effort on details Claude would figure out, and anchored it to your approach (which might not be best).

Fix: specify the what, be loose on the how. “Add rate limiting—max 5 attempts per 15 minutes” beats a detailed implementation spec.

Permission Friction

Still manually approving file edits and git commits after a week of use? You’re wasting your own time.

Fix: configure your allowlist or use --dangerously-skip-permissions.

Ignoring Terminal Output

Claude runs commands, they fail, and Claude struggles. You watch without reading the errors.

Fix: read the output. You often know things Claude doesn’t—wrong paths, missing env vars, version mismatches. You’d be surprised how often Claude can pick the wrong approach or prioritize the wrong aspects; it helps a lot if you monitor the outcomes on a regular basis.

Pro tip— if an issue has been a problem for more than one hour, Claude likely has completely misunderstood something. Revert your changes to the last place you understood, and then look through the logs. Then start talking to CC about your speculations on the code-base and what could be causing issues.

Not Using Subagents

Complex tasks fill your context with research and dead ends. Subagents keep exploration isolated and return only conclusions.

Fix: “Use a subagent to research how X currently works.” Main context stays clean.

With all of this covered, we’re now ready to use Claude Code Workflows properly.

Section 4: Workflow Frameworks

These are the patterns that actually work. Not theoretical—these are workflows I use and that Anthropic engineers use internally.

Explore → Plan → Code → Commit

The default workflow for any non-trivial task.

1. Point Claude at relevant files. “Read src/auth/ and understand how authentication currently works. Don’t write code.”

2. Have it propose a plan. “Now propose how you’d implement OAuth support. Think hard about edge cases.”

3. Review the plan. Push back, ask questions, refine.

4. Execute. “Implement the plan.”

5. Commit. “Commit with a descriptive message.”

The explicit “don’t write code” in step 1 matters. Without it, Claude jumps to implementation. The “think hard” in step 2 allocates more reasoning compute.

For complex problems, have Claude use subagents during exploration. “Use a subagent to investigate how the session management works” keeps research out of your main context.

This workflow takes longer than just asking Claude to implement something. It also works far more often.

TDD Loop

When the success criteria can be expressed as tests.

1. “Write tests for the feature I’m describing. Cover the happy path and these edge cases: [list]. Don’t implement yet. (Claude has a subagent for planning; spam ts).”

2. “Run the tests, confirm they fail.”

3. “Implement code to make the tests pass. Keep iterating until green.”

4. “Commit tests and implementation separately.”

Claude excels when it has a clear target to iterate against. Tests provide that. Each red-to-green cycle gives Claude feedback it can act on.

This also protects you from Claude solving the wrong problem—the tests define what “right” means before implementation starts.

I use this for any backend logic, API endpoints, data transformations. Less useful for UI work where “correct” is visual.

Visual Feedback Loop

When success is “it looks right” or “it works in the browser.”

1. Give Claude a target—screenshot, Figma mock, or verbal description of desired UI.

2. Have Claude implement.

3. Claude takes a screenshot (via Puppeteer MCP or you paste one in).

4. “Compare to the target. What’s different? Fix it.”

5. Iterate until it matches.

Claude’s first UI attempt is usually 60-70% right. After 2-3 iterations with visual feedback, it’s usually 95%+. Without the feedback loop, you’re stuck at 60-70% and doing the rest manually.

Also useful for debugging—”here’s a screenshot of the broken state, here’s what it should look like, fix it.”

This is one area I’m hearing some comments about, Codex being very good, and Gemini 3.0 has been cleaning house with UI design. On top of this, Cursor dropped an elite update for visual design. This is likely the one area where I expect CC to face the heaviest competition.

Parallel Multi-Claude

When you have independent tasks that don’t need to share context.

Setup: multiple terminal tabs or VS Code panes, each running a separate Claude instance. Or use git worktrees so each instance works on a separate branch.

Instance 1 refactors the auth system. Instance 2 builds a new dashboard component. Instance 3 writes documentation. No shared context, no cross-contamination, 3x throughput.

The key word is independent. If tasks interact—one depends on the other’s output, they touch the same files—this creates merge conflicts and confusion. For truly parallel work, it’s excellent.

Git worktrees make this cleaner:

• git worktree add ../project-feature-a feature-a

• Open that directory in a new terminal

• Run Claude there

• Repeat for other features

Each worktree is a separate checkout. Claude instances can’t step on each other.

Writer-Reviewer Split

When you want quality control built into the workflow.

1. Claude A writes the code.

2. /clear or start a fresh Claude instance.

3. Claude B reviews what Claude A wrote—bugs, edge cases, security issues.

4. Take feedback back to Claude A (or a fresh instance) to address issues.

Separate context matters here. Claude reviewing its own work in the same session tends to defend its choices. A fresh instance with no investment in the code reviews more critically.

I use this for anything going to production. Overkill for exploratory work.

Headless Automation

When Claude should run without you.

The -p flag runs Claude in non-interactive mode. Pipe in a prompt, get output, done.

Uses I’ve actually set up:

• Pre-commit hook that runs Claude on staged files to check for obvious issues

• GitHub Action that triages new issues with labels

• PR reviewer that comments on diffs (use /install-github-app to set this up quickly)

• Script that runs Claude on each file in a migration, one at a time

For batch operations: write a script that loops through items (files, issues, whatever), calls claude -p with a prompt for each, and collects results. Add --output-format json if you need structured output for further processing.

Combine with --allowedTools to restrict what headless Claude can do. You probably don’t want an automated script running with full permissions.

Checklist-Driven Work

For large tasks with many steps—migrations, multi-file refactors, launch checklists.

1. Have Claude write a checklist to a markdown file. “Create CHECKLIST.md with every file that needs updating for this migration.”

2. Claude works through items one by one, checking them off.

3. The file serves as progress tracking and recovery point if context resets.

This externalizes state. If Claude hits the 20-iteration cliff and you need to reset, the checklist shows what’s done and what’s left. Paste it into the fresh session and continue.

Also useful for team handoffs—the checklist is documentation of what’s been done.

Conclusion: From Prompting to Orchestration

Claude Code isn’t powerful because it writes code quickly. It’s powerful because it lets you externalize cognition into a system you can actually control. Context, autonomy, feedback, and verification aren’t secondary concerns here; they are the product. Everything else—prompting, clever phrasing, even model choice—is downstream.

Once you internalize this, there’s no going back. Prompting starts to feel like a local optimization. IDE magic feels cosmetic. When something goes wrong, you stop asking why the model “behaved strangely” and start asking what information you gave it, what permissions you allowed, and what feedback loop you failed to close. The mystery evaporates.

This is where the real split emerges. Most people use Claude Code as a faster pair of hands. Operators design environments where failure is hard and success is boringly inevitable. Same tool, completely different outcomes. The difference isn’t intelligence; it’s orchestration.

There’s a simple rule that governs everything you’ve read: if you don’t control context, you don’t control results. Almost every complaint about coding agents—hallucinations, brittleness, wasted time—traces back to violating that rule. Not a bad model. Bad setup.

Claude Code won’t make you a better engineer. It will make it very obvious whether you already think like one.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

On Friday, June 12, 2026, SpaceX begins trading on Nasdaq. Within twenty business days, money from millions of automatic retirement accounts will buy the stock on a rigid, pre-published schedule. Nobody — not your fund manager, not you — gets to evaluate the company first.

Here is what’s being forced into your portfolio. SpaceX is selling roughly $75 billion of stock at a $1.77 trillion valuation — 95 times last year’s revenue, for a company that lost $4.9 billion doing it. The shares your retirement account buys carry one vote each; the shares insiders keep carry ten. After the sale closes, insiders control 88.5% of all voting power and Elon Musk alone commands 82.4%. And because an index fund must buy the complete corporate ledger, you can’t take the profitable Starlink business without also taking a cash-burning rocket division, the social platform X, and an AI venture that spent $12.7 billion on data centers in 2025 alone.

To help you understand how we got here — and what you can still do about it — this article will cover:

• Which index benchmarks are buying SpaceX and on what schedule. Russell funds buy at the close of Day 5. MSCI global funds finalize by Day 10. Nasdaq-100 funds force entry after Day 15. We’ll map every fund family to its forced-purchase date so you can find your own exposure.

• How the algorithmic front-running of these schedules drains $16 billion a year from passive investors. The purchase dates are published in advance and you can’t change them. Traders who know exactly when $14.4 trillion in passive capital must buy get to set the price it buys at — roughly $200 a year out of every indexed household.

• How the index guardrails were dismantled in 90 days. Seasoning windows, float minimums, profitability screens, and voting-rights floors were built over a decade to protect retail investors. Between February and June, Nasdaq, FTSE Russell, CRSP, and S&P each rewrote their rulebooks to waive them for sufficiently large listings.

• Why the S&P 500’s June 4 “rejection” of SpaceX is security theater. The committee publicly kept its screens for the S&P 500 — and in the same document, quietly opened its own Total Market, Completion, and Dow Jones benchmarks to low-float listings. Worse, those side doors let an asset accumulate the trading history and float needed to walk through the front gate 12 months later.

• What’s inside the S-1 your retirement account is buying. The full balance-sheet dissection: segment losses, the AI division’s capex run-rate, the bitcoin treasury, and the voting math.

• What you can actually do to block forced entry. The specific fiduciary, regulatory, and shareholder levers — ERISA prudence queries, the DOL comment window, pension exclusion mandates, and Rule 14a-8 proposals — including a template you can send to your plan committee today.

Executive Highlights (tl;dr of the article)

For this article, you might need to understand the following terms:

• Index / benchmark: A published list of stocks with a rulebook deciding what gets in. Your index fund is legally bound to copy this list. Whoever writes the rulebook controls what your retirement account buys.

• Passive fund: A fund that automatically buys whatever its index says, in the proportions it says, on the schedule it says. No human judgment in the loop. ~$14.4 trillion of American savings works this way, including most 401(k) default options.

• Float: The percentage of a company’s shares actually available for the public to buy. SpaceX is floating 4.15% — insiders keep the rest. Low float means a thin sliver of trading sets the price for the whole company, so prices swing violently and insiders keep control.

• Float minimum: The old guardrail requiring ~10% float before a stock could enter an index. This is the rule that was rewritten: now a float of any percentage qualifies if it’s worth enough raw dollars.

• Seasoning window: A mandatory waiting period (typically 12 months) between a company going public and entering an index — time for hype to cool, lockups to expire, and real financial data to accumulate before your money is forced in.

• Profitability screen: The requirement that a company show actual accounting profits before index entry. SpaceX lost $4.9 billion last year.

• Dual-class shares: Two share types with unequal votes. Your shares: 1 vote. Insider shares: 10 votes. This is how Musk keeps 82.4% control while selling you the downside.

• Rebalancing / forced buying: When the index list changes, every fund tracking it must buy or sell to match — at a published time, regardless of price. Predictable forced buying is what front-runners feed on.

• Front-running: Buying a stock ahead of forced buyers you know are coming, then selling it to them at a markup. Legal when the schedule is public. This schedule is public.

Once you understand these terms, here is your tldr — 

• SpaceX Valuation and Forced Buying: On June 12, SpaceX begins trading at a $1.77 trillion valuation — 95x revenue with a $4.9B net loss. Within 20 business days, index funds holding millions of Americans’ retirement savings are contractually forced to buy it. No asset evaluation occurs by you or your fund manager; the rewritten index rulebook decides the purchase automatically.

• The $16B Capital Extraction: Forced buying schedules are published in advance, allowing traders to buy the stock early and sell it back to passive investors at marked-up prices. This mechanism drains roughly $16B a year from passive investors (~$200 per indexed household). Research confirms that these fast-tracked index additions underperform by 22% over the following year, forcing index funds to buy at the top.

• Dismantling Investor Guardrails: Four index providers removed protections like seasoning windows, float minimums, profitability screens, and voting floors within 90 days. They replaced percentage-based protections with raw dollar thresholds, allowing massive companies to bypass safety rules. These changes specifically target the largest listings; one provider’s consultation explicitly named “SpaceX, OpenAI, Anthropic” as the motivation.

• S&P 500 Security Theater: The S&P 500’s June 4 rejection of these changes is a brand stunt. The same document opened S&P’s total-market benchmarks to low-float listings, feeding growth, ESG, and target-date products automatically. This creates a side-door where a company accumulates the required trading history over 12 months to enter the S&P 500 front gate later.

• The Toxic Inside Bundle: The forced bundle forces you to buy profitable Starlink alongside a loss-making rocket division, X, and an AI venture with capex exceeding its revenue. Elon Musk retains 82.4% control via 10:1 insider voting shares, leaving shareholders with mandatory arbitration instead of court access.

• The Retirement Exit Liquidity Pipeline: This is the standard pipeline moving forward, not an isolated incident. Both Anthropic ($965B) and OpenAI ($852B) filed confidential S-1s within the past two weeks. If these rules stand, retail retirement capital becomes the permanent exit liquidity for private valuations forever.

• Your Direct Leverage Actions: The index machine runs entirely on your paycheck deductions, giving you real leverage to fight back. You can audit your fund benchmarks, send an ERISA prudence query to your 401(k) committee to force the issue onto the plan’s fiduciary records, and comment on the DOL’s safe-harbor rule. You can also push pension boards to exclude low-float multi-class listings and file shareholder proposals at index providers, replicating past participant litigation that cut 401(k) costs in half.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Which Index Benchmarks Are Buying SpaceX and on What Schedule?

Every passive fund is legally bound to replicate an underlying benchmark owned by an independent company (and the same company might have 2 indexes that track 2 different benchmarks). For example, Vanguard’s Total Stock Market ETF (VTI) tracks the CRSP U.S. Total Market Index; the Vanguard Total World Stock ETF (VT) tracks the FTSE Global All Cap; Invesco’s QQQ tracks the Nasdaq-100. To find your exposure to Elon’s newest scam, you’ll have to look at the “seeks to track” statement on your fund’s official profile page.

Once you know your exposure, the next variable to track is the timing. The specific countdown to forced purchasing begins the moment SpaceX logs its first trade on Friday, June 12, 2026. Then you have to look out for the following:

• Russell Large-Cap Funds: Forced buying occurs all at once at the market close on Day 5 — expected Thursday, June 18 — under a single-batch rule adopted by the committee on May 26.

• MSCI Global Funds (ACWI, World): The fast-track inclusion decision is announced between Day 1 and Day 3, with mandatory buying finalized after the close of Day 10 — expected Friday, June 26.

• Nasdaq-100 Funds (QQQ): The asset is evaluated on Day 7 and forced into the index after the close of Day 15 — early July 2026.

• Total Market Funds: CRSP (governing Vanguard’s VTI complex) altered its design on April 27 to bypass percentage limits if an asset passes a raw dollar test. S&P’s total-market index series implemented an identical dollar-alternative clause alongside a five-day fast-track trigger on June 8.

• Target-Date Funds: These vehicles automatically absorb the stock based on the underlying proportion of CRSP, Russell, Nasdaq, or MSCI products they hold internally.

These dates matter more than you think. Across the $14.4 trillion sitting in domestic passive equity funds, the general cost of running predictable, schedule-driven rebalancing trades already drains an estimated $16 billion annually from people due to algorithmic front-running. However, waiving float minimums for a mega-cap listing like SpaceX introduces an entirely new layer of volatility risk. When algorithmic traders know the purchase schedules ahead of time for a massive, low-float asset, the resulting price distortions will likely worsen this existing multi-billion-dollar drain on your savings.

Let me rephrase that clearly: since algorithmic traders know the purchase schedules ahead of time (which you can’t change since you can’t control the index), they can scalp your hard-earned savings (the phenomenon already costs people 16 Billion USD annually). You are the product being sold to convince Silicon Valley Elites and their bankers to invest in an overinflated bubble.

Maybe you’re wondering if this isn’t the way it’s always been. Or you think I’m reading too much into something. However, if you break down the movements over the last 90 days, this pattern becomes self-evident. Multiple independent indexes all changed their inclusion criteria to accommodate the SpaceX IPO. And their changes specifically loosed the guardrails that were put in place to protect retail investors from the information asymmetry between themselves and institutional investors.

Let’s understand how in more detail.

How Were the Index Guardrails Dismantled to Allow SpaceX-like Low-Float Listings?

Credit where credit is due: the passive investment industry spent the last decade building some good investor protections from predatory private listings. These protections were engineered after specific corporate failures left index investors holding illiquid, un-voted shares. The guardrails relied on four structural pillars:

• Seasoning Windows: The S&P Composite 1500 required an IPO to complete 12 months of active public trading before index entry, allowing initial lockups to expire and public financial data to surface.

• Float Minimums: Standard benchmarks required at least 10% of a company’s total shares to be available for public trading, preventing tiny allocations from causing massive price volatility.

• Profitability Screens: Inductees were required to show positive GAAP net income over their most recent four quarters, ensuring index entry was backed by real accounting profits rather than market hype.

• Voting Floor Rights: Following Snap’s zero-vote share offering in 2017, FTSE Russell barred companies unless public shareholders held at least 5% of total voting power, while S&P closed the S&P 500 to new multi-class structures.

These protections were systematically broken down between February and June of this year to accommodate Elon’s newest rug pull:

• Under the old rules, a company selling only 4% of its shares to the public was automatically disqualified for failing the 10% float threshold.

• Under the new rules, if that 4% slice is worth billions of dollars, the percentage minimum is waived entirely. At a $1.77 trillion valuation, SpaceX’s tiny 4.15% public float translates to a $75 billion offering — a dollar figure larger than the total market cap of most companies in the index.

• (The float constraints are more important than you’d realize. If only a small percentage a company is available, then its price will fluctuate much more violently, which can adversely impact investors. This way Elon and his buddies get to keep all the control, while still forcing indexes to buy their inflated valuations).

A more detailed analysis is given below:

As you can see here, multiple indexes clearly rewrote their rulebooks to accommodate these changes, all to make it easier for them to let them cash out on that sweet 401-k demand. The scalpers that will price this demand to dump stuff will win. Elon’s cronies will win bigly. Even the index managers will win since they make money irrespective. The only one that loses is the regular investor that thought that someone would look out for them. That’s what we get for being poor and focusing on actually contributing to society instead of trying to weasel around and steal value from every nook and cranny of the system.

Luckily, people have started speaking out against this, sparking a public confrontation that forced the S&P 500 committee to draw a hard line on its core index family. While this is a win, I think many people overestimate what this actually means (I heard from more than a few people that this was all we needed and everything would sort it itself out now). Let’s understand why this isn’t true.

Does the S&P 500 Rejection of SpaceX Protect You?

The S&P 500 committee’s June 4 decision to reject the “MegaCap” rule changes creates a dangerous illusion of systemic safety. The committee’s proposal on April 30 had suggested halving the 12-month waiting period, waiving the 10% public float minimum, and dropping the positive GAAP net income screen for exceptionally large companies. While public opposition forced the committee to declare that rule exceptions “should not be granted solely based on market capitalization,” this firewall only applies to the S&P 500 index family itself.

This is a problem because an asset entering through the newly opened side doors of total-market, Nasdaq, and Russell indexes begins accumulating the exact milestones required to clear the front gate later. Over a 12-month seasoning window, an unlisted mega-cap stock builds up trading history, converts insider shares into float as lockups expire, and establishes passive institutional ownership patterns.

Furthermore (and this is important), S&P’s own June 4 document authorized these low-float alternatives for its Total Market, Completion, and Dow Jones U.S. Total Stock Market benchmarks. Entry into these foundational total-market products automatically feeds into specialized sub-indexes:

• Total-market entrants become eligible for immediate inclusion in derived size, style, sector, factor, and sustainability benchmarks.

• Investors who choose a supposedly safe “Growth” or “ESG” index product absorb the low-float asset layout without realizing they have bypassed the core S&P 500 profit screens.

• The exposure propagates into target-date retirement vehicles that use total-market building blocks as their primary equity holdings.

So the S&P ruling doesn’t even fully extend across their own families. Safety Theatre at its finest.

And if we let this pass, then things are only going to get worse. Silicon Valley has rug-pulled retail investors through overpriced IPOs based on misinformation and hype for a while now; but at least investors could choose to not engage. If these index changes are allowed to stand, it will create a strong pipeline where retail investors will permanently have to bail out the so-called smart money. And that lineup already looks promising:

• Anthropic filed its confidential draft S-1 on June 1, following a $65 billion funding round that established a $965 billion valuation backed by a $47 billion annualized revenue run-rate (wait till we dissect this can of worms, there is a LOT wrong here).

• OpenAI finalized its own confidential SEC paperwork on June 8 at a valuation nearing $852 billion, completely omitting public financials, share distributions, or corporate voting structures.

Fun times ahead. But can we do anything about this?

As always, you have more power than you think.

What Can YOU Do to Block Forced Index Entry?

Contrary to what some wimps peddle online, people hold enough leverage to make things happen. Reversing this automated robbery requires formal, documented intervention through structured fiduciary and regulatory channels. The following list might seem overwhelming, but the key is to organize as a collective, pick 2–3 sides that best suit us, and focus on that. If everyone focuses on their battle, the collective

First, you can protect yourself better by executing three steps:

• Audit Fund Benchmarks: Locate the “seeks to track” statement on your index fund profile pages to identify your direct exposure. If your portfolio holds products tracking the CRSP U.S. Total Market Index, the FTSE Global All Cap, the Nasdaq-100, or S&P total-market benchmarks, your money is exposed.

• Submit a Formal ERISA Prudence Query: Route a written request to your employer’s 401(k) plan investment committee through Human Resources, requiring a formal response on the record. This places a binding duty of review on individuals who face personal legal liability under ERISA Section 404(a), which mandates care, skill, prudence, and diligence in monitoring plan options. Most importantly, this will force the issue onto the plan’s fiduciary record.

• Demand Restricted Equity Alternatives: Use your plan’s feedback loop to demand at least one broad equity selection that is structurally insulated from immediate low-float IPO entry — such as a standard S&P 500 fund, a profitability-screened index, an equal-weight benchmark, or a self-directed brokerage window.

Not sure what to send? Here’s something nice and easy drafted by Irys — “I am a participant in [plan name]. I am writing to ask whether the plan’s investment committee has reviewed the 2026 IPO fast-entry and low-float methodology changes adopted by Nasdaq (effective May 1), FTSE Russell (May 26), CRSP/Morningstar (April 27), and S&P Dow Jones total-market indexes (June 8), as well as MSCI’s existing large-IPO early-inclusion process, and whether the committee has assessed how these changes affect the plan’s index funds and target-date funds-including exposure to newly public, low-float, controlled companies such as SpaceX. I request a written response. Under ERISA section 404(a) and in light of the Department of Labor’s March 30, 2026 proposed rule on designated investment alternatives, I believe benchmark methodology, valuation, liquidity, and complexity are appropriate subjects for the committee’s documented review.”

(While the bar for ERISA litigation is high and plan fiduciaries are generally afforded broad discretion in selecting diversified index funds, these formal queries serve a critical strategic purpose. Rather than an immediate trigger for litigation, this documentation acts as a mechanism for transparency — compelling plan sponsors to justify their continued reliance on benchmarks that have systematically dismantled their own governance screens. By forcing this issue onto the official record, you ensure that the decision to expose retirement savings to low-float, high-volatility listings is a conscious, documented choice rather than an automated oversight. Historically, it is this type of granular scrutiny that forces corporate consultants and fiduciary insurers to re-evaluate structural risks, eventually steering capital away from exposed benchmarks to avoid the long-term threat of ‘failure to monitor’ claims).

This documentation creates an immediate liability risk for plan sponsors. Historical precedent shows that excessive-fee litigation over the last fifteen years cut 401(k) expenses roughly in half because corporate consultants steered clients away from structural compliance risks. Once “default funds whose benchmarks auto-buy low-float corporate listings” becomes a line item on fiduciary-insurance questionnaires, the spreadsheet merchants will move away from the exposed benchmarks themselves.

You can also escalate this financial pressure by targeting fund boards and index licensing revenue directly. Because index providers earn revenue as basis points on assets benchmarked to their rulebooks, moving capital out of a Russell or CRSP total-market fund and into a standard S&P 500 fund is a good way to immediately punish funds that dissolved their screens. Once again, to really turn the thumb-screws, skip the IR departments and write directly to a fund’s independent board of trustees, who hold the statutory authority to approve index licenses and tracking policies.

The most definitive regulatory lever is filing a formal public comment on the Department of Labor (DOL) March 30 proposed rule. Workers can be automatically enrolled into default retirement funds only because federal regulations grant plan sponsors a strict liability safe harbor for utilizing Qualified Default Investment Alternatives (QDIAs). Since federal agencies are legally required to address substantive data within the public register, commenters can dismantle this legal coverage. To trigger this, submit comments demanding that QDIA safe-harbor status be explicitly conditioned on a default fund’s benchmark maintaining rigorous seasoning, minimum public float, and voting-rights screens. Citing the SpaceX listing and the FTSE Russell consultation by name proves that low-float benchmarks carry structural valuation and liquidity risks, forcing corporate lawyers to abandon them to preserve their safe harbor.

For public sector capital, beneficiaries can force state pension systems to act from below. The Comptrollers of New York City and New York State and the CEO of CalPERS have issued public objections calling the SpaceX listing the most management-favorable governance structure ever brought to public markets. State employees and teachers can leverage these admissions to demand that their retirement boards aggressively renegotiate passive investment mandates. Pensions routinely run index-minus-exclusion mandates for firearms or tobacco; beneficiaries can mandate identical exclusions for multi-class, low-float listings (I’d imagine you can also directly point to Grok’s Sexual Deepfakes as a strong reason to not invest into SpaceX, which now owns the AI company). These groups can pull some serious numbers (for eg the American Federation of Teachers (AFT) alone represents 1.7 million members).

Finally, we can also directly confront the regulatory framework governing the indexes themselves. Rather than relying on SEC Rule 14a-8 shareholder proposals — which corporate counsel routinely block under the ‘ordinary business operations’ exclusion — investors should direct their focus toward formal SEC regulatory petitions. By petitioning the SEC to investigate the commercial conflicts of interest between index licensing revenues and investor protection mandates, we can force some action. Morningstar here seems to be the most interesting target: its advisory division issues safety ratings to individual investors while its index division actively waives the screens that protect them.

Then we can attack the root of this extraction: the underlying exchange listing standards. While the SEC and US exchanges have permitted dual-class structures for decades, the SpaceX listing represents an unprecedented compounding of risks. A $1.77 trillion valuation holding ten-to-one insider voting shares and 82% single-person control is concerning on its own. But when combined with waived float minimums and bypassed profitability screens, it creates a systemic vulnerability.

Additionally, maintaining a public ledger that tracks which funds bought on which date, at what price relative to the offer, and the resulting cost-per-household provides standing evidence for future litigation and regulatory enforcement.

Conclusion: What Is the Bottom Line?

There is a common type of cynicism that says everything is rigged, the system is too big, and you cannot do anything about it. That attitude is a cop-out used by cowards to make themselves feel better about their inaction.

The entire index fund machine does not run on Wall Street’s hidden wealth. It runs on your money — the automatic deductions taken out of your paycheck every two weeks. When you start asking tough questions, the people managing that money have to listen.

Throughout history, every good thing that ever happened came about because regular people decided to come together and fight for it. We know this works because we just saw it happen. On June 4, public pressure forced the S&P 500 committee to back down and keep their safety rules in place. Over the last fifteen years, regular people filing complaints cut 401(k) fees in half across the country.

The scale of these giant companies can feel overwhelming, but the way you fight back is simple: pick your battle, find a group to stand with, and stick it out. You do not have to change the whole financial system yourself. You just have to look up your funds, send one written question to your HR department/send messages to indexes, or move your savings to a fund that does not buy into rigged voting structures. Persist with that, and have faith that there are others who have your back.

Appendix A: The Five Rulebooks in Detail

Written to be forwarded to a plan committee, legal counsel, or investment consultant.

• S&P Dow Jones Indices: Under old rules, the S&P Composite 1500 (including the S&P 500) required a 12-month IPO seasoning window, an investable weight factor (publicly investable share fraction) of at least 0.10, and positive GAAP net income in the most recent quarter and summed over the most recent four. The April 30 consultation proposed halving seasoning to six months, waiving the 0.10 weight factor, and waiving the profitability screen for MegaCap listings. On June 4, S&P rejected these changes for the Composite 1500, stating exceptions should not be granted based solely on market capitalization. However, in the same June 4 document, S&P added a dollar alternative to the 10% float test and a five-business-day fast-track entry clause for its Total Market, Completion, and Dow Jones U.S. Total Stock Market indexes, implemented June 8. Entrants in these total-market families may become eligible for derived size, sector, style, factor, and sustainability indexes.

• Nasdaq (Nasdaq-100): Old rules required a 10% minimum free float, and new listings waited for scheduled reconstitutions. A February consultation proposed fast entry for new listings with full market capitalization within the top 40 constituents, replacing the float minimum with a weight cap of five times free float. Nasdaq finalized the rule effective May 1, adopting fast entry after the 15th trading day based on evaluation on the 7th trading day, and softening the weight cap to three times free float. A company with 5% float that was previously excluded now enters at reduced weight. Reuters reported March 10 that SpaceX made early Nasdaq-100 inclusion a necessary condition for its listing venue; Nasdaq states the rule was not written specifically for SpaceX.

• FTSE Russell (Russell U.S. Indexes): Old rules added IPOs during scheduled quarterly reconstitutions and required a minimum 5% free float alongside a minimum 5% of total voting rights in public hands. On May 26, FTSE Russell adopted a fast-entry rule for IPOs with investable market capitalization above the Russell Top 500 breakpoint, forcing index addition after the close of the fifth trading day based on first-day closing prices. The platform’s February consultation explicitly named projected 2026 IPO targets motivating the shift: “SpaceX, OpenAI, Anthropic.” The final rule waives the 5% float and voting floors if future lockup expirations are expected to cure the deficit within 12 months, and adds the stock all at once rather than staging entry. Russell U.S. indexes benchmark $11.6 trillion, with $2.7 trillion in passive mandates.

• CRSP / Morningstar Indexes: Old rules required a strict 10% public float for fast-track IPO entries. Effective April 27, CRSP altered its float screen to allow fast-track inclusion if an IPO has a 10% float or if its float-adjusted market capitalization is at least one-half basis point of the index-eligible universe, which is roughly $3.3 billion. This index suite benchmarks more than $3 trillion, including Vanguard’s total-market complex. CRSP is transitioning its governance to Morningstar Indexes on July 1.

• MSCI (Global Investable Market Indexes): MSCI made no rule changes in 2026 because its Global Investable Market Index methodology has contained early inclusion loops for large IPOs since 2007. To qualify for early entry, an IPO must clear length-of-trading and liquidity screens, hit a full market capitalization of 1.8 times the interim size segment cutoff, and hit a free-float-adjusted market cap of 1.8 times half that cutoff. These dollar-scaled rules mean a trillion-dollar company with a low percentage float clears the hurdle automatically. On June 8, MSCI confirmed this process applies to SpaceX, with inclusion announced between the first and third trading day and made effective after the close of the tenth.

Appendix B: The SpaceX S-1 Balance Sheet Dissection

All figures are compiled directly from the June 3 amended S-1 corporate filing.

• The Offering Layout: The company is selling 555,555,555 Class A shares at an expected price of $135 each, raising approximately $75 billion before any underwriter overallotments.

• Valuation Multiples: Multiplying $135 by the post-offering Class A and Class B share counts (including outstanding restricted Class B) establishes a total valuation of $1.77 trillion. This multiple is approximately 95 times the company’s full-year 2025 revenue of $18.674 billion.

• The Voting Disparity: Class A shares carry one vote per share; insider Class B shares carry ten votes per share. Post-offering, Class B holders retain 88.5% of total voting power, and Elon Musk controls 82.4%. This triggers a “controlled company” status under Nasdaq rules, exempting the firm from standard board governance protections and giving Class B holders the unilateral right to elect a board majority.

• Segment Bundling: The corporate vehicle reflects a February 2026 transaction where SpaceX acquired xAI, following xAI’s 2025 acquisition of X. Financial statements are recast retroactively under common control to bundle four distinct operations: Starlink, launch/space, X, and xAI.

• Consolidated Loss Metrics: In full-year 2025, the company posted revenue of $18.674 billion, an operating loss of $2.589 billion, and a final net loss attributable to common shareholders of $4.937 billion, alongside adjusted EBITDA of $6.584 billion. For the first quarter of 2026, the company recorded revenue of $4.694 billion, an operating loss of $1.943 billion, and a net loss of $4.947 billion.

• Segment Operational Realities: The Connectivity segment (primarily Starlink) generated $11.387 billion of 2025 revenue, $4.423 billion of operating income, and $7.168 billion of adjusted EBITDA. The Space segment posted $4.086 billion of revenue and an operating loss of $657 million. The AI division logged $12.727 billion in capital expenditures for 2025 and accelerated to $7.723 billion in capital expenditures for the first quarter of 2026 alone — a run rate that annualizes to more than the entire company’s consolidated revenue.

• Treasury Crypto Holdings: As of March 31, 2026, the corporate treasury held 18,712 bitcoin with a reported fair value of $1.293 billion, down from a fair value of $1.637 billion at year-end 2025.

Appendix C: Academic Literature + Links for Your Research

Compiled academic literature detailing the transaction drag imposed by automated, schedule-driven rebalancing trades. These were cited and linked in-line, but I’m also inclufing them so you have an easier time sourcing the research/claims.

• Harvey, Mazzoleni, and Melone (“The Unintended Consequences of Rebalancing”): Documents that predictable, mandatory institutional index rebalancing trades cost investors roughly $16 billion per year in execution penalties. This extraction transfers approximately $200 per year out of every indexed U.S. household.

• Sammon and Shim (“Index Rebalancing and Stock Market Composition”): Evaluates rebalancing execution data in the Journal of Financial Economics and finds that predictable index rebalancing trades impose a structural drag of 46 to 69 basis points per year on passive index fund returns. The paper demonstrates that capitalization-weighted index math mechanically forces funds to purchase shares at peak run-ups and high-volume corporate issuances.

• Arnott, Brightman, Kalesnik, and Wu (“The Avoidable Costs of Index Rebalancing”): Tracks the performance delta of discretionary adjustments and finds that companies deleted from the S&P 500 outperform the mandated new stock additions by an average of 22% over the 12 months following index rebalancing. This proves that accelerated entry mechanisms force passive capital to buy assets when they are overvalued.

• Greenwood and Sammon (“The Disappearing Index Effect”): Analyzes inclusion day price behavior and details how the historic price pop on the day a stock joins the S&P 500 fell from an average of 7.4% in the 1990s to just 0.3% over the last ten years. The data shows that forced buying has not become cheaper, but has shifted chronologically: algorithmic traders chart the published index methodologies weeks in advance, accumulate the target stock early, and sell it to index funds at prices that already reflect the automated demand.

Sources:

• S&P Dow Jones Indices, June 4 results announcement and April 30 MegaCap consultation.

• Nasdaq, February 2026 consultation, summary of responses and conclusion, and methodology update explanation.

• Reuters, March 10 report on SpaceX seeking early Nasdaq-100 inclusion.

• FTSE Russell, IPO fast-entry consultation, May 26 press release, and technical notice.

• CRSP, float-shares investability screen notice and methodology guide.

• MSCI, SpaceX IPO update, Megacap IPOs explainer, GIMI methodology, and Corporate Events methodology.

• SpaceX, June 3 amended S-1.

• Anthropic, confidential draft S-1 announcement and Series H announcement.

• AP, OpenAI confidential SEC paperwork report; CBS News, OpenAI confidential IPO statement.

• ICI, Active and Index Investing, April 2026.

• Vanguard, VTI benchmark and VT benchmark; Invesco, QQQ benchmark.

• AFT, May 6 letter to the SEC; NYC Comptroller, NYS Comptroller, and CalPERS, letter to SpaceX.

• U.S. Department of Labor, March 30 proposed rule fact sheet.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

How can you develop AI projects that create measurable business value? This is the trillion-dollar question on everyone’s minds. Especially with the stories floating that most enterprise LLM projects never leave the demo/prototype phase.

The friction in deriving business value isn’t that language models are useless. The problem is that most teams do not know how to decide whether a model-powered feature is good enough to ship, too expensive to continue, or simply not worth building. So they keep optimizing with more prompts, switching to different models/frameworks, and overall burning more engineering time. But the business case does not always improve just because the system gets a little more accurate.

Production AI needs a different kind of discipline. Before a team debates model choice or prompt quality, it needs to understand the economic boundary of the project: what the system has to achieve, what failure is allowed to cost, and when the team should stop working on it.

This article is about how to deploy AI projects that are valuable instead of just technically impressive.

In this article, we’ll cover:

• How to define the minimum useful performance level for an AI feature before the team starts optimizing.

• How to decide when an AI project should be stopped instead of endlessly improved.

• Why higher accuracy does not always mean better business value.

• How to evaluate production AI systems around cost, reliability, latency, and operational risk.

• How to turn AI deployment from an open-ended experiment into a controlled engineering decision

To access the full article—and all premium breakdowns going forward/written prior—upgrade to a premium subscription below.

Subscribe now

If you believe deep insight deserves support, become a premium subscriber to allow me to keep doing the same.

Flexible pricing available—pay what matches your budget here.

Most companies offer learning or professional development budgets. You can expense this subscription using the email template linked here.

Read more

Stateful Swarms tackle this directly. Instead of relying on volatile inference windows or endless reprocessing loops, this framework coordinates multiple specialized agents through a persistent blackboard state. Each agent performs a targeted task and updates a structured, auditable memory. This persistent state accumulation means expensive processing is done once, after which systems can cheaply update, query, and reason over the structured knowledge indefinitely.

We validated this by running Stateful Swarms against the complete 1,251-task Harvey Legal Agent Benchmark, achieving an 83.74% pooled criteria pass rate, a 17.75% strict all-pass rate, at $1.30 per task. Critically, these results came from structural coordination improvements rather than raw model scaling or specialized engineering (all details, including code and experiment details, published here).

In this deep dive we will cover:

• Why long-context inference is fundamentally limited

• How standard agentic systems repeatedly waste resources

• What are Stateful Swarms and their structural advantages for agentic reasoning tasks

• Detailed results from our evaluation against Harvey LAB

• The essential role of auditability in production-grade systems

• Why persistent state is critical for scalable knowledge work

Given our commitement to open source and open science, the entire repository, benchmarks, and reasoning traces are open-sourced under an MIT license. We invite you to examine, validate, and extend our work directly (including using it commercially). Additionally, we are actively seeking engineering collaborators interested in enhancing the framework and building the next generation of agentic tooling.

This is a foundational step towards intelligent systems that truly remember.

(PS: If you want to use our Swarm Infrastructure, sign up for our API here: https://www.irys.ai/irys-api)

Executive Highlights (tl;dr of the article)

• Long-context inference is architecturally broken. Attention scales quadratically, memory bandwidth is physically capped, and model performance collapses as input grows (Liu et al., Chroma 2025).

• Expanding the context window doesn’t work. Training bigger models doesn’t fix positional encoding bias — a fact on page 3 gets treated differently than the same fact on page 97.

• Current agentic systems fail through statelessness. Decomposing tasks avoids context limits, but agents end up rereading processed documents and losing critical findings when context compacts.

• Conversational handoffs degrade signal. Passing information through natural language summaries causes single agents to outperform multi-agent systems under equal token budgets (Tran & Kiela, Stanford 2026).

• Stateful Swarms use a persistent blackboard. Specialized agents write provenance-tracked entries to an append-only, typed knowledge base instead of passing volatile context to each other.

• Read once, query forever. The blackboard accumulates structured understanding so you only pay the token cost to read a document set once.

• Costs drop by 39x. On the 1,251-task Harvey Legal Agent Benchmark, this architecture hit 17.75% strict all-pass at $1.30 per task, beating Harvey’s published 7.1% at $50.90 per task (w/ Opus hitting 10,7%, no costs mentioned, can assume same cost as 4.7 b/c the models have identical pricing). Doing the math, Swarms are between 66-98x more efficient than the best performers.

• Structure drives performance. Frozen Gemini models that scored 0% in standard agentic setups produced these results purely through structural coordination.

• Auditability is built in. Every blackboard entry tracks its type, source document, creating worker, iteration, confidence score, and support links.

• Failures are fixable. You can trace exactly where extraction succeeded but synthesis failed, which is a strict requirement for production in regulated domains.

• State is unavoidable. Recompute is waste, and prompt-as-memory is not a real solution for the economics of knowledge work.

Why Generative AI Currently Fails at Complex Queries?

The original promise of AI solving long-context knowledge work was AGI — a model that would solve whatever problem you threw at it in one shot. Just prompt and forget.

We’ve moved on from this fantasy, but some people still ride for a meeker version of this idea: eventually context windows will get big enough to where we don’t need any great engineering. However, when you dig into the details of how LLMs work, this fantasy face-plants real quick.

How Self-Attention becomes very Expensive in Long Context Inference

Attention computes a score between every pair of tokens. As context grows, compute scales quadratically. This means as your context length increases, the cost of computing Self-Attention grows very, very quickly. And this is something you have to recompute with every new message: meaning that you’ll spend a LOT of money recomputing mostly irrelevant tokens to answer even basic questions (keep in mind as your input context grows, you’ll most of your input tokens will likely be irrelevant or distracting to your actual query).

FYI, we still don’t know how to counter this issue. The most efficient edge models have kicked the can down the road by layering in their expensive full self-attention layers between stacks of cheaper variants. By not computing the cost of full MHA every layer, you can drive down costs of your network, but we still have to pay the piper eventually.

But lets say you bite the bullet on this because you’re determined to send all your tokens to one model. Your worries don’t end here. The most expensive aspect of long context work is in the hardware.

How HBM Makes Serving Long Context Expensive

To serve those tokens, the GPU must constantly move the KV cache in and out of High Bandwidth Memory (HBM). HBM is the most expensive component on the chip, and its bandwidth is physically capped.

Dao et al. formalized this as the IO-awareness problem in FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (NeurIPS 2022) when they found that the bottleneck in long-context serving is not the FLOP count of attention but the number of reads and writes between GPU SRAM and HBM. FlashAttention reduces memory accesses but does not eliminate the quadratic scaling of the mechanism itself.

Tldr: you want big memory chip for big context; you give Micron big money.

But let’s say you have a lot of money, and no actual AI expertise to solve problems (all too common in legal tech). So we go ahead with trying to build long context models, no matter the context. This is where you hit the next set of problems.

Training Models is Unreliable (especially over Long Contexts)

With all that money floating, couldn’t you eventually train a super long context model to solve all your problems? Not really.

The scaling curve for knowledge injection is not linear. Forcing a single model to hold all knowledge creates interference. Optimizing weights to improve reasoning in one domain often degrades performance in another. You can push a model to ace a benchmark, but you cannot reliably bake deterministic facts into its weights for production use. This is why LLMs are still hit with all kinds of random and unpredictable hallucinations, even for things that should be in their cutoff dates.

What about in-context learning (stuffing all the important context in memory)? Assuming your users are willing to go through the effort of sharing their entire strategy, relevant context, and jurisdictions in the context window, you hit attention degradation. A fact on page 3 receives different attention than the exact same fact on page 97. Liu et al. demonstrated this directly in Lost in the Middle: How Language Models Use Long Contexts: on multi-document QA, model performance dropped over 30% when the answer document was positioned in the middle of the context versus the beginning or end. As long as LLMs use Rotary Position Embedding (RoPE), which introduces a decay that biases attention toward sequence boundaries, we will continue to have to deal with this.

Chroma’s 2025 Context Rot: How Increasing Input Tokens Impacts LLM Performance study extended this finding across 18 frontier models and found that every model exhibited performance degradation as input length increased. The degradation is nonlinear and unpredictable: models can hold near-perfect accuracy to a threshold, then collapse.

So even if you really want to set your money on fire by training larger models for more context, you’re not guaranteed the returns you want. And when your model eventually fails, you’ll have no clue what steps to take to fix it (you can retrain, but that’s costly and doesn’t guarantee behavior).

This bitter pill is what has shifted LLM training priorities over the last 1.5 years. Open any modern LLM training recipe, and you see that the models are trained on tool use. This is because the labs have explicitly shifted their priorities away from single-shot capability to agentic systems — instead of hoping one call does everything, call a model a bunch of times, often including tools like calculators, code runners, web search etc. This way we can cover the gaps of each individual model call/method through a variety of techniques.

(when engineers use variety to make their lives better, they’re celebrated; when I want variety in my life, people say I have commitment issues. smh these double standards).

The added benefit of agents (imo the main advantage) is the traceability. If something fails, you can isolate it very quickly by tracing the tool calls and their I/O. You can finally see if your “DO NOT HALLUCINATE” instruction fired off as it should’ve.

Agentic Systems are amazing, and very useful. It’s no coincidence that the mainstream adoption of Claude and GPT in knowledge work came from their agentic systems CoWork/Anthropic and Codex, respectively. However, they have a few issues:

1. They’re very expensive to run, and they waste a lot of tokens relearning things (rereading docs they’ve already studied, rereading their agent trails etc) or redoing things because they lost instructions somewhere.

2. Remember the problems with building long context models? Still an issue. So they deal with long contexts by compacting their windows, which makes the above worse since a lot of important things are lost when we compact.

3. These agents can be hard to extend beyond their baseline

4. The pace of LLMs makes agentic systems very hard to trace and stop mid-run since they spew out an overwhelming amount of text very quick. This makes their outputs very unmaintainable and increases the amount of rework required. This is why many people report negative ROI from these agents (since they build systems/do work that they don’t understand and thus end up having to redo).

These were some of the issues that we’ve been trying to address at Irys when building meaningful long context agents for knowledge work. We’ve realized that the solution to this is not to incrementally improve a limited paradigm, but to rebuild stronger foundations.

Why Stateful Swarms Are All You Need for Knowledge Work

To understand why this architecture works, you have to break down the two terms individually:

• Swarm: The system replaces the single-model monolith with a fleet of coordinated AI agents. Instead of expecting one generic prompt to parse an entire matter, the architecture delegates tasks to specialized worker agents that handle distinct execution phases.

• Stateful: The system maintains a persistent, append-only, typed knowledge base that accumulates data over multiple iterations. The intermediate state survives across sessions, meaning the system doesn’t lose its mind when a user opens a new window.

We solve the coordination problem through the Blackboard Pattern. The blackboard is a structured shared state that functions as the primary asset of the system. Agents never pass context blocks directly to one another. Instead, a worker reads the blackboard, performs a tightly bounded piece of cognitive labor, and writes its findings back as a typed entry with explicit source provenance.

The system executes this process across ordered phases. A seed planning agent first maps the document landscape to define key extraction criteria, specific questions, and target schemas. Parallel extraction workers then populate the blackboard with observations, tracking metrics like source quotes and confidence scores. Higher-tier analytical reviewers step in next to synthesize these extractions, resolve conflicting data, and trace explicit evidence chains. Finally, a supervisor runs a deterministic convergence check: if knowledge gaps remain, it dispatches targeted follow-up workers to parse specific document sections rather than re-reading the entire corpus.

To prove the validity of this architecture, we ran it against the full 1,251-task Harvey Legal Agent Benchmark (LAB). The data proves that structural coordination matters far more than raw model scale.

The system achieved an 83.74% pooled criteria pass rate and a 17.75% strict all-pass rate across 24 practice areas. It did this at a total compute cost of $1,626.08, which averages out to $1.30 per task. Harvey’s published baseline on their twin holdout distribution reports a 10.4% strict all-pass rate at an average cost of $50.90 per task. The stateful swarm delivers a 39x structural cost reduction while significantly outperforming their best system.

(If you compute the cost per passing point, this jumps up to 98x the efficiency, but we’re not stressing that too much since performance is the biggest driver right now and we don’t have the private hold-out set).

This economic shift is driven by precise model routing over cheap, commodity infrastructure. The system routes high-volume parallel extraction tasks to Gemini 3.1 Flash Lite at $0.25 per million input tokens and $1.50 per million output tokens. We reserve the mid-tier Gemini 3.5 Flash strictly for reasoning-heavy steps like planning, cross-document analysis, and final synthesis, at $1.50 per million input tokens and $9.00 per million output tokens.

In Harvey’s baseline evaluations, these exact same Gemini models achieved a 0% strict all-pass rate when deployed inside standard agentic setups. We specifically picked these models to prove a point: by changing how the models interact — moving from stateless tool-calling pipelines to a stateful, iterative blackboard — the same frozen weights jump from absolute failure to 17.75% perfect execution. This is proof of our larger thesis that structural improvements can unlock capabilities not present in relying purely on the model layer.

Our system design allows us to tackle one of the biggest blockers in multi-agent systems. Tran & Kiela (Stanford, 2026) argued in Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets that every inter-agent handoff loses information — grounded in the Data Processing Inequality, which predicts that coordination can only degrade performance.

We show that their finding holds for conversation-based multi-agent systems where agents pass context through natural language summaries (which creates a loss of information and an ability to enforce constraints). It does not hold for blackboard architectures where agents write typed, provenance-tracked entries to structured shared state. Our blackboard doesn’t lose information at handoffs. That is why the same Gemini models that achieve nothing individually produce 17.75% strict all-pass when coordinated through one.

So far, we’ve discussed how Irys’s Stateful Swarms paradigm solves 3 of the main issues plaguing current agentic systems:

1. The shared blackboard means we can accumulate knowledge over time, reducing the tokens from re-learning.

2. Blackboard allows prevents loss of information on compaction and other issues.

3. Our use of self-defining swarms (analyst writes custom prompts for whatever problem needs it) also ensures that it can handle a wide amount of flexibility in the problems we can solve (I’d encourage you to try the GitHub CLI on your own problems; part of the reason we’re open sourcing it is to get your insights into what you’d prefer differently).

We haven’t talked much more about the audibilty and transparency. That was because our new blackboard allows us to pull off one of of the best feats of auditability yet.

Why Auditability Matters More Than the Benchmark Score

No matter how good, a system is bound to fail. But a wrong answer can still be a useful answer, if we understand where it went wrong and how to avoid that.

Let’s see how the blackboard enables that. In our Stateful Swarms, every task produces a complete, structured record of how the system arrived at its conclusions. Each blackboard entry carries its type, source document and section, the specific worker that created it, the iteration it was created during, its confidence score, and which other entries it supports or contradicts.

You can literally study the blackboards as they progress over iterations and use them to see how the system evolved it’s understanding and how it learned things over time. In an upcoming UI update to Irys, we’re going to expose this learning evolution to our users so they can stop/redirect things mid-investigation as our system finds interesting learnings.

The main advantage of the blackboard for transparency is that it’s much clearer to read and it centralizes all important information to one place (instead of scattering the learning fragments all over like Sukuna’s fingers). This allows us to study both the success and failure to improve the outcomes. Let’s understand how.

When the system succeeds, the trace demonstrates why. On a credit agreement comparison that scored 40/40, the blackboard evolved from 7 seed entries to 2,400 grounded findings over 12 iterations: 2,044 source-grounded observations, 87 cross-document calculations, 135 identified gaps, and 113 analysis entries identifying specific deviations. Each analysis entry carries a supports field linking back to the evidence it synthesizes. A reviewer follows the chain from conclusion to evidence to source document.

When the system fails, the trace reveals why — and whether the failure is fundamental or fixable. On a sanctions entity extraction task that scored 80/85, the five missed criteria shared a diagnostic pattern. The blackboard contained entries with two name variants — “Zenith Petrochem Industries LLC” from one document and “Zenith Petrochemical Industries LLC” from another. Both were correctly extracted. Both existed in the state. What the system failed to do was cross-reference these entries and flag the discrepancy. In a separate miss, the system identified that a 49% ownership stake sat 1% below the OFAC 50% threshold and even mentioned that aggregate ownership by blocked persons could trigger a violation — but didn’t elaborate the aggregation principle with sufficient specificity.

The pattern — correct extraction, incomplete cross-referencing — is qualitatively different from a system that never found the information. It tells you the failure is in state processing, not in extraction capability. These are fixable failures, and the blackboard is the mechanism that makes them identifiable. A monolithic system that missed the same criteria would give you nothing: did it not read the document? Read it but not extract the entity? Extract the entity but not notice the variant? No way to know.

For legal work specifically, this is not optional. Attorneys are professionally responsible for the accuracy of their work product. A system that produces correct answers 83% of the time is useful only if the remaining 17% can be identified and corrected.

(From a product perspective, this also means that we know why our system fails and we can improve our product much faster and w/o guesswork/relying on the luck of training a new model, since our improvements are structural).

What This Means in Production

Furthermore, this performance profile generalizes beyond legal analysis. The machinery of state tracking, multi-iteration convergence, and typed provenance requires no domain-specific training or weight adjustment. We are releasing the system as an open-source CLI that accepts any document set and task instruction. We’ve validated the architecture across internal repositories spanning multiple domains, and we are actively running benchmarks on non-legal verticals that we’ll be publishing in the coming months. The coordination logic is identical — the benchmark adapter is the only domain-specific component.

We checked this on 7 Datadog 10-K annual filings (FY2019 through FY2025). We asked a non-legal question — “Analyze how Datadog’s strategic priorities have shifted over the last 5–7 years using their annual 10-K filings (2020–2026). Produce a comprehensive investment memo covering product strategy evolution, go-to-market shifts, risk factor changes, competitive positioning, and financial trajectory.”

What happened (shared here):

• irys-stateful-swarms completed in 800 seconds. 12 iterations, 2,115 blackboard entries, 218 signals, 2.7M tokens. Produced a 12,657-word investment memo covering product strategy evolution (2012–2024 timeline), GTM transformation (10,500 → 29,200+ customers), competitive positioning shifts, financial trajectory ($362.8M → $2.68B revenue), and risk factor evolution.

• Claude Opus sub-agent failed after 415 seconds. Hit “autocompact thrashing” — context window filled up after 2–3
  filings, compacted, refilled, compacted again, gave up. Produced nothing.

(session export of the claude code run so you can fact check me is on the Github here; feel free to confirm for yourself).

In production environments, this stateful foundation unlocks a massive cost advantage over multi-turn interactions. Our enterprise platform, Irys, pairs swarm coordination with hierarchical embeddings and persistent indexes, and knowledge graphs to cut multi-turn inference costs by up to 100x compared to stateless re-computation on user defined matters (it’s also why we can ingest as many Gigabytes of documents, which none of our competitors can do).

Because the blackboard entries are structurally typed and tracked by worker provenance, the system can run deterministic algorithms directly on the state graph without calling an LLM at all. Operations like updating confidence scores across linked nodes, tracking fulfilled obligations, and flagging internal contradictions are handled via pure graph traversals and mathematical updates. When a user asks a follow-up question or introduces a new document amendment, the system targets and updates only the affected subgraph. You pay the steep token cost to read and extract a document set exactly once, then query against the accumulated understanding cheaply forever.

Conclusion: The Future is Stateful Swarms

Our open-source code uses standard API calls to baseline frozen language models — no latent space reasoning, custom embeddings, or proprietary retrieval engines. This minimalist approach ensured alignment towards benchmarks (which tend to be very atomic in what they allow) and also let us show what raw coordination logic can do on its own. But standard API loops, while fine for controlled benchmarks, crumble under real-world engineering realities and production costs.

The industry is obsessed with brute-force scale with massive models and sprawling context windows. The assumption is that stuffing infinite memory into a prompt is sustainable. It isn’t. As we detailed, relying entirely on the context window leads to attention decay, information loss, and spikes your hardware costs. Constantly reprocessing the same 3 2000-page document to answer a basic follow-up question is fundamentally broken. You end up paying full price to read the same text over and over again (yes yes prompt caching, but that’s a bandaid, not a fix).

Stateful swarms change this by moving memory out of volatile inference and into structured reasoning. This is the structured, auditable state the system refines and preserves. You invest in understanding a document set exactly once, then query it cheaply forever.

I’ll leave you with an observation. Recompute is waste. Pinning is waste. Session loss is waste. Someone will solve these problems, because the economics demand it. The future belongs to stateful systems that retain and refine memory persistently. This means that whether or not you are fully sold on stateful swarms, you can’t deny that state is one of the core problems for agentic systems to solve. And taking that as a given, it’s also impossible deny that the current “prompt as memory/state” solutions are woefully lacking.

So whether you agree with stateful swarms or you have an alternative idea, I’d say that there’s no denying that it makes sense to explore this part of the world together, as fellow believers in Statefulness. I’ll look forward to hearing from you.

Repository: github.com/dl1683/irys-stateful-swarms

Contact: devansh@iqidis.ai

Thank you for being here, and I hope you have a wonderful day,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

(Some of this article is inspired by my interview at the well respected tech publication—The Register— where we talked about some misconceptions around the costs of AI. Check it out here)

The AI market has a measurement problem. A year ago, you could evaluate a new model by looking at its MMLU score or open-source benchmark rankings. Today, those numbers are compromised. Models are routinely trained on the test sets (this is especially bad with the post training of models, which is why the new models are falling apart). Viral demos are heavily curated to hide latency and edge-case failures. Funding announcements are treated as proxies for product-market fit.

The result is a market where the signals decision-makers rely on are decoupling from reality. The metrics have become increasingly niche and decoupled from actual end-user priorities, due to which capital and engineering hours are being allocated based on data that is optimized for optics.

All that to say that the old signals are breaking and the market is scrambling to find new ways to measure the value of their work. We’ll show this manifests in various areas of the AI Industry Value Chain—

• “Better” is a meaningless metric. Benchmarks evaluate single dimensions. Real-world capabilities are multi-dimensional. The gap between what leaderboards measure and what production deployments require is breaking standard evaluation.

• Token maxing as an internal testing strategy, not a best practice. Tokenmaxxing will not make employees more productive. But it’s still the right bet for foundation model providers because tokenmaxxing allows gives them insights into the limitations of their current systems for live knowledge work. Labs are treating their staff as highly paid beta testers, harvesting their daily workflows to map exactly how models need to handle complex knowledge work to gain an alpha over the rest of the industry. All b/c the traditional measurements of good are completely worthless.

• Enterprises are canceling new AI licenses. Tokenmaxxing doesn’t benefit the non model providers where the tools provide baseline value, but they do not justify uncapped per-seat pricing. That margin between “useful” and “indispensable” is forcing a permanent correction in how AI is priced.

• Popularity metrics are actively harmful. On the distribution side, social impressions (which were initially a signal for how loved a product was) are gamed and entirely decoupled from enterprise buyers. Startups copying consumer growth playbooks are optimizing for reach that does not convert to revenue. Another deep example of how Goodharting traditional metrics has disconnected outcomes (sales) from traditional metrics that led there.

• Deployment partnerships are the only verifiable signal. Integrating a model requires a customer to change their operations and commit engineering resources. It is the only metric a startup cannot fake with a press release (althugh some startups are cooking their books).

• Where this goes from here. If traditional metrics fail at verifying capabilities, we need to reengineer our concept of verification to match the new technology.

(A quick note before we jump in: I normally write deep technical dives. Today is different. This is a shorter format for a market shift that doesn't require a 10,000-word essay, but does require your attention. If you find this useful, let me know and I will mix more of these into the rotation.)

Why LLM Benchmarks are Sorry.

Because billions of dollars in enterprise spend depend on capability rankings, the industry requires a single baseline metric for model quality. This is something that the market lacks right now.

The historical approach to this problem was the static benchmark leaderboard. For years, labs used centralized evaluation suites—such as MMLU for general knowledge or HumanEval for coding—to establish architectural superiority. If a model scored 5% higher than its competitor on a standard evaluation set, it was marketed as a universally superior system.

However, due to various reasons, this approach has hit a mathematical and operational wall. Static benchmarks degrade the moment they become performance targets. When a lab’s valuation is tied to an evaluation suite, engineers optimize the training process for that specific test. Evaluation data leaks into the pre-training mix, models are fine-tuned exclusively to match the test set’s exact formatting quirks, and leaderboard scores climb while underlying production capability stays completely flat.

Ultimately, we must remember that capability is a multi-dimensional optimization problem, not a single scalar value. A model that wins a leaderboard by maximizing accuracy on a static test often fails in production because it requires massive memory overhead, introduces unacceptable latency, or lacks reliable error-recovery loops during tool invocation. This dynamic is especially important since no one pays for best output on one individual inference call; people pay for best final work product. This often requires chaining multiple inference calls in different ways (and combining them with various deterministic calls), which dramatically ups the sources of variance and error. This is where benchmarks completely fall apart since the combinatorial variations of evaluations are practically infinite.

This complexity is why enterprises are struggling to optimize knowledge work. There are too many operational variables, and new interfaces constantly shift how users interact with text and code. This disconnect leaves external model providers completely blind to real-world deployment challenges. This blindness is unacceptable when we remember that these labs have raised hundreds of billions on the assumption that they will automate knowledge work, and well, you can’t automate what you don’t understand.

This lack of clarity is where the foundation of one of tech’s most hated trends comes in.

2. Why Token Maxing Is a Great Idea

Evaluating models in production requires continuous telemetry. This operational reality is the foundation of “token maxing”—a mandate from frontier labs requiring their internal employees to maximize their use of generative AI and their internal tools.

The market completely misreads this behavior. Outsiders look at these mandates and assume token maxing is either a financial trick to fake demand for investors, or a crude productivity metric. Let’s dissect why neither holds up.

The fake-demand theory breaks immediately against hardware constraints. Internal usage generates zero revenue and actively burns a lab’s margins. Compute is scarce. Every major lab is fighting internal power struggles over allocating GPUs between internal research and serving external API customers. Nobody burns constrained compute on vanity metrics when they are struggling to serve paying clients.

(And yes, labs lie, but there’s a bit difference between narrative manipulation and active fraud to investors. Trying to pass off internal usage as external demand would lean a lot towards the latter).

If labs aren’t faking demand, the only assumption left is that they are trying to boost employee output (mo tokens—> employee gets more done). But token maxing is not a productivity hack either (something that the critics get right; using more tokens is not going to get you more productive and will likely create all kinds of problems if not careful).

So why do it?

Because standard benchmarks are dead, labs do not actually know the limits of their own models. The capability frontier is jagged, and they are flying blind on billion-dollar infrastructure bets. By forcing tens of thousands of highly paid employees to use AI for everything, the lab harvests detailed signals on exactly where the model breaks, where it spikes, and what structural problems the next training run/engineering run must solve. In other words, this is institutional dogfooding, all to understand usage patterns/pain points that can guide the next generation of building (doubly important because most tech people don’t know non-tech workflows and would just be building in the dark w/o this kind of institutional-level signal extraction).

This seems like a lot more reasonable than whatever is being peddled publicly. You might be wondering why they haven’t clarified this publicly (or at least communicated this to their internal employees to ensure that the employees test it in the right ways instead of trying to game the usage)

The issue is that leadership at these labs can’t reveal this publicly. Confessing that they force token maxing because they don’t understand their own system’s boundaries would destroy any authority they have for making their grandiose predictions (building up hype) and expose their benchmark marketing as fiction (benchmark marketing is very convenient since it lets the labs worry about doing well in one, well-defined task instead of relying on vibes). So they bite the bullet on all the noise being mixed in with the hope that the signal will outweigh the noise.

Let me ensure that you really understand what I’m saying:

1. Tokenmaxing won’t make individual employees more productive.

2. It also won’t work for most organizations since they won’t really have a meaningful use for the signals derived and the ROI will be negative.

3. It is fantastic for frontier labs as IRL data collection.

All the criticisms of token maxing are from people who focus on 1 and 2 without considering 3.

Unfortunately, this secrecy is starting to backfire massively. By definition, most groups are not foundation labs, and thus these groups have a diminishing return on using AI. These groups make up the majority of the customer base for the labs, and they have not been happy with the token maxing outcomes.

3. Enterprises are starting to cancel AI licenses

AI tools were initially sold like traditional software with flat monthly seats, assuming predictable consumption, low marginal costs of usage, and a linear relationship between product usage and value to user. But generative AI breaks this in two ways. The value of an agentic loop is logarithmic: using Codex on the right problem can be very productive; trying to use Codex on every problem will see a massive output falloff (take something like a memo— one memo done fast is useful; a 100 memos published is overtly indulgent and will never be read).

The cost curve, however, is super-linear. As an agent works, the context window grows. Every subsequent step requires reading all previous inputs and outputs. If the model gets confused, it falls down rabbit holes and executes failed bash calls, drastically multiplying the compute required for a single task.

This structural waste is baked into the tools themselves. A product is a direct reflection of the constraints its builders operate under. Because foundation labs mandate unconstrained token maxing internally to gather telemetry, their own engineers never feel the financial friction of a super-linear cost curve. They build agentic systems that default to brute force—re-reading static context, forgetting learned facts, running unconstrained loops, and relearning codebases from scratch. The labs (implicitly) ship tools optimized for their own data collection, not the customer’s cost efficiency.

The labs cannot mitigate this by sharing disciplined, cost-saving best practices with their buyers. They cannot publicly tell customers to strictly constrain their token usage while simultaneously mandating internal token maxing. That hypocrisy would be called out immediately. So they stay quiet and leave enterprise buyers flying blind.

Without those constraints, enterprise budgets break. When Uber deployed Claude Code to 5,000 engineers, per-user costs spiked to $500–$2,000 a month. The company burned its entire 2026 AI budget in four months. Microsoft cut off internal Claude Code licenses for its Experiences and Devices division, forcing engineers back to Copilot CLI because the usage-based bill became unsustainable.

These cancellations permanently decouple token consumption from product-market fit since an enterprise ripping out a tool because the cost didn’t scale with usage is a failure. This leaves AI startups with a massive problem. If high usage metrics are now a financial liability, and real enterprise revenue is stalling behind budget caps, startups have no viable numbers to show investors. Desperate to demonstrate momentum without underlying fundamentals, the market is migrating to the cheapest, most easily manipulated signal available: manufactured popularity.

4. Cluely-fication is Ruining Startups

Let’s take stock of where we are:

1. Benchmarks are kinda worthless.

2. Enterprises are leery about purchases since the big name labs publish things that aren’t aligned with their needs (and enterprises typically don’t buy from startups).

3. All in all, we know that AI can be useful, we just don’t know how useful and when that scaling of utility inverts.

So how does a new founder/startup stand out in this space? How can they prove themselves to investors? Desperate startups and founders are migrating to the next cheapest signal: manufactured popularity.

To understand the most recent iteration of this trend, we must study Cluely. When co-founder Roy Lee faced disciplinary action at Columbia over an AI interview cheating tool, he turned the controversy into a content engine. Cluely set a target of a billion views, hired interns to farm daily TikToks, and leveraged that viral reach to raise a $15 million Series A from Andreessen Horowitz. Founders watch this timeline and assume stunt marketing is the new growth mechanism. Since then, social media has been inundated with cheap rage/engagement bait from founders desperate for a seed check.

(investors have made this problem worse by encouraging this behavior under the guise of community building).

This is a very short-term, immature read of the situation. Cluely proved that controversy can manufacture attention, but that attention does not convert into sales. In March 2026, Lee admitted that Cluely’s claimed $7 million in annual recurring revenue was false. The social reach was real, but the pipeline was an illusion.

Startups/Founders copying this playbook compound the error by applying it to enterprise infrastructure. Selling an AI agent requires CTO actual performance, strict security reviews, and a strong ability to inspire trust. A marketing motion built for a consumer scrolling TikTok at 1 a.m. does not shorten the path to that trust or act as “dev-rel”. Virality is only useful when it builds credibility with your core peer group. Pumping out provocative clips generates raw reach, but attention is worthless if it doesn’t convince your customers that you have what they need.

This collapse of proxy metrics leaves the AI ecosystem facing a hard floor. When leaderboard scores are optimized for PR, unconstrained usage becomes a line-item liability, and social virality fails to build peer trust, the industry runs out of measurements to rely on. Because they can no longer evaluate a vendor based on what a model might do, they are shifting to the only un-fakeable signal left on the board: deep, operation-level deployment partnerships designed to calculate the technology’s actual economic return in production.

And so we’re seeing the rise of AIs new hottest jobs, and partnership model.

5. Deployment is the only remaining signal

When benchmarks are gamed, usage burns budgets, and social virality fails to build trust, the market runs out of abstract proxies. Buyers now demand operational proof. This marks the end of the self-serve API era. You cannot deliver complex corporate transformation through a clean endpoint. You deliver it with engineers on the ground.

Deployment is the final signal because it is too expensive to fake. In May 2026, OpenAI launched DeployCo, a subsidiary backed by $4 billion from firms including TPG, Bain Capital, and McKinsey. Instead of building a traditional sales team, OpenAI acquired Tomoro to secure 150 Forward Deployed Engineers (FDEs) on day one. These engineers do not sell software. They embed inside client organizations to wire models directly into legacy data and core processes. The physical presence of an FDE is the new benchmark.

These embedded teams solve two critical vulnerabilities beyond telemetry. First, they protect the lab’s reputation. When untrained enterprise users brute-force an API and burn their budget, the foundation model takes the blame. FDEs control the implementation to ensure the system operates efficiently and delivers a measurable return. Second, they eliminate churn. A self-serve API is a commoditized endpoint that an enterprise can swap for a competitor’s model in an afternoon. An AI system wired deeply into a company’s core operational logic by an on-site engineering team creates massive switching costs.

This physical integration drives a shift from software licensing to equity swaps. OpenAI recently took a stake in Thrive Holdings, a private equity vehicle acquiring traditional accounting and IT firms like Shield Technology Partners. OpenAI provides no cash. It embeds engineering teams into these legacy companies in exchange for equity. By placing engineers inside messy, high-volume operations, the lab secures structural lock-in and a continuous stream of operational data.

Anthropic has responded with its own $1.5 billion deployment joint venture, while Google has tried to play a similar game by hooking Gemini credits within Google Cloud. All 3 have realized that competing on leaderboard scores yields diminishing returns. The enterprise fight is no longer about who has the best model. It is about who captures the enterprise workflow surface first. The base model is replicable. A proprietary workflow with permanent switching costs is not. Going forward, I’d expect this trend to grow and deeper partnerships to pop up for all the reasons mentioned earlier.

Conclusion: Where Do We Go From Here

Every failed signal in this piece failed for the same reason: it measured something adjacent to the work instead of the work itself.

This distance creates a system where our own traditional proxy measurements lie to us. Benchmarks lied about capability. Token-maxing lied about demand. Virality lied about adoption. All three were convenient fictions the market bought, hoping to avoid a harder truth: that intelligence can’t be scored, it can only be audited.

This shifts the game completely. The next era doesn’t belong to whoever engineers the best model. It belongs to whoever engineers the deepest visibility into how and where AI converts intelligence into margin.

If the old benchmark was “Who builds the best model?”, the new one is far nastier: “Who owns the accounting for intelligence?” The one who can verify intelligence/value more granularly then everyone else will be the winner.

More on this soon.

Thank you for being here, and I hope you have a wonderful day.

Dev <3

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Which parts of your work become more valuable when AI exists, and which parts become cheap? How do we upskill ourselves to ensure that we stay relevant in the agent of AI? People are grappling with such questions. To answer them, we must understand how the technology works at an architectural level — what deep learning actually does, where the architecture structurally stops working, and what falls out of that for careers, companies, and markets.

James Wang’s What You Need to Know About AI is one of the best attempts at doing that. left Bridgewater Associates to move to the Bay Area in 2013 after reading the AlexNet paper. He cofounded an AI healthtech company, did a stint at Google, completed graduate work in machine learning, and is now a venture capitalist at a deep tech firm specializing in AI, robotics, and synthetic biology. He also runs one of my favorite Substacks ( ) and I highly recommend his work for investors, technologists, and policymakers.

In this article, I will blend the book’s core ideas with my own frameworks on startup defensibility and AI market dynamics to help you understand how to thrive in the age of AI. More specifically, we will cover:

• Why deep learning works by forgetting, and what that mechanism tells you about where AI structurally stops being reliable

• How “bounded variation” answers three questions at once: will AI take my job, is this AI startup defensible, and is AGI close

• What Wang’s models-compute-data framework reveals about AI business moats, and how it connects to my own work on why incumbents fail to crush paradigm-shifting startups

• What “oracle-grade” expertise actually looks like in practice — not “develop judgment” but the specific capabilities that separate people who catch machine errors from people who cannot

• Why AI threatens the apprenticeship path that builds expertise in the first place, and how to use AI as a training partner instead of a shortcut that hollows you out

All to answer the question on everyone’s mind: how do you win against the Holy Ghost in the Shell?

Executive Highlights (tl;dr of the article)

• Deep learning works by forgetting. Its layers act as lossy compression that forces abstraction over memorization, bending the classical bias-variance tradeoff in ways that should not work according to standard statistical theory. That architectural property is why deep learning scales where classical ML could not.

• “Bounded variation” is the concept that resolves every AI career question, startup question, and AGI timeline question into something actionable. If your work lives within patterns AI has already seen, you are getting compressed. If it requires genuine novelty, counterfactual reasoning, or creative judgment beyond the training distribution, you are about to become more valuable than ever.

• The real class divide is not AI users versus non-users. It is people who can catch when the machine is wrong versus people who cannot tell the difference between a plausible answer and a correct one. Wang calls the first group “oracles.”

• Most expertise gets built through grunt work that AI now lets you skip. Unless you deliberately use AI as a training partner rather than a replacement for thinking, the tool hollows out the judgment it needs you to have.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Why Does Deep Learning Generalize Instead of Memorizing?

“Computer” used to be a job title. It meant a room full of people grinding through arithmetic for artillery tables and census data. When the calculator arrived, that job vanished. But computation didn’t. The people who survived the transition were the ones whose judgment about what to compute was the scarce resource. If you want to know whether your job is about to go the way of the human calculator, you have to look at what modern AI actually does at the architectural level.

Every stats 101 student learns the bias-variance tradeoff. If your model is too simple, it’s just inaccurate. If it’s too complex, it perfectly memorizes your training data but completely breaks the second it sees something new. Classical machine learning is a permanent balancing act between those two failures.

By that logic, deep learning should be a disaster. Modern LLMs are massive enough to just memorize their entire training sets verbatim. Zhang et al. proved this a while back. They took standard deep neural networks, fed them datasets with completely random labels, and the models perfectly memorized them anyway. The raw capacity for rote memorization is sitting right there. And yet, on real data, they generalize. Something in the architecture physically stops them from taking the easy way out.

That something is forgetting. Deep neural networks act like a giant lossy compression pipeline. As data moves from layer to layer, the specific details get destroyed. No single layer has enough memory to hold the whole dataset. Each layer only keeps the structural shape of the data that the next layer needs. By the time information has passed through a hundred layers, the specifics are gone. Only the general pattern survives.

Shwartz-Ziv and Tishby looked at the math behind this and found that training actually happens in two phases. First, the network learns to represent the data. Then, it actively starts throwing away anything that doesn’t help with the task. That compression phase is where generalization happens. Wang calls this “learning by forgetting.” It is structurally a lot like how human memory works. You don’t store a photographic record of every dog you’ve ever seen. You store a compressed, abstract prototype of a “dog,” and you match new animals against it. The layers force the neural net to build prototypes instead of databases.

Some internet AI gurus read stuff like this and assume that this mechanism makes deep learning strictly superior to classical methods. Make sure you stay away from such people because these guys are dum dums.

“Know-nothing machines” are brilliant at generalizing because they burn away the specifics. But sometimes you actually need the specifics. If you are working with strict logic, exact retrieval, or 5,000 rows of tabular customer data, deep learning’s best feature becomes its fatal flaw. The network will hallucinate a perfectly shaped, highly plausible answer that is completely wrong, because it literally threw away the exact facts to save the pattern. The key is in mixing all the relevant techniques, as discussed here.

But back on topic, this understanding of Deep Learning’s generalization is what leads us to one of Big Man J’s most interesting frameworks for evaluating the capabilities of AI.

What can AI Do? Where Does AI Break?

JW calls the direct consequence of this forgetting mechanism “bounded variation.” Because deep learning compresses experience into prototypes, it handles variation perfectly within its training distribution. Step outside that distribution, and it doesn’t gracefully degrade. It confidently outputs garbage that matches the structural shape of a correct answer without actually being correct. This is just how compression works.

So, will AI take your job? Only if your day-to-day is bounded variation — tasks fully described by patterns in the training data, where you are just rearranging surface-level structures. If your work requires genuine novelty, counterfactual reasoning, or judgment in contexts a model has never seen, you are on the safe side of the boundary.

What about scaling businesses? How do you build a defensible AI startup? Wang breaks it down into models, compute, and data. Models are open (and the gap between closed models is mostly a mirage). Compute is just capital — xAI caught up to the frontier by buying 100,000 Nvidia GPUs for $6 billion, which is minute to Meta’s $72 billion 2025 capex. That leaves data, but only data with friction. Everyone has the same scraped internet text. Moats only exist in data that touches the physical world, where collection is painful, expensive, and slow. Wang points to Oncoustics: they turn raw ultrasound signals — data hospitals usually throw away — into liver diagnostics. You can’t replicate their model because you physically cannot get their data.

This is where we will diverge from our author, though. Data friction is a great foundation, but in my opinion, it’s not enough. I’ve written before, true defensibility requires changing the fundamental “unit of value” and crossing an “evolutionary valley.” Incumbents are great at copying wrappers, but they choke when a product forces them to restructure their workflows and org charts. Look at coding: Copilot optimized the old autocomplete workflow, but Cursor and Claude Code broke it by shifting the value to repo-scale delegation and autonomous execution. Incumbents stretch their old systems; disruptors build new ones.

I think Data with Friction is a relatively low-value way to shift your unit of value/build a moat because my experience as AI engineer + talking to lots of AI guys through my open source community leads me to believe that the model layer is a really bad place to build your business. Competitors can spwan, it’s actually really hard to control how things will happen ahead of time, and you’re stuck fighting costly wars of attrition. Data + Frictions and the Models they lead to are a fine starting place, but they have to be a lead into something else in your ecosystem to really value-max. But that’s just my take on this situation and I’m not going to spend too much time on it here.

Let’s bring this back to JW, because his analysis of bounded variation defines the ceiling for AGI.

How will we know when AGI is close? When a system crosses from bounded to unbounded variation. JW maps this constraint to Judea Pearl’s Ladder of Causation.

• Level one is association — finding correlations. Deep learning owns this.

• Level two is intervention — understanding what happens if you actively change a variable. AI can only fake this if the intervention exists somewhere inside its training distribution.

• Level three is counterfactual reasoning — imagining what would happen in a completely novel situation that has never actually occurred.

This is where the architecture hits a hard stop. Counterfactual reasoning requires generating genuine, structural novelty. Deep learning operates entirely by remixing compressed prototypes of things it has already seen. You cannot compress your way to something fundamentally new. This is why ARC AGIs have been gamed by types of scaling (we see this since every new generation of the benchmark resets the whole board to 0, which would not be the case if models had actually generalized/built intelligence).

If you’re interested in attempts at counterfactual reasoning, my recommendation would be to check out the following deep dive on Diffusion Models and play with our repository on Latent Space Reasoning to see next gen techniques that can emulate these reasonings.

This framing also gives us a great framework for evaluating our own skills.

What Does Real Expertise Look Like When AI Can Fake It?

AI produces acceptable output at scale for almost nothing. A junior with ChatGPT can generate fifty ad variations in an afternoon, or scaffold an application they do not know how to maintain. The real danger isn’t genius output. It is passable output that gets accepted without scrutiny.

This is problematic because failure rates are massive, and they are not dropping. A 2025 Stanford RegLab study tested Lexis+ AI and Westlaw AI-Assisted Research — both platforms marketed on eliminating hallucinations via retrieval-augmented generation. Lexis+ AI hallucinated 17 percent of the time. Westlaw hit 33 percent. These weren’t obvious fakes. They were subtly mischaracterized cases and inapplicable authorities. A 2024 JAMA study found ChatGPT misdiagnosed 83 percent of pediatric cases. At NeurIPS 2025, expert peer reviewers missed 100 fabricated citations across 53 accepted papers. You cannot engineer this out. A 2025 mathematical proof by Karpowicz confirmed that under current LLM architectures, hallucinations are structurally inevitable.

An AI is only as powerful as the person checking its output. An expert using AI moves faster because they see exactly what is missing or wrong. A non-expert using AI just moves faster toward errors they cannot detect .This leads to our new mental model. Expertise is the ability to judge a model’s outputs and distinguish the good from the bad. Wang calls the human verifier the “oracle.”

More concretely, here are some flavors of Oracle-grade competence:

• You identify the failure before it happens. You know where the training data is thin, where edge cases live, and where ambiguity breaks statistical patterns. A lawyer knows which precedents are unsettled. An engineer knows which system interactions produce undocumented emergent behavior.

• You specify what good looks like upfront. Not “write a contract,” but “draft a clause that survives a Delaware Chancery challenge on fiduciary duty using these three precedents.” Anyone can prompt. The constraint is the expertise.

• You distinguish plausible from correct. AI pattern-matches to correctness. A doctor catches when a statistically likely diagnosis doesn’t fit the specific patient. An analyst spots the wrong baseline.

• You know the right question. Most people use AI to answer the question they already have. The oracle sees when that question is incomplete or downstream of the actual problem.

JW illustrates this with Circuit Mind, a company compressing electronics design from weeks down to seconds. The AI handles the drudgery of ingesting data sheets and solving layout constraints. But the engineer makes the final call on which generated design to ship, weighing tradeoffs in cost, power, and domain nuance. The AI strips away the grunt work and exposes judgment as the actual job.

This gives us a simple test for the depth of your skills. Imagine a smart but inexperienced person with ChatGPT and unlimited patience. Could they produce output close enough to yours that a client accepts it? If yes, your expertise is not deep enough yet.

However, this presents us with an interesting paradox, one that the world is already struggling with. If the best people to use AI tools are experts, and these tools can do a lot of the tasks that turned juniors into experts, how are we to raise a new generation of experts in the era of AI?

As the Bible says, Modern Problems Require Modern Solutions.

How Do You Build Expertise When AI Automates the Grunt Work?

Typically, seniors built their expertise by being juniors. People learn from the process of breaking down a problem, producing an output, and iterating on feedback until they can implicitly learn what good means in various circumstances.

AI lets you skip the reps. A junior developer can generate working code without struggling through the logic. An analyst can produce a financial model without knowing which assumptions actually matter. The output looks identical. The understanding does not exist. This also means that a senior can likely get work done faster through AI than by guiding juniors, which creates less of a short-term incentive to hire them. The result is the youth unemployment crisis we’re all hearing about.

This leaves us with two problems to solve:

1. How do we ensure that a junior can upskill in a world with AI without losing out on the productivity of using AI?

2. How can a to-be junior succeed in finding work and standing out in a hiring crunch?

The latter is complicated, but in my experience the best bets are to be very active in tech communities (to increase your surface area/exposure to more people) and to look more aggressively at startups. Startups tend to be much less concerned with credentials and have a bias towards hiring GOOD juniors (cheaper + younger people are more likely to keep up with the startup grind). This leads us back to the first question — how to get good?

(this also applies to anyone that wants to learn more of a skill with AI).

Put simply, you have to engineer your own friction.

• Do the work first. Draft your own solution, then compare it to the AI’s output. If the machine produces something better and you cannot explain exactly why, you just found your next learning target.

• Build an error log. Track exactly where AI fails in your domain. That log maps the boundary where statistical pattern matching breaks down and human judgment is required. That boundary defines your market value.

• Stay at the edge. If you only use AI on tasks you have already mastered, you are automating comfort. Use it on hard problems that stretch your judgment in real time.

• Find human pushback. Find communities to share your ideas (reddits, slacks etc) where you can pressure test your mental models and iteratively refine them.

• Study the catastrophes. Do not just learn current best practices. Learn the historical failures that made those practices necessary. Tracing a standard back to the disaster that created it gives you context the training data cannot reliably synthesize.

A more detailed learning framework was broken down in our article: How to learn AI (or anything technical) in 2025. It contains the guide for both technical and non technical people to learn about AI.

James arms us with several powerful concepts with which to study the past of technological revolutions and use them to model the future. Let’s end by bringing them all together to understand how we can thrive in the Age of AI.

(again, if you want to learn more about these ideas and pick up some cool history, buy the book here).

Conclusion: Therapists Will Save the World

To see where AI takes us, look at the last technology that drove the cost of access to zero. When WebMD launched, people thought doctors were obsolete, just like they thought StackOverflow solved programming. Neither happened. The internet drove the cost of raw information to zero, shifting the market premium entirely from access to judgment.

When knowledge becomes abundant, it stops acting as a signal. Anyone can audit MIT lectures online today, but that didn’t democratize hiring — it just forced companies to use automated tracking systems to filter out non-elite degrees before a human ever reads the resume (and irony of all ironies — some of the people that talk most aggressively about how college is no longer needed to learn have the highest amount of credentialism). Once everyone has the baseline knowledge, the market premium migrates to whatever stays scarce, like elite pedigree or unfakeable track records. Free information didn’t kill the gatekeepers. It just narrowed the gates.

AI runs the exact same playbook, just faster, by handing everyone free junior-level competence. The floor rises, bringing back the naive prediction that the field is level and everyone is a generalist now. But when junior-level competence is free, it doesn’t differentiate anyone. This is why I think that AI actually drives hyper-specialization. You use the machine to patch your weaknesses, stealing baseline generalizations from other fields so you can push your actual specialty to the absolute limit. This is the only meaningful way to differentiate yourself going forward.

Seen that way, your only real constraint is figuring out exactly where to build that deep specialization and which adjacent fields to steal from. You have to understand your own mind well enough to know what is actually yours to amplify. To know yourself enough to know which adventure you can keep compounding on the longest.

So I guess we’ve spent a few thousand words to come to a pretty weird conclusion: the secret to thriving in the age of AI is to go to therapy. Who’d have thunk?

Thank you for being here, and I hope you have a wonderful day,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

1. AI was getting deeply unpopular right now because it was alienating many sections of society to enrich a very small minority.

2. The current paradigms of investing in AI (stuff like SaaS productivity tools network effect marketplaces, and social media/entertainment) were hollowing out since the space had become overcrowded and competitive. The next generation of era-defining technology would require hard engineering/scientific research and questioning the foundations of our system.

3. Both the above meant that the next generation of great companies would be the ones that came from the ignored aspects of society (the anti-thesis) such as: deep tech, AI for low tech industries (which we broke down in depth here), etc.

Resharing this article because several events have validated this analysis. People hate OpenAI enough to post fake ads like this on the trains—

while data centers, AI slop, and other AI related things continue to be at an all time low.

On the deep tech front, Cerebras had an explosive IPO (covered here) while YC put out a call for startups specifically targeting the avenues we mentioned (agriculture, hardware supply chains, and more):

Given that this trend is likely to continue, I’m resharing the framework so it helps y’all.

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction. We run on a “pay what you can” model—so if you believe in the mission, there’s likely a plan that fits (over here).

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

“Outsized returns often come from betting against conventional wisdom…”

-Jeff Bezos

Anthropic just secured a potentially era-defining legal win: the right to use copyrighted material in LLM training — though they’ll still stand trial for pirated sources. The legal fallout will take time to unfold (we’ll cover it on the Iqidis LinkedIn), but there’s a more immediate signal I want to track: AI hate is metastasizing.

This development will no doubt compound the massive undercurrent of AI hatred that’s already brewing online and in many spaces. If you follow popular Substack writer , you will see more and more of his posts have taken an anti-AI sentiment as well.

Some of this negative emotion is based on very real concerns that Tech has ignored, as many Silicon Valley voices push a very rigid techno-utopia while forgetting to ask anyone if they want to live there. Some is general distrust given Silicon Valley’s legendary knack for birthing problems while preaching salvation.

A lot of it is just professional whiners trying to squeeze a few more fear-clicks out of doomerism.

But either way, this is a generationally good space.

The way I see it: where there is smoke, there is an opportunity to have a sick barbeque. A BBQ where I can shove a lot of big, thick, juicy meat in my mouth. And it’s my life philosophy to never pass on such moments.

In this article, I want to lay out a high-level investment framework to lay out why investing in this increasing antipathy is the correct thing to do —

1. Financially: lots of assymetric opportunity where small investments will net bigger upside

2. For better technology: Every dominant thesis loses sight of it’s limitations. A strong antithesis is important to explore new directions.

To do so, I will adopt elements of the Hegelian Dialectic to present why the “AI hating movement” can be seen as the antithesis to a world that is overrun with AI Hype and increasing disconnection b/w Tech and many other groups. Meaningful progress requires the combination of thesis and the antithesis to create a synthesis — where a new more robust system is created that can be adopted by the masses.

Let’s get a lot of girthy meat in our mouths.

Executive Highlights (TL;DR of the Article)

This article presents a strategic analysis of the current AI landscape and its likely evolution, arguing for a contrarian investment and development approach. Key points include:

1. Current AI “Thesis” Imbalance: The prevailing AI development paradigm is characterized by rapid expansion, high valuations, and a tendency towards copycat solutions, often overlooking significant risks and alienating potential user segments due to unaddressed concerns.

2. Emergence of an “Antithesis”: This imbalance creates a natural opportunity for an “Antithesis” — solutions, platforms, and narratives that directly address the weaknesses and overlooked areas of the dominant AI Thesis. This includes focusing on issues like data sovereignty, algorithmic transparency, constrained/specialized AI, and decentralized infrastructure.

3. The “Antithesis” as an Investable Opportunity: The growing discontent and unaddressed needs represent significant market opportunities. Investing in or building these anti-thetical solutions is positioned not as a negative stance, but as a strategic bet on a necessary market correction and evolution.

4. The “Synthesis” as the Path to Mass Adoption: The interplay between the Thesis and a robust Antithesis is projected to lead to a “Synthesis.” This phase will integrate the strengths of both, resulting in AI technologies that are more trustworthy, secure, and aligned with broader societal needs, thereby enabling true mass market adoption and creating new, expansive value streams.

Understanding this dialectical progression (Thesis-Antithesis-Synthesis) provides a framework for anticipating market shifts and making proactive strategic decisions, rather than merely reacting to prevailing hype cycles.

Why Continue Reading?
The subsequent sections will provide a detailed breakdown of:

• The Hegelian dialectic as an analytical tool for understanding technological and market evolution.

• A critical analysis of the current AI Thesis and the shortcomings of existing critiques.

• The defining characteristics and pillars of a viable, constructive AI Antithesis.

• Actionable strategies and tactical vectors for investing in and developing these an-thetical approaches, including a timing matrix for peak opportunities.

• A deeper exploration of how the Synthesis phase leads to broader market creation and universal technology adoption.

This framework is intended for those looking to understand the deeper structural dynamics of the AI market and identify strategic opportunities beyond mainstream narratives.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

2. The Dialectic: Why Every Dominant Idea Eventually Digs Its Own Grave

The Hegelian dialectic isn’t about ideas fighting in a vacuum. It’s about how power systems decay.

Every dominant idea — every thesis — is built on exclusions. Not just what it says, but what it doesn’t allow. And those exclusions are not static. They fester. They grow. They compound into something that eventually bites back.

Thesis: The Guiding Myth of the Present

A thesis starts as clarity: a compelling, high-leverage worldview that makes things move. In AI’s case, that’s the belief in scale, speed, and intelligence as ultimate goods. The more data, the more compute, the more automation — the more “progress.” The problems don’t start b/c this belief is “wrong”.

The problems start when it thinks that it’s enough. All consuming. Good enough to meet all your needs. Worth a complete leap of faith, an unquestioning surrender. The problems start when it tries to be your one and only.

But here’s the structural flaw: a thesis breeds overcommitment. The market doubles down. Institutions calcify around it. Smart people stop asking questions and start building copies. Demo Days become parades of product incest.

And then something breaks. And that, you whose smile justifies all of existence, is what we’re interested in right now.

Antithesis: Counter-Culture by Another Name

The Antithesis isn’t born from random malice or irrational fear. It doesn’t just pop out of thin air. It emerges from the very fabric of the thesis itself — from its blind spots, its broken promises, its power imbalances, and its arrogance. It’s the return of those suppressed variables, the bill coming due for all those conveniently ignored costs. It is a necessity, not mere negativity.

1. Oversights & The Unseen: No single idea, no matter how brilliant, captures the whole map. By its very nature, a dominant thesis optimizes for certain variables and actively ignores or devalues others. Think of it as a spotlight: whatever it illuminates brilliantly casts equally deep shadows. AI, in its current frenzy, optimizes for speed, scale, and computational power. What gets thrown into those shadows? Human nuance, ethical friction, the unquantifiable, the deliberately slow. These aren’t just minor omissions; they’re accumulating debt.

2. Contradictions & Scale Failure: An idea might look flawless on a whiteboard or in a controlled lab. But scale it up, unleash it into the messy, unpredictable real world, and the internal contradictions start to buckle. The AI dream of unbiased omniscience crashes into the reality of biased training data. The promise of democratized power meets the reality of centralized control by a few mega-corps. Every triumphant scale-up inevitably hits these failure modes — the points where the thesis chokes on its own success. All kinds of AI techniques are forced to grapple with their inadequacies, forced to confront all the ways their simplifying assumptions limit their potential.

1. Power Shadows & The Revolt of the Unrepresented: Ideas don’t exist in a vacuum. They serve power. The AI thesis, for all its utopian rhetoric, is currently concentrating power and wealth at an astonishing rate. Those left out, those whose livelihoods are threatened, whose values are ignored, whose data is hoovered up without consent — they don’t just disappear. They become the raw material of the antithesis. Their resentment, their fear, their legitimate grievances become a potent, reactive force. After all, Power ignored always, always, retaliates.

2. Hubris & The Corrective Backlash: When a thesis becomes too dominant, its proponents too dismissive, its narrative too unquestioned, it breeds a special kind of arrogance. Critics are labeled Luddites, fools, or enemies of progress. Systems that reject all critique don’t become stronger; they become brittle. They increase the number of people who want to take the systems apart, who will dance when things fall apart (look no further than the ecstasy with which so many AI critics painted the Apple Reasoning Paper as their Jean D’Arc).

This is a crucial point to understand. The antithesis isn’t a Twitter troll. It’s not the “revolutionary” that hates the world b/c they’re too cowardly to face their own inadequacies. It’s the structural consequence of the thesis’s own expansion. It is what the thesis cannot metabolize.

The antithesis rises — not out of spite, but out of structural necessity. It gives voice to the excluded, leverage to the misfit, and clarity to the fatigued.

This is where we are now. The AI thesis is so dominant that it’s blind. And that blindness is breeding heat: cultural, legal, infrastructural. The backlash isn’t noise. It’s the next investable signal.

To understand this better, let’s take a deeper look into our current thesis and all the elements that it misprices/underestimates.

3. The Current AI Thesis: A Case Study in Overconfidence and Misplaced Risk

In the previous section, we outlined the mechanics of the dialectic: how every dominant idea eventually collapses under the weight of its own blind spots. But theory is only useful when it meets the moment.

So let’s bring it home.

Today’s AI thesis — the set of assumptions, incentives, and narratives driving the current wave of investment, deployment, and cultural fixation — isn’t a diversified portfolio of bets. As financial genius Benjamin would say — “we’re jacked to the tits” on a handful of fragile premises.

This isn’t just about overconfidence. It’s about systemic mispricing — a market drunk on inevitability, blind to its own counterparty risk, and dismissive of the liabilities it’s compounding. Let’s understand how.

The Underestimation of Inertia

The thesis assumes AI’s rise is inevitable, frictionless, “already won”. Anyone can now create better processes with AI, and then make millions by selling to people who are dying to enhance their productivity.

But that’s not how large systems behave. At all. For proof, look no further than where so many Silicon Valley giants put their energy. Peter Thiel, who will not shut up about government inefficiency, the importance of freedom, and the principles of small government, is also lapping up massive government contracts building surveillance systems used by vigilantes to circumvent due process. Read that sentence a few times.

This view discounts the sheer inertia of entrenched infrastructure, regulatory drag, organizational resistance, and cultural fatigue. It ignores the energy required to overcome embedded workflows, jurisdictional law, unionized labor, and psychological resistance to automation. Thiel and the rest realize this, which is why they’re aggressively targeting massive government contracts (to take advantage of this inertia).

And yet, every founder who parrots him builds like inertia doesn’t exist. They believe virality and clever prompt chaining will break institutions that barely adopted PDFs.

At Iqidis, we sell to lawyers. Ask us how that’s going.

What’s actually happening in the space:

• Valuations are priced for frictionless adoption.

• Early traction (mostly from warm intros and FOMO) masks the grind.

• Copycats flood the space, creating paralysis by similarity for buyers.

People don’t buy when they feel overwhelmed. They stall, nitpick, or default to ChatGPT.

The Rise of “Move Fast and … Collapse Suddenly” Players

“Move fast and break things” worked when the stakes were low. But the AI thesis keeps applying it to environments that punish brittleness.

The AI thesis systematically undervalues:

• Architectural depth (here I don’t use depth as “neural network w/ more layers”).

• Rigorous testing.

• Robust iteration loops.

It rewards flashy launches and first-mover advantage, even when the system beneath is rotting. Foundational infrastructure is ignored in favor of ship-now features. Long-term adaptability is sacrificed for an increasingly complex demo.

This was a phenomenon noticed by a16z, who dedicated a whole section in their lessons on applying AI for Enterprise to how Demos and Product are different beasts-

If you want an example of this phenomenon in action (and another example of how ahead of the curve our research is), look through our deep dive on Cursor and its many security issues. They’re caused by a poor understanding of AI, context management, and on how to build multi-step reasoning. That’s why Cursor does great with small demos (where these things don’t matter) and horribly with actual software work — 

This has been the downfall of many startups, and this lack of appreciation for friction-heavy systems is why so many sectors view AI w/ so much distrust and readily adopt “AI is a snake-oil” narrative, even when AI would only make their lives easier. To top this section off, here is a quote from JP Morgan:

“Companies rushed to deploy AI without understanding the consequences. The mandate was clear: innovate or die. But JP Morgan’s latest security assessment reveals that:

• 78% of enterprise AI deployments lack proper security protocols
• Most companies can’t explain how their AI makes decisions
• Security vulnerabilities have increased 3x since mass AI adoption

The problem? Speed > security.

JP Morgan’s CTO Pat Opet put it bluntly: “We’re seeing organizations deploy systems they fundamentally don’t understand.” The financial sector is particularly vulnerable — with trillions at stake.

What JP Morgan recommends:

→ Implement AI governance frameworks before deployment
→ Conduct regular red team exercises against AI systems
→ Establish clear model documentation standards
→ Create dedicated AI security response teams

JP Morgan itself has invested $2B in AI security measures while slowing certain deployments.”

-Source. One might argue that the increased interest in AI specific security is an early indication of people hitting against the limits of the thesis.

If you’re looking to invest in the next generation of AI-Security, here are some of the trends that we’ve marked as useful-

More options and deeper analysis is present in AI Market Report for April.

The Reckless Bet on Disruption

Disruption is treated as an unqualified good. But in many sectors — the things being disrupted include oversight systems, institutional safeguards, and trust layers. Think of how crypto mocked slow regulation… until billions evaporated to vaporware and fraud.

The AI thesis is repeating the same mistake:

• Deploying systems with zero audit trails.

• Integrating tools with undefined failure boundaries.

• Replacing human intermediation with opaque, auto-generated outputs.

The systems being built now are attack surfaces.

And the real cost will show up not as a failed product, but as a failed system.

The Discounting of Dependency Risk

Every new layer of AI infrastructure adds dependency on centralized APIs, on foundation model vendors, on foreign GPU supply chains.

The thesis prizes efficiency and integration. But it underprices:

• Vendor lock-in.

• Infrastructure fragility.

• Geopolitical exposure.

• Data ownership erosion.

We are building a generation of startups — and, soon, institutions — on infrastructure they do not control. And the cost of that dependency is still unpriced.

Given these glaring mispricings, one might expect a robust counterforce — the antithesis — to be gaining ground. Isn’t that what a lot of the “AI Skeptics” or so called “anti-bullshit” people do?

While these guys should have been the contrarian voices, the state of AI critique has devolved into mostly noise. Their shallow critique often turns useful debates into a discussion on semantics, is easily ignored by many since their criticisms generally lack meaningful bite, or adds to the hype around AI, just in a different direction (cue X-Risk folk).

In each case, these critics only add to the thesis.

The Failed Opposition: Why No Real Correction Has Happened Yet

“ Amidst this cacophony, the public conversation is increasingly being framed as a duel between the Alarmists, who foresee doom, and the Accelerationists, who champion unrestrained progress. In bypassing a critical examination of the arguments from these polarized viewpoints, we risk legitimizing claims and scenarios that may lack any grounding or information value.”

-A Risk Expert’s Analysis on What We Get Wrong about AI Risks.

That’s the real problem: polarization isn’t revealing truth — it’s scrambling our priors. And the people most responsible for holding the thesis accountable are instead helping it metastasize by misdirecting attention, weakening the signal, or just playing for clicks.

Let’s break them down.

The Safetyists & Doomers

The existential risk crowd obsesses over AGI gods and paperclip apocalypses, while the actual systems being deployed today escape meaningful scrutiny.

Here’s some of what gets ignored:

1. Attribution for Gen AI Training data: Creators fuel the GenAI machine. Our work is never credited in LLMs or image models. Creators enable the viability of Deep Research. And yet Deep Research also cause a direct drop of traffic to creators. Furthermore, platforms hosted by creators deal with increased hosting costs(Wikipedia is paying tens of thousands of Dollars monthly in extra costs from the crawlers constantly hitting their website) and no compensation. This is broken.

2. The constantly falling transparency around their models, making evaluations and assessments of true deployment risk much harder.

3. Complete lack of transparency on the environmental impacts (water and emissions being the two large ones) of these models. The people at most risk from climate shifts (poor people, especially in global south) are not the ones who benefit most from these technologies. The people who pay the costs aren’t the ones who reap the rewards,

And yet what do we get from the AGI priesthood? Dario Amodei and Ilya Sutskever warning us about superintelligence and “misalignment,” while positioning themselves as the only ones qualified to “safely” build the future — conveniently centralizing control in their own hands. B/c they’re the only ones capable of saving us idiots from building tech that will hurt us.

(Some of you may think I have something about Dario. You’re very correct. I hear some very concerning things about him and his whole camp. I wrote our “Why you should read: Fyodor Dostoevsky” in a large part directly for him. Read that piece w/ this context).

The people from the XRisk camp are like people who sell meteor insurance while the floor is collapsing.

The Policy Parrots & Ethics Panels

Corporate ethics teams and regulatory panels mostly generate paperwork. Risk matrices. Governance frameworks. Whitepapers about “alignment” that don’t address control, deployment, or actual misuse.

They do not challenge:

• Core incentive structures.

• Economic capture.

• Strategic centralization.

The Pundits of Balance

By treating the AI thesis and its critics as symmetrical “sides,” these commentators flatten the conversation. They erase the asymmetries — of power, access, funding, and architectural control — and insist on tone instead of substance.

They think they’re moderating. They’re muddying.

The Superficial Hype-Chasers

Finally, the bulk of media and casual critics focus on headline absurdities — prompt injection hacks, goofy chatbots, viral PR demos — while ignoring the core vulnerabilities. These are your parrots that repeat headlines (even when false, such as w/ New York Times and their claims around o1’s medical diagnostic abilities), add very little to the conversation, and are in it to make a quick buck.

Their goal isn’t accuracy. It’s impressions.

In other words, they are not serious people.

Setup for Collapse — or Correction

None of these actors — safetyists, parrots, pundits, panels — are forcing a structural reevaluation of the thesis. They’re not challenging how risk is priced. They’re not shifting how value is measured. They’re not breaking the logic of build-at-all-costs.

They’re ambient noise.

And that leaves the thesis overexposed, under-hedged, and headed straight for correction. It’s a poorly diversified portfolio on leverage.

And when the liquidation starts, there will be a huge crop of winners. Those who built tools, systems, and narratives around what the thesis couldn’t see. People who built for a blue ocean, and avoided fighting bloody wars of attrition in spaces oversaturated by the thesis worshipers.

So, the AI thesis is a bloated, overconfident giant, and its “critics” are mostly court jesters or well-meaning fools flailing in the dark. A satisfyingly bleak picture, perhaps, but strategically incomplete. Because, as the dialectic dictates, the very overreach of the thesis guarantees the emergence of a genuine antithesis.

Not more performative hand-wringing.

We’re talking about a valid antithesis — a structurally potent counterforce.

4. What a Real Antithesis Looks Like

What does this real deal look like?

A valid antithesis is active. It doesn’t just point out problems; it builds, or enables the building of, actual alternatives. It philosophizes with a hammer. It’s not just a different opinion; it’s a different operational logic.

Constraint as a Design Principle (Not a Bug):

The current thesis worships boundless scale and speed. A true antithesis champions intentional constraint. This isn’t about being anti-progress; it’s about being pro-precision, pro-efficacy, pro-sustainability.

Think:

• AI systems designed for specific, narrow tasks with verifiable performance and clear boundaries, not nebulous AGI ambitions.

• Models optimized for minimal viable data, minimal energy consumption, and maximal interpretability.

• Systems where “less is more” isn’t a limitation, but a feature, a mark of superior design and focused utility.

This is about building surgical tools, not just bigger hammers.

Human Sovereignty & Agency as Core Infrastructure (Not an Afterthought):

The thesis trends towards opaque automation and the subtle (or not-so-subtle) erosion of human agency. A valid antithesis builds tools and platforms that reinforce human control, creativity, and ownership.

Think:

• Verifiable credentialing for human-created content.

• Tools that allow individuals to definitively opt out their data from training sets or control its use with granular permissions.

• Systems designed for human-AI collaboration where the human is the non-negotiable principal, not a wetware component to be optimized out.

This is about empowering individuals against the encroaching tide of algorithmic homogenization and data feudalism.

Architectural Pluralism & Decentralization (Against the Monoculture):

The AI thesis is rapidly coalescing around a few centralized foundation model providers, creating a dangerous monoculture and single points of failure. A real antithesis fosters architectural diversity and resilience.

Think

• Development of smaller, specialized, open-source models that can be run locally or in federated networks.

• Protocols for interoperability between different AI systems, preventing vendor lock-in.

• Investment in alternative hardware and compute paradigms that aren’t dependent on a handful of chip manufacturers. This is about building a robust, distributed ecosystem, not a fragile empire built on a few chokepoints.

Narrative Detonation & Value Re-Calibration (Beyond Corporate Branding):

The thesis is propped up by a massive narrative machine — corporate PR, fawning media, and the self-serving mythologies of “visionary” CEOs. A valid antithesis actively works to dismantle these narratives and propose alternative value systems.

Think:

• Rigorous, independent auditing and benchmarking of AI claims, cutting through the marketing hype.

• Platforms that elevate human artistry, critical thinking, and deep work as inherently valuable, distinct from AI-generated outputs. Exposing the true costs (social, environmental, cognitive) of the current AI trajectory and championing metrics of progress beyond mere computational power or shareholder value.

This is about reclaiming the definition of “value” itself from the clutches of the techno-capitalist thesis.

These pillars aren’t just abstract ideals. They are the emerging fault lines where the current AI thesis is most vulnerable. They represent the unmet needs, the suppressed desires, and the ignored risks that the dominant paradigm has created.

A valid antithesis, therefore, isn’t just about critique. It’s about identifying these points of structural weakness in the thesis and then strategically building or investing in the solutions, systems, and narratives that address them. It’s not anti-AI; it’s anti-stupid-AI. Anti-brittle-AI. Anti-dystopian-AI.

These aren’t fringe ideas percolating in academic basements. They are the early, often messy, blueprints of the next stable state — the synthesis. A state where we address some of these issues, and in doing so bring more people into AI. Creating a new thesis, one at a higher baseline of progress.

This might seem like a bold claim. Let’s understand why the synthesis is so important to progress.

5. Why the Synthesis Matters

The thesis refines.

The antithesis rejects.

But the synthesis? It builds the future people will actually live in.

We don’t reach that by building better demos. We reach that by changing the terrain — who participates, how they participate, and what gets built for them.

Synthesis Creates New Value Streams

The thesis phase has already flooded the zone.
You can see it at any Demo Day:

• 12 variations of the same Slack-integrated copilot. Or the newest VC Obsessios: AI Matchmakers. How many of these are y’all going to fund before you get bored and remember that somewhere in your value prop you talk about next-gen solutions that build the future?

• 30 founders with the same pitch deck structure. How many “Cursor for X” do we need?

• A culture of wrappers, not reinvention.

That’s what happens when you’re trapped inside a single idea.
Everything gets optimized. Nothing gets expanded.

The synthesis breaks that trap.

It creates:

• New surfaces for value: sovereignty, interpretability, human-first workflows.

• New buyers: skeptics, legacy operators, institutions with trust constraints.

• New metrics: resilience, control, cultural alignment — not just model output.

Synthesis and the Curve of Tech Adoption

Every major technological wave follows the same diffusion curve:

1. Built by experts

2. Used by generalists

3. Owned by everyone

Confused? Time for a History Lesson in Computing.

Stage 1: The Domain of Experts (1940s–1970s)

In the early decades, computers were:

• Massive, fragile, and expensive.

• Operated by physicists, engineers, and specialists.

• Built in labs, military programs, or research centers.

They were inaccessible — not just technically, but culturally. You didn’t “use” computers. You built them, maintained them, debugged them.

The tools were raw. The workflows were alien.
And almost no one knew how to make them useful to non-experts.

Stage 2: The Domain of Generalists (1980s–2000s)

This is where the inflection began.

• PCs entered offices, schools, and homes.

• Software abstracted away the need to “speak machine.”

• The personal computer, productivity suite, and GUI stack turned experts into builders — and builders into product designers.

Still: to truly leverage the technology, you needed to be semi-technical.
Early adopters. Developers. The “computer literate.”

It was usable — but not yet default.

Stage 3: The Domain of Everyone (2000s–Today)

Today, the average user:

• Doesn’t care what a processor is.

• Doesn’t know how file systems work.

• Doesn’t read manuals.

But they can navigate phones, apps, cloud systems, and workflows with near-fluency — because the infrastructure shifted.

UX, defaults, guardrails, and operating models matured to support non-technical, skeptical, or distracted users at massive scale.

That’s what synthesis does.
It doesn’t just refine the tech — it architects universal compatibility.

Now Map That to AI

This isn’t just a quaint historical analogy. The AI trajectory is running the same gauntlet, facing the same evolutionary pressures. And understanding where we are on this map is critical to seeing why the current AI Thesis, for all its power, is fundamentally incomplete.

AI Stage 1: The Domain of Experts (The Alchemists — e.g., early pioneers of neural networks, symbolic AI, knowledge-based systems).

• This was AI in its academic crucible. Models built from scratch, algorithms understood by a select few. The “users” were the creators. The output was often proof-of-concept, not product. The barriers were immense, the tools arcane.

AI Stage 2: The Domain of Generalists & Early Adopters (The Current Frontier — e.g., developers leveraging TensorFlow/PyTorch, prompt engineers using GPT-4 via APIs, businesses deploying off-the-shelf ML solutions).

• This is where we largely stand today. Foundation models, APIs, and low-code platforms have dramatically expanded access. A new class of “AI generalists” can now build and deploy sophisticated applications without needing to architect a neural network from first principles. ChatGPT and its kin have even brought a semblance of AI interaction to a wider public.

But let’s be clear: this is still Stage 2. True fluency requires technical understanding, an ability to navigate complex interfaces, or at least a significant investment in learning new interaction paradigms. The “average user” is dabbling, perhaps impressed, but often confused, wary, or rightly concerned about implications the current Thesis glosses over — bias, job security, privacy, the sheer inscrutability of these powerful tools. We are far from “owned by everyone.”

AI Stage 3: The Domain of Everyone (The Unwritten Future — AI as truly ubiquitous, trusted, and seamlessly integrated utility).

• This is the promised land the current AI Thesis aspires to but is structurally incapable of reaching on its own. Why? Because a Thesis built on breakneck speed, opaque systems, winner-take-all centralization, and a dismissive attitude towards legitimate societal fears cannot architect universal compatibility. It inherently creates friction, distrust, and exclusion.

It’s about making AI functional for the forgotten 80%.

That’s how every transformative technology hits scale — not through more performance, but through deeper fit.

“NVIDIA should back real-world initiatives like these that make AI matter to people, turning them into compelling narratives stamped with “Powered by NVIDIA.” It’s a strategic brand play with mutual upside.

…

Want to go mainstream?

Imagine future keynotes — instead of talking abstractly about tokens, NVIDIA could show before-and-after videos of the lines at the local DMV. Talk about great fodder for viral social media videos!

…Yes, the examples are hypothetical, but the core issue is real: Jensen talks like everyone already uses and values AI. But most people don’t. The best way to reach the public is to make AI real in their daily lives”

—A particularly brilliant section from the exceptional Austin Lyons.

And whoever builds that synthesis infrastructure? They don’t win the debate.

They win the adoption curve.

Final Notes on the Synthesis

The Synthesis doesn’t just make AI “better” for the experts or the adepts. It fundamentally re-architects AI’s relationship with society, addressing the core anxieties and practical barriers that prevent mass adoption. It’s how AI transitions from being a powerful but often unsettling force used by a relative few, to becoming a trusted, integrated, and indispensable utility for the many.

You want to avoid friction in adoption? To get more people comfortable building on and using your product? You can do it by pushing the “use us or perish” agenda that AI has been pushing. But there’s only so far threats take you. You have much easier time getting people to pay you when they like you.

That’s what the antithesis does. It shows the ones you alienate that you’re paying attention. It starts to bring them into the fold, leading to the new markets and Cambrian explosion that are associated with the synthesis stage.

You want Jevon’s Paradox to work for your revenue? Get to synthesis. And that’s not something you can get to by skipping the antithesis phase. Ignore the antithesis, and you end up with a Habsburg jaw.

6. Conclusion

We’ve jumped across a lot of ideas, so let me summarize the ideas in one thread-

1. Our current AI industry is “thesis heavy”: it loves it’s own assumptions too much. This leads to mania, inflated valuations, and lots of copycat startups.

2. While this brings in a lot of attention, it’s also alienating large numbers of people, who are actively looking for reasons to reject AI.

3. Some of the reasons are based on legit concerns that AI doesn’t adequately address. The arrogance of the thesis and the lack of addressing create further polarization.

4. This sets a great opportunity for a meaningful antithesis — solutions that specifically position against the weakness of the thesis; these will start to draw in the aforementioned alienated people.

5. The antithesis will allow us to create new markets and build systems that address the limitations of the current systems.

6. This leads to the synthesis, where we get the Cambrian explosion and mass adoption of technology.

Betting on the antithesis isn’t betting on negativity. It’s betting on progress. It’s betting on the intelligence within people that our current systems have overlooked.

The ROI of including more people within the economic system has always justified itself, many times over (more grunts to add value for shareholders).

And isn’t that ultimately a vision worth getting behind?

Thank you for being here, and I hope you have a wonderful day.

Let’s look for the One Piece,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. : https://rb.gy/zn1aiu

My YouTube: https://rb.gy/88iwdd

Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y

My Instagram: https://rb.gy/gmvuy9

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

In December 2025, NVIDIA paid 20 billion dollars to acqu-hire Groq, explicitly buying its SRAM-heavy inference architecture to patch their own decoding bottleneck. Earlier this month, Cerebras (another company with a similar thesis) executed a massive initial public offering, pricing at 185 dollars a share to achieve a 66 billion dollar market capitalization. They went to market backed by a 20-billion-dollar Capacity-as-a-Service contract with OpenAI.

This is far from the only major event in the AI hardware space. Billions of dollars (trillions if you think about the plans to build Fabs) are being thrown around to address the fundamental limitations of GPUs when it comes to serving large language models at scale.

Cerebras represents one such bet against GPUs. Where GPUs spend their entire power and clock budget waiting for model weights to travel across a shared physical bus from distant memory modules, Cerebras built a chip the size of a dinner plate to physically short-circuit this memory bandwidth ceiling.

By keeping 44 gigabytes of elite SRAM permanently resident next to 900,000 active compute cores, they eliminate the off-chip round trip entirely. To drag this machine to market, they had to solve five brutal physics problems — from reticle limits to coefficient of thermal expansion mismatches — that left a forty-year graveyard of bankrupt semiconductor startups behind them.

Cerebras’s massive IPO is a clear indication of the AI industry recognizing one of it’s biggest fault lines and scrambling to find the solutions for it. However, to understand the viability of this solution, we must understand the current issue with AI Inference, why Wafer scale has historically failed, and what Cerebras is trying to do differently. This deep dive will do that by deconstructing the physical reality of wafer-scale processing against the brutal unit economics of real-world deployment. I wrote it for builders, capital allocators, and technical leaders who need to separate structural hardware moats from venture-backed marketing and ultimately make decisions on how to best engage/benefit from this massive shift in the market.

To do so, in this article, we cover:

• The Bandwidth Wall: Why inference is a memory problem, and why adding GPU compute wastes silicon.

• The Physics: The five physical constraints of wafer-scale manufacturing that destroyed conventional chip designs over the last forty years.

• The Silicon: How Cerebras bypassed the reticle limit, power droop, and thermal expansion to manufacture a single continuous computational surface.

• The Memory Math: The exact first-principles calculation behind their memory bandwidth advantage and bytes-per-FLOP scaling.

• The Production Reality: The severe operational costs of unique wafer compilations, a barren software ecosystem, and unproven physical failure modes.

• The Competitive Horizon: How NVIDIA’s 20-billion-dollar Groq acquisition, Google’s bifurcated TPU strategy, and the SRAM density wall dictate the survival of the architecture.

PS: I have not taken any compensation from Cerebras or anyone else for this deep dive (we have a strict independence policy). The timing of this deep dive close to their IPO is a coincidence. I started the research conversation with their team months ago, it just took me a while to really research and understand the space + I was busy with the paperwork for the o1 visa.

Executive Highlights (tl;dr of the article)

• Cerebras is a bet on one claim: LLM inference is limited by memory bandwidth, not compute. During decode, GPUs have to read model weights again for each new token. The math units sit idle because the weights can’t arrive fast enough. More FLOPs don’t fix that. More bandwidth does.

• Cerebras attacks this by moving memory next to compute. Instead of cutting a wafer into separate chips, it keeps the full wafer intact and stitches the regions together. The result is a dinner-plate-sized chip with 900,000 active cores, 44 GB of on-chip SRAM, and 21 PB/s of aggregate memory bandwidth.

• This was hard because wafer-scale computing breaks normal chip assumptions. A full wafer runs into five problems: lithography reticle limits, manufacturing defects, power delivery, cooling, and thermal expansion. Cerebras solves them with reticle stitching, redundant cores, compile-time routing, vertical power delivery, liquid cooling, and a connector that lets the wafer move without cracking the package.

• The key comparison is bytes per FLOP. WSE-3 can deliver about 0.168 bytes per FLOP. H100 delivers about 0.0034 bytes per FLOP. That gives Cerebras roughly a 50x advantage in feeding data to math units during bandwidth-bound decode. This is why its Llama 3 70B single-stream output speed can reach around 2,100 tokens/s, compared with roughly 30–50 tokens/s on a single H100.

• The speed comes with a capacity problem. WSE-3 has only 44 GB of SRAM. A 70B FP16 model needs about 140 GB of weight storage, so Cerebras needs four CS-3 systems to host one full-precision instance. GPUs are slower per stream, but their HBM capacity makes large-model hosting much cheaper and simpler.

• Cerebras changes serving economics. GPUs need batching because many users must share the same expensive HBM weight reads. Cerebras can run fast at Batch-1 because weights are already stored in local SRAM. But when a model spans multiple wafers, Cerebras has to keep the pipeline full with several users at once or most of the cluster sits idle.

• The training story is weaker. Cerebras originally targeted training by streaming weights from external MemoryX while keeping activations on wafer. The architecture made sense on paper, but frontier labs still train on NVIDIA GPUs, Google TPUs, or custom hyperscaler chips. The software ecosystem mattered more than the hardware elegance. CUDA remains the moat. Annoying little goblin. Very real.

• The business case is still unproven. Cerebras has a strong physics story, but revenue concentration, the forward-looking OpenAI capacity deal, and weak baseline unit economics make the IPO story fragile. The OpenAI contract only works if Cerebras can resell idle capacity, add high-margin services, and improve system efficiency over time.

• The software risk is serious. Every wafer has a unique dead-core map, so binaries are tied to specific machines. Custom compilation can take hours. Novel models may fail to compile. The CSL developer pool is tiny. The normal GPU stack — vLLM, FlashAttention, Triton, CUTLASS, xformers, HuggingFace workflows — does not carry over cleanly.

• The reliability story also needs proof. A 23 kW wafer has little thermal margin. Cooling failures, voltage regulator issues, local brownouts, silent data corruption, connector fatigue, and post-deployment core failures are all real fleet-level risks. Cerebras has shown the machine can work. It has not yet shown enough public evidence that it can scale cleanly under hyperscaler operating conditions.

• The final question is timing. If GPU memory systems improve fast enough, Cerebras becomes a brilliant transitional architecture. If single-stream latency becomes central to agentic AI and HBM remains the bottleneck, Cerebras sits exactly where GPUs break. The engineering is real. The advantage is measurable. The company still has to prove the market window, software stack, manufacturing scale, and unit economics all arrive before NVIDIA crushes the oxygen out of the room.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

Part 1: Background

Why Does LLM Decode Hit a Memory Bandwidth Wall on GPUs?

We’ve covered this idea a lot, so I’m going to breeze through this one here. If you want to understand this gap more (and how the Roofline models work), check out:

1. Our analysis of how much a token costs.

2. Our deep dive into Liquid AI and what makes them special for edge AI.

3. Our deep dive into AI inference and why its emerging as a separate category over here

An H100 executes 989 trillion floating-point operations per second. During the decode phase of large language model inference, the chip uses less than 1% of that capacity. The compute units sit idle waiting for weights to arrive from memory. Why does this happen?

Inference consists of two distinct workloads. The prefill phase processes the input prompt in parallel, generating the high arithmetic intensity (The ratio of useful math to memory traffic) that keeps a GPU busy. The decode phase generates the output sequentially. To produce one token, the chip must read essentially every weight in the model from memory, multiply those weights once, and move on. The arithmetic intensity plummets to roughly 1 FLOP per byte. At 30Gs a chip and these abysmal numbers, you’re better off paying Scale AI to whip 3rd world orphans to run the numbers manually. You might even get a Mr Beast video out of this.

This exposes the hardware’s roofline. The H100 balances compute and memory bandwidth at roughly 295 FLOPs per byte. Any workload running below that ratio is bandwidth-bound. Because decode runs at 1 FLOP per byte, it hits the memory ceiling at 1/200th of the chip’s theoretical peak. You could double the H100’s compute capacity and decode speed wouldn’t increase by a single token (this is the reason behind the memory wall we broke down here). The physical pipe between memory and compute dictates the limit.

NVIDIA’s current approach is to build a wider pipe. The H100 moves 3.35 TB/s, and the B200 moves roughly 8 TB/s. But each hardware generation also scales up compute, which pushes the required balance point even higher while decode stays pinned at 1 FLOP per byte. Every additional transistor dedicated to arithmetic is wasted silicon during decode.

So what can we do here? GPUs store weights in memory stacks located millimeters away from the compute cores, connected through shared controllers. The physical alternative is to embed the memory directly adjacent to every compute core. When memory is part of the core itself, adding cores automatically scales memory bandwidth and eliminates the off-chip round trip entirely.

Sounds obvious enough, we run into a slight complication here: building a unified compute-memory architecture at scale has killed every hardware company that attempted it over the last forty years.

Let’s understand why.

Why Did Wafer-Scale Integration Fail for Forty Years?

If memory bandwidth is the bottleneck, the obvious fix is to build one massive chip where memory sits physically adjacent to every compute core. You eliminate off-chip communication entirely by weaving compute and memory across a full silicon wafer. Engineers tried this for forty years. Tried being the operative word here.

• Gene Amdahl raised 230 million dollars in 1980 to build an IBM-compatible mainframe on a 2.5-inch wafer for Trilogy Systems. The fabrication technology of the era could not hit the defect densities required for a full wafer circuit. Interconnects delaminated, a prototype shorted and glowed red during a demonstration, and the company dissolved.

• ETA Systems built a liquid-nitrogen-cooled wafer-scale supercomputer in 1983. The silicon worked. The ecosystem failed. Customers could not maintain the specialized nitrogen plumbing, and software development lagged behind the custom hardware.

• Anamartic shipped a wafer-scale solid-state disk in 1989. The engineering functioned in production. Then commodity DRAM prices collapsed. Customers achieved the same performance by stringing together cheap DRAM modules, which destroyed the economic justification for Anamartic’s manufacturing premium.

• Tesla proved modern wafer-scale packaging works with its Dojo D2 chip. They ran real training workloads on full-wafer silicon in production racks. But in April 2026, Elon Musk shut Dojo down. The chip functioned, but the narrow workload could not justify the cost of maintaining a custom software stack to compete against NVIDIA’s entrenched CUDA ecosystem.

So why does our aspiration to engage with Wafer scales keep getting Yamcha’d??

What Are the Five Physical Constraints of Wafer-Scale Silicon?

Wafer-scale computing is structurally different from conventional chip design. Playing at a bigger size means it hits five distinct physical and mathematical limits.

1. The Reticle Limit The Reticle Limit is a fundamental problem for wafer-scale integration because the lithography process can only expose a maximum area of about 858 square millimeters per shot, meaning a full 300-millimeter wafer requires 54 separate reticle fields. For the wafer to function as a single, unified chip, electrical connections must be routed across the boundaries between these fields. However, these boundaries are the “scribe lines,” which are strictly designed for physical cutting (dicing) and not for carrying electrical signals. This means the standard process creates discontinuities across the wafer where a unified electrical path is required, making a single functional chip mathematically impossible without a specialized technique like reticle stitching.

2. The Defect-Rate Cliff Mature semiconductor manufacturing produces roughly 0.001 killer defects per square millimeter. For a standard 814-square-millimeter die, the Poisson probability allows for 60 to 80 percent yields. Scale that math to a 46,225-square-millimeter wafer, and the probability of a completely defect-free wafer drops to 10^-20. In conventional design, a single defect kills the entire die. Without a mechanism to dynamically route around dead silicon, a functioning wafer-scale chip is mathematically impossible.

3. Power Delivery A wafer-scale chip at full load draws roughly 28,750 amps at sub-volt levels. You cannot push that much current horizontally across 300 millimeters of silicon b/c the voltage drops as it travels — a physical phenomenon known as IR drop. By the time the current reaches the interior of the wafer, the voltage sags too low for the cores to operate. You cannot power a wafer from its edges like a standard chip.

4. Cooling A modern wafer-scale chip dissipates roughly 23 kilowatts across 46,225 square millimeters. That creates a heat flux of 0.5 watts per square millimeter — five times the thermal density of an H100. Conventional air cooling cannot move this much heat. Standard liquid cold plates also fail because the heat spreads uniformly across an area the size of a dinner plate rather than concentrating in a small GPU footprint.

5. Thermal Expansion Silicon has a coefficient of thermal expansion (CTE) of 2.6 parts per million per Kelvin. Standard printed circuit board (PCB) material expands at 14 to 17 parts per million per Kelvin. Across a 215-millimeter span with a 60-degree Celsius temperature swing, the silicon expands 33 microns while the PCB expands roughly 200 microns. Every thermal cycle, the board and the wafer try to slide against each other by nearly a fifth of a millimeter. This relative motion shears standard solder joints apart on the first cycle. It is a physical hard stop for standard packaging.

So Why Does Cerebras Exist?

Given all these challenges we’ve been discussing, you might be wondering what kind of people decide to dedicate themselves to wafer-scale chips in our present age. If the generations of Arsenal fans have taught us anything, humanity’s delusion and hubris are inexhaustible commodities. So the founders of Cerebras looked at all the challenges and said “Nah, I’d win”.

Cerebras formed in 2016 around the core team from SeaMicro, a low-power server startup acquired and later shut down by AMD. They realized early that the industry’s default scaling method — adding more boxes and connecting them — was hitting a power and cost wall. The interconnect itself was becoming the bottleneck.

Now that we have the background, let’s look at how our Cere-bros have set out to solve the reticle limit, defect tolerance, power delivery, cooling, and thermal expansion simultaneously on the same wafer at volume.

Part 2: How Cerebras Builds a Chip the Size of a Dinner Plate

Building a commercially viable chip required solving all 5 problems together. Let’s see how this is accomplished.

1. The Reticle Limit and the Dicing Step

Lithography optics can only print 858 square millimeters of silicon per exposure. Foundries normally print a pattern 84 times across a wafer and use a saw to cut the silicon into 84 independent chips. Cerebras alters this process. They use standard equipment to print the 84 patterns, but then add lithography steps to print over one million microscopic metal wires directly across the “scribe lines” where the saw would normally cut.

Cerebras simply skips the dicing step. This is the foundation of the architecture. By leaving the wafer intact, they preserve those one million wires. This transforms 84 separate chips into a single continuous computational surface. Data never leaves the silicon to travel between regions. This eliminates the massive latency, bandwidth bottlenecks, and power penalties required to send signals across standard copper cables or optical switches (this is one of the reasons that photonics has popped off too; they use photons to do math, which is a very cool idea, will do a deep dive into the viability of it sometime).

2. The Defect-Rate Cliff and Redundant Cores

Previous wafer-scale attempts failed because they built monolithic circuits. A monolithic circuit acts like a single mechanical watch; if one microscopic gear has a manufacturing defect, the entire watch stops. Mature fabrication guarantees roughly 46 defects per wafer. Under standard design rules, a functioning wafer-scale chip is mathematically impossible.

Cerebras abandoned the monolithic approach. They built the wafer as a grid of roughly 970,000 completely independent computational cores. This way, if a manufacturing defect strikes a core, only that specific core dies. The surrounding cores continue operating. The Cerebras compiler maps the location of dead cores and routes data around them at compile time, exactly like a GPS routing traffic around a closed road. Cerebras intentionally budgets for 70,000 disabled cores to absorb the 46 expected manufacturing defects (budgeting several orders of magnitude higher is smart given the complexity of their build and the relative unfamiliarity the ecosystem has with their stuff). This architecture converts a fatal yield problem into a manageable routing exercise, achieving 93 percent active silicon.

3. Power Delivery and the Horizontal Bottleneck

A wafer-scale chip draws 28,750 amps at sub-volt levels under full load. Conventional chips route power horizontally from the package edges inward. However, metal wiring naturally resists electrical current. This resistance causes the voltage to drop as it travels — a phenomenon known as IR drop. On a standard chip, the horizontal distance is short and the voltage drop is harmless. On a massive 300-millimeter wafer, pushing power horizontally from the edges would starve the interior cores since they would receive too little voltage to turn on.

Cerebras bypassed the horizontal bottleneck entirely by delivering power vertically. A dedicated circuit board sits directly behind the wafer. It pushes current perpendicularly into the back of the silicon through hundreds of local voltage regulators. Current travels only millimeters to reach any given core. Every region receives direct, clean power regardless of its location on the wafer.

4 & 5. Thermal Expansion and the Floating Connector

The WSE-3 generates 23 kilowatts of heat. Modern liquid cold plates can easily extract this heat. The actual barrier is mechanical expansion. Silicon and printed circuit boards (PCBs) expand at different rates when heated. Across a massive 215-millimeter span, this difference compounds. During a normal thermal cycle, the circuit board expands nearly a fifth of a millimeter further than the silicon wafer. If you soldered the wafer directly to the board, this sliding motion would shear the metal joints completely off on the first cycle.

Cerebras solved this by refusing to solder the wafer. The wafer floats. A proprietary mechanical connector sits between the bare silicon and the underlying circuit board. This connector acts as a microscopic suspension system. It bends, flexes, and compresses to absorb the differing thermal expansion while maintaining continuous electrical contact for power and data.

What Is Inside Each Core?

With the wafer-scale physics solved, Cerebras packed the silicon with 900,000 active processor tiles. Each core is tiny — about 0.05 square millimeters. But the most important architectural decisions aren’t about the size of the core; they are about what Cerebras stripped away from standard GPU design.

How Cerebras Handles Memory

On a standard GPU, memory is a centralized pool. Thousands of compute cores pull data from off-chip High Bandwidth Memory (HBM) or shared internal caches. This creates a traffic jam since cores fight for bandwidth and wait in line for their data to arrive through complex memory controllers.

Cerebras abandons shared memory entirely. Inside each of the 900,000 cores sits 48 kilobytes of dedicated Static Random-Access Memory (SRAM). Cores do not share this memory, and no other core contends for it. Because memory physically touches the compute units, aggregate bandwidth scales linearly with every core added. This is how the WSE-3 hits 21 petabytes per second of memory bandwidth — thousands of times more than a GPU. The compute units never sit idle waiting for weights to arrive.

There is an important tradeoff here: 48 kilobytes is incredibly small. A GPU can load a massive, gigabyte-sized matrix into shared memory for all cores to access simultaneously, but Cerebras cannot. To make this work, the compiler must surgically slice neural networks into microscopic pieces and distribute the weights across hundreds of thousands of independent memory vaults. There is a lot of general life wisdom in how the hardware team successfully bypassed a massive problem (the physical memory wall) by offloading 100% of the psychological trauma to the software engineers. Bois, take notes.

How Cerebras Handles the Datapath

Modern GPUs use “scalar” units for standard math and specialized “tensor cores” built specifically to smash matrices together. Constantly moving data between these separate units consumes time and power.

Cerebras, on the other hand, treats tensors as first-class operands within the instruction set. There is no separate tensor core. Each core executes eight 16-bit floating-point operations per cycle through a simple six-stage pipeline. The cores run at approximately 870 megahertz, which is roughly half the clock speed of an H100. The architecture happily trades the sheer speed of an individual core for the overwhelming, synchronized parallelism of 900,000 cores working simultaneously.

The Fabric: How Cerebras Kills the Network-on-Chip

To manage communication, standard processors rely on a “Network-on-Chip” — a dedicated layer of hardware routers and buses built into the silicon just to manage traffic. This creates a problem: If data needs to leave the chip entirely to reach another GPU, the toll skyrockets. It must exit the die, cross a circuit board, convert into optical signals, hit an external switch, and travel down an InfiniBand or NVLink cable. Every boundary adds latency, protocol overhead, and a bandwidth bottleneck.

Cerebras eliminated the Network-on-Chip entirely. Because the wafer is never diced, data never has to leave the silicon. Every core has exactly five physical connection ports. Four connect directly to its immediate neighbors, and the fifth connects to its local memory. The two-dimensional grid of compute cores is the communication layer. Data travels seamlessly across the silicon as 32-bit “wavelets,” carrying tiny instruction tags that tell the receiving core exactly what math to perform as the data passes through.

Think about how fast this makes your operations. A single hop between adjacent cores takes roughly one nanosecond. Crossing the entire 215-millimeter wafer — roughly 1,000 hops — takes about one microsecond. That worst-case latency on a Cerebras wafer matches the best-case latency between two adjacent GPUs connected by NVLink in a server rack. Because communication requires only microscopic metal wires rather than optical cables, the internal fabric provides 214 petabits per second of bandwidth.

However, a two-dimensional mesh has a structural weakness: distance equals latency, and intersecting traffic causes congestion. If cores on opposite sides of the wafer constantly talk to each other, the center of the grid jams. Cerebras handles this entirely in software. The compiler is forced to be aggressively intelligent, mapping the neural network spatially so that layers requiring heavy communication are placed on physically adjacent silicon tiles.

All of this is incredibly sophisticated, and we should really take a step back and appreciate the engineering that makes this possible. As I mentioned earlier, it took me about a 1–2 months of on and off research to even get to a functional understanding of Cerebras and what it does. However, just as Pep’s ability to overthink tactics against Burnley can sometimes cause him to bench Rodri to play Gundogan deeper (???? it’s been years, and I’m still so fucking confused by that shit), intricate engineering can be a challenge to scale well. And this might have already started to show up with Cerebras.

Three Generations and the SRAM Density Wall

The physical limits of this architecture are beginning to show in its generational scaling.

The first WSE shipped in 2019 with 400,000 cores, proving that the wafer-scale physics problems had commercial solutions. The WSE-2 shipped in 2021, proving the architecture could scale by doubling both core count and memory capacity.

The WSE-3 shipped in 2024 and hit a wall. Compared to the previous generation, core count grew only 6 percent and SRAM capacity grew only 10 percent (tbf, raw compute capacity exploded 17-fold to 125 petaflops so maybe I’m tripping out over nothing).

This imbalance exposes the “SRAM density wall” plaguing the entire semiconductor industry. The WSE-3 uses TSMC’s 5-nanometer process. As fabrication nodes shrink, standard logic transistors scale down beautifully. SRAM memory cells do not. They require six carefully balanced transistors just to hold a single bit of data, and they hit a physical floor where shrinking them further compromises their electrical stability. Wall Street is currently modeling software-like margin expansions through 2030 for these hardware companies. Its going to very interesting to see how their spreadsheet merchants try to adjust their discounted cash flow models for quantum tunneling.

Because Cerebras relies entirely on embedded SRAM rather than external memory modules, they are highly exposed to this limitation. Forced to choose how to spend their 5-nanometer silicon budget, Cerebras couldn’t double the memory or the core count. Instead, they spent their transistor budget inside the core, widening the compute datapaths to massively increase mathematical throughput. I’m not a semis guys so I’m not going to make any strong claims about this, but this did seem like a possible trend to monitor. Now I might be turning windmills into giants here, but I think that any analyst trying to price the future of this company should be looking at how the Cerebros (and the rest of this industry) tackle this problem and its various siblings.

For now, let’s get back to our analysis of the present by understanding the insane performance numbers that have underpinned Cerebras’s amazing IPO.

Part 3: How Good is Cerebras Really?

The architectural argument hinges on a single claim: moving memory directly adjacent to compute cores eliminates the bandwidth bottleneck. To evaluate this honestly, the claims must be translated into first-principles mathematics and measured against observed production benchmarks.

Deriving the 21 PB/s Number

Cerebras’s headline specification — 21 petabits per second (PB/s) of memory bandwidth — presents an apparent 6,272-fold advantage over an NVIDIA H100’s 3.35 terabytes per second (TB/s).

Normally, when we see such large deltas, it’s fair to have some skepticism. However, this aggregate figure is real, but it measures a completely different physical architecture than a GPU’s shared memory system.

Each of the 900,000 active cores on the WSE-3 has 48 kilobytes (KB) of private, dedicated local SRAM. Operating at a clock speed of approximately 870 megahertz (MHz) with a datapath width of 24 bytes per read, a single core can pull data from its private SRAM at roughly 23 gigabytes per second (GB/s). Multiplying this private, contention-free performance across all 900,000 active ports running in parallel yields the aggregate figure:

900,000 cores multiplied by 23.3 GB/s equals approximately 21 PB/s

This massive number represents the sum total of 900,000 independent, simultaneous memory transfers. It does not describe a single centralized memory pool flowing through a massive wire.

By contrast, the H100’s 3.35 TB/s describes a single, shared ceiling. All 132 streaming multiprocessors (SMs) must compete for access to the same High Bandwidth Memory (HBM) modules through a shared memory bus. When one SM reads a model weight, it consumes a portion of that fixed bandwidth pool, creating structural contention that reduces effective per-core throughput under real workloads. On the WSE-3, Core #47 reading its local SRAM has zero physical impact on Core #48’s ability to read its own memory at full speed. Because bandwidth is simply the measurement of how much data can flow from memory to the compute units in a given second, giving every core its own memory port means the system’s total bandwidth is just the sum of all cores operating at once. This allows the architecture’s data delivery to scale linearly without a shared physical bus to bottleneck traffic.

But given that these are different operations, how do we make an apples-to-apples comparison for this operation?

The Balance Metric: Bytes per FLOP

To understand how effectively these chips can actually use their raw performance, we have to evaluate their architectural balance. We do this by dividing memory bandwidth (the numerator) by peak floating-point operations per second (the denominator). The numerator tells us how fast the chip can fetch data from memory, while the denominator tells us how fast the chip’s arithmetic units can process that data. The resulting bytes-per-FLOP ratio defines exactly how many bytes of data the hardware can deliver for every single mathematical operation it performs, revealing whether the chip is starving for data.

• WSE-3: 21 PB/s divided by 125 PFLOPS peak compute equals 0.168 bytes per FLOP.

• H100: 3.35 TB/s divided by 989 TFLOPS peak compute equals 0.0034 bytes per FLOP.

This reveals a 50-fold structural advantage for Cerebras. For every floating-point operation executed, the Cerebras architecture can deliver 50 times more data to its arithmetic units than an H100.

Sheesh. Do you think now would be a good time to apply for a job with them? To my low-tech Indian mom, the terms “I convince tech companies to invest in open source research” and “I’m a glorified beggar” are dangerously similar.

During the sequential decode phase of an LLM, the chip is starved for data because it only performs roughly 1 FLOP of math for every byte of weights it reads. The H100’s physical balance point requires a massive 295 FLOPs of math per byte to keep its transistors busy, pinning it at 1/295th of its peak capacity during decode. Because the WSE-3 feeds its units 50 times more data per operation, its balance point sits at roughly 6 FLOPs per byte, allowing it to operate at 1/6th of its compute peak.

Benchmark Verification against Production Reality

This roofline model predicts that a bandwidth-bound model should decode roughly 50 times faster per stream on the WSE-3 than on an H100. Production benchmarks validate this physical reality: a Llama-3 70B parameter model achieves a single-stream decode speed of roughly 2,100 tokens per second on Cerebras, compared to 30 to 50 tokens per second on a single H100.

This speedup confirms that the performance delta is driven by weight-reading latency, not raw math throughput.

Given how important inference is to their company, it makes sense to understand how Cerebras handles inference.

How LLM Inference Physically Operates on Wafer-Scale Silicon

Serving a model like Llama-3 70B at FP16 precision requires roughly 140 gigabytes (GB) of weight storage. Because a single WSE-3 contains only 44 GB of local SRAM, the model cannot fit on one wafer. Cerebras chains four CS-3 systems together, creating 176 GB of aggregate on-chip SRAM, leaving 36 GB of headroom for activations and the Key-Value (KV) cache.

The model’s 80 transformer layer boundaries split evenly across the cluster, allocating roughly 20 layers per wafer. This creates pipeline parallelism at the layer boundaries:

1. MemoryX cold-loads the permanent weight shards into each wafer’s local SRAM, then goes silent.

2. During inference, a single user token enters Wafer 1 and processes sequentially through layers 1 to 20.

3. Wafer 1 transmits the resulting output activations to Wafer 2 via an external Ethernet interconnect.

4. Wafer 2 processes layers 21 to 40 using its own SRAM-resident weights, passing subsequent activations down the chain until the token exits Wafer 4.

At no point during active generation does any core make an off-chip round trip to fetch a model weight. Every weight read is a local, zero-contention SRAM access operating at full speed.

This physical layout upends traditional GPU serving economics. On a GPU, inference providers must stack multiple user requests into large batches (e.g., Batch-8 or Batch-64). This amortizes the bandwidth cost: reading a weight once from HBM allows it to be reused across multiple independent user tokens, inflating the arithmetic intensity to keep the compute units busy. GPU profitability depends on accumulation.

On Cerebras, because the weights are locked inside the local SRAM of each core, there is no shared memory bus to amortize. A single user runs at Batch-1 — commanding the entire local bandwidth of their dedicated core cluster — and still hits the peak 2,100 tokens per second. Bandwidth is not a shared resource; every user achieves maximum per-stream velocity because no concurrent streams compete for memory access.

How Cerebras Resolves the Pipeline Bubble

Pipeline parallelism at Batch-1 introduces a severe structural flaw: if a single token flows sequentially through four distinct wafers, then three out of the four wafers are completely idle at any given millisecond. Left unmanaged, 75 percent of the cluster’s silicon sits dark, completely erasing the wafer-scale performance premium.

Cerebras resolves this by filling the physical pipeline with independent user streams, replicating traditional hardware pipelining at a macro level. In steady state, the system processes four concurrent users simultaneously:

• Wafer 1 processes Layer 1–20 activations for User D.

• Wafer 2 processes Layer 21–40 activations for User C.

• Wafer 3 processes Layer 41–60 activations for User B.

• Wafer 4 processes Layer 61–80 activations for User A.

As Wafer 4 finishes User A’s token and emits it to the user, User A’s next token enters Wafer 1 behind User D. This scheduling optimization fills the pipeline, pushing hardware utilization close to 100 percent without sacrificing individual per-stream speeds. The system achieves efficiency at small, tightly bounded batch sizes (4 to 10 concurrent users) rather than the massive concurrent batches of hundreds required by GPU continuous-batching frameworks.

Cerebras does not compete with NVIDIA on high-density commodity throughput. Its addressable commercial market is strictly bounded by the latency floor: applications where per-stream execution speeds must exceed the physical capabilities of an HBM bus. Everything below that floor belongs exclusively to the GPU ecosystem.

The Training Moat and why Cerebras Pivoted to Inference

The WSE architecture was originally designed for AI training, which requires an inversion of the GPU memory model.

On a GPU, model weights must remain resident in HBM during training because a backward pass requires immediate, sequential access to read weights, calculate activations, compute gradients, and execute weight updates for a given layer. The data and intermediate activations are what stream through the hardware.

On Cerebras, the 44 GB on-chip SRAM is too small to hold giant training states but large enough to hold all activations for a single layer. The paradigm flips: activations are kept resident in SRAM, while model weights stream into the wafer from the external MemoryX subsystem one layer at a time. The wafer loads Layer 1’s weights, computes the output activations, updates the activation state on-chip, discards Layer 1’s weights, and loads Layer 2’s weights from the external pool. For multi-wafer training scales, an external fabric called SwarmX broadcasts these weights across systems and reduces the resulting gradients.

This decoupling means model sizes can scale without a hard physical ceiling, as weights are stored externally and streamed through the wafer. In principle, an arbitrarily large model can be trained on a fixed slice of wafer hardware.

In practice, zero frontier labs utilize Cerebras for training. OpenAI, Anthropic, Google, Meta, and xAI train exclusively on NVIDIA GPUs, Google TPUs, or custom hyperscaler silicon. The weight-streaming training model failed to achieve commercial adoption due to the sheer gravitational pull of the existing software ecosystem. CUDA, PyTorch’s native GPU backends, and an industry-wide library of optimized human kernels created a software moat that a raw hardware memory advantage could not breach. It is a brutal lesson in hardware engineering: you can successfully rewrite the physical boundaries of semiconductor memory, but you cannot convince a machine learning researcher to learn a new API.

Cerebras’s commercial survival was dictated by the shifting economics of production inference. The same SRAM bandwidth profile that was a secondary consideration during training became a decisive bottleneck during sequential LLM decode. The company’s strategic pivot to inference was a forced entry into the only market segment where their architecture’s physics mapped directly to the industry’s primary cost center.

However, there is a blocker that Cerebras will have to address if it is to move forward on capitalizing on this shift.

The Capacity Deficit: What the Cerebras Architecture Gives Up

The severe tradeoff of choosing 44 GB of elite on-chip SRAM over 141 GB of commodity HBM is a devastating capacity deficit that hits large-scale frontier models immediately.

Serving a standard Llama-3 70B model at full precision requires four interconnected CS-3 systems, representing an infrastructure cost of 8 to 12 million dollars to host a single model instance. A single NVIDIA H200 or B200 GPU chip, carrying up to 141 GB of high-capacity HBM, can house that entire 70B parameter model on a single 30,000-dollar card. The GPU delivers significantly lower per-stream token speed, but at a capital footprint that is orders of magnitude smaller.

This constraint becomes fatal when scaled to massive mixture-of-experts models. Serving DeepSeek-R1 (671 billion parameters) at full precision would require a minimum cluster of 30 connected CS-3 systems — a raw hardware outlay of roughly 75 million dollars per model replica. Consequently, Cerebras cannot practically host or serve the largest, highest-volume open-weights model in the current market. As a direct result of this physical capacity floor, Cerebras’s live inference catalog remains constrained to just four models, two of which are scheduled for complete deprecation on May 27, 2026. However, the rise of attention given to edge models (which are the best types of models for Cerebras), might provide the perfect outlet to Cerebras.

The OpenAI Capacity-as-A-Service Contract

To survive outside its primary sovereign revenue base — where Abu Dhabi entities accounted for 86 percent of 2025 recognized revenue — Cerebras executed a massive Master Revenue Agreement with OpenAI in December 2025. The headline terms outline an infrastructure allocation of 750 megawatts (MW) of datacenter capacity from 2026 to 2028, expandable to 2 gigawatts (GW) by 2030, valued at a gross contract potential exceeding 20 billion dollars.

The underlying unit economics of this agreement reveal that the contract operates as a loss-leader to justify the capital buildout. The contract prices out to approximately 9 million dollars per megawatt per year. Based on the active power profile of the CS-3, this yields roughly 207,000 dollars of contracted OpenAI revenue per system, per year.

However, the comprehensive Total Cost of Ownership (TCO) for a single CS-3 system — combining amortized hardware acquisition, power draw, liquid cooling infrastructure, and onsite operations — sits at approximately 1.095 million dollars per year.

This yields an implied negative 81 percent gross margin per system on the baseline OpenAI tenant.

For Cerebras to achieve profitability on this infrastructure, they must execute a multi-tenant stacking strategy. Because this contract describes Capacity-as-a-Service rather than dedicated hardware ownership, Cerebras must dynamically resell the physical wafer time to secondary enterprise customers during OpenAI’s idle windows, layer high-margin software services on top of the raw compute layers, and rely on generation-over-generation silicon efficiency gains to decrease the long-term TCO per watt.

Because the 2025 annual report explicitly shows OpenAI accounting for zero dollars in recognized 2025 revenue, the diversification thesis is being sold on forward projections. The market has completely priced this 20 billion dollar agreement into the 56.4 billion dollar IPO valuation, but the underlying physical infrastructure has yet to officially demonstrate its first dollar of positive long-term operational margin on the public balance sheet.

For what its worth, I do think this partnership will generate a lot of value for people. While the numbers don’t look great right now, their collaboration has led to masterpieces like GPT-Codex-Spark, which is an amazing model loved by most Codex users. It’s been more reliable than Claude Code, which is an exceptional demonstration of reliability, and that should count for a lot when Cerebras tries to expand its customer base.

This expansion will have contend with several challenges, which we will detail next.

Challenges to Cerebras

How It Handles Compilation

Because Cerebras relies on independent core redundancy to achieve viable manufacturing yields, every wafer possesses a completely unique pattern of roughly 70,000 disabled cores. This makes each chip a physical snowflake, meaning compile binaries are non-portable across a fleet. A binary compiled for one unit cannot run on another because their functional core maps do not line up. If a unit fails, the entire model must be recompiled against the specific defect map of the replacement silicon.

This layout forces the Graph Compiler to calculate all core placement and message routing at compile time. For pre-optimized architectures in Cerebras’s Model Zoo, compilation takes 15 minutes, but custom operators stretch this loop to over 3 hours — and novel architectures frequently fail to compile entirely. When automated compilation fails, engineers must write custom kernels in Cerebras Software Language (CSL), a proprietary model requiring manual coordination of data placement across a physically unique 2D grid. Unlike the GPU ecosystem, which features thousands of CUDA developers contributing optimized open-source kernels, the pool of CSL experts is virtually non-existent.

The Software Ecosystem Cliff

The software isolation of a proprietary platform creates an immediate barrier to standard deployment frameworks. Dominant open-source inference serving frameworks like vLLM do not run on Cerebras hardware. There are no native HuggingFace Trainer integrations, and load-bearing optimization libraries like FlashAttention, Triton, CUTLASS, or xformers do not exist for the platform. As of May 2026, Cerebras supports exactly four models, and two of them — Llama-3.1 8B and Llama-3.3 70B — are scheduled for deprecation on May 27, 2026. While the HuggingFace repository hosts hundreds of thousands of model variants that deploy instantly on NVIDIA hardware, Cerebras supports roughly 0.001 percent of that ecosystem.

Furthermore, proprietary closed-source frontier models are structurally inaccessible to Cerebras, as they remain locked behind their providers’ cloud APIs.

(We are also hit with the inaccessibility of their system to big open weight models as detailed earlier)

What Are the Operational Moats and Trade-offs?

Evaluating the operational model requires balancing where the architecture simplifies infrastructure against where it complicates development. For models smaller than 44 gigabytes, a model fits entirely on a single wafer. This eliminates the distributed-systems complexity of multi-GPU setups, bypassing NVLink topology tuning and NCCL communication failures entirely. Execution is strictly deterministic because the compile schedule is fixed, preventing the non-deterministic numeric drift common in multi-GPU debug loops. Out-of-memory errors are safely trapped at compile time rather than triggering an unpredictable crash 17 hours into a run.

Conversely, the absence of a community kernel ecosystem means developers must independently build unsupported operators from scratch. Tooling is dangerously thin, lacking any granular equivalent to NVIDIA’s Nsight Compute for per-core performance profiling. The long compile loops fundamentally degrade development velocity compared to GPU workflows.

What Are the Physical Failure Modes of Wafer-Scale Systems?

The extreme power and thermal densities of a wafer-scale chip introduce unprecedented operational risks, yet public documentation regarding its reliability metrics remains entirely empty. If the liquid cold plate experiences a localized drop in flow or a loss of surface contact, the WSE-3’s 23-kilowatt power draw allows sub-second time-to-overtemperature limits. The wafer can hit catastrophic thermal damage within seconds before standard datacenter monitoring systems can generate an alert. GPU servers, by contrast, feature long thermal runway and firmware-level automated throttling.

Electrical failures compound this risk. If one of the 300+ voltage regulator modules on the backing board experiences a partial failure, the corresponding reticle zone suffers a local voltage sag. The system cannot dynamically recompile around a localized power brown-out during operation. Instead, this creates a severe silent-data-corruption risk, where local cores generate mathematically incorrect outputs without throwing an explicit hardware error flag. If a core fails permanently post-deployment, the entire machine must be pulled offline to manually re-map the defect layout and run a fresh multi-hour compilation. There is also no public data on how long the proprietary, compliant connector survives the physical shearing forces of 150-to-190-micron thermal expansion cycles.

The Latency Gotchas: Prefill and Multi-Tenant Constraints

Cerebras’s heavy marketing of its 2,100+ tokens-per-second decode speed masks a critical latency flaw: time to first token (TTFT). TTFT requires running a prompt prefill phase, which is not comparably advantaged by the architecture’s sequential design. For a high-reasoning model like gpt-oss-120B, the published TTFT sits at 1.53 seconds.

While an individual user completing a single-turn prompt will not mind a 1.53-second initialization lag, this delay is devastating for agentic workloads. If an autonomous AI agent must chain together 30 sequential LLM calls to complete a multi-step task, the fixed prefill cost multiplies across every iteration, dragging the total loop time to nearly three minutes. In these multi-turn loops, a standard GPU cluster with faster prefill units can deliver comparable total execution times despite generating tokens slower on a per-step basis.

Concurrently, multi-tenant serving introduces strict scaling walls. Cerebras’s pipeline-interleaving mechanism relies on tiny concurrency batch sizes, typically capping a single machine at 10 to 20 concurrent users to avoid bubble latency. At a fleet level, Cerebras’s public API supports roughly 18,000 concurrent users across an estimated deployment of 1,000 to 1,800 active CS-3 machines. Cerebras operates this multi-tenant concurrency model as a complete black box, hiding exact capacity limits, queue behaviors, and rate-limiting thresholds from enterprise buyers.

Sovereign Concentration and Financial Reality

Cerebras’s corporate financial statements reveal extreme customer concentration beneath the high-level metrics. In 2025, a staggering 86 percent of all recognized revenue originated from just two Abu Dhabi entities: the Mohamed bin Zayed University of Artificial Intelligence at 62 percent and G42 at 24 percent. Both operate within the same ultimate state apparatus, meaning this contract value represents an accounting shift rather than genuine market diversification.

Furthermore, the headline 2025 GAAP net income of 237.8 million dollars is entirely driven by a single, non-recurring 363.3-million-dollar accounting gain from the extinguishment of a historical G42 forward contract. Strip this one-time contract adjustment from the books, and the underlying core commercial business operated at a non-GAAP operating loss of 75.7 million dollars. Because the landmark OpenAI agreement contributed zero dollars to recognized 2025 revenue, Cerebras is heading toward its IPO backed by a single sovereign network and a profitability narrative constructed from a one-time book event.

The Manufacturing Scale-Up Risk

Fulfilling the forward order book requires an unprecedented industrial transformation. Cumulatively through the end of 2024, Cerebras has manufactured and shipped approximately 192 total systems. The forward OpenAI infrastructure agreement requires delivering roughly 32,600 systems by 2028 — demanding a 170-fold increase from cumulative lifetime production to an annualized delivery cadence in less than four years.

Every single CS-3 machine requires a full, un-diced 12-inch wafer, yet Cerebras’s public SEC disclosures explicitly state that the company possesses no committed long-term wafer allocation from TSMC. They must compete for limited cleanroom allocations against massive chip buyers like Apple, AMD, Qualcomm, and NVIDIA at TSMC’s most advanced nodes. At the packaging layer, the critical thermal-expansion connector is manufactured under an exclusive, single-source agreement, leaving the entire multi-billion-dollar scale-up exposed to severe supply chain dependencies without the globally distributed networks of its competitors.

Conclusion: Why Did Cerebras Raise at 56 Billion Dollars

Cerebras is wagering on a specific physical reality: that memory bandwidth will dictate the future of inference, and standard GPU packaging cannot scale fast enough to solve it.

• If standard High Bandwidth Memory (HBM) improves enough over the next three years to eliminate the decode bottleneck, Cerebras becomes a brilliant but transitional technology.

• If it doesn’t — and if single-stream latency becomes the absolute speed limit for agentic AI — Cerebras is positioned exactly where standard architecture physically breaks.

The interesting thing is that Cerebras doesn’t actually need to win the market for their underlying thesis to be right. The industry has already conceded the point. NVIDIA spending 20 billion dollars on Groq, and Google splitting its TPU line into separate training and inference chips, proves that the incumbents know the current hardware is wrong for the job. Starving compute units during decode is a waste. Someone will capture the value of fixing this.

The engineering is real. The physics advantage is quantifiable. The question has never been whether Cerebras built something that works. They did.

The question is whether the market window stays open long enough, and the software ecosystem grows deep enough, for wafer-scale silicon to survive the incumbents it forced into action. And most importantly, how much risk you’re willing to stomach on potentially paradigm-altering bets. Your answer to that determines how much attention you gove this architecture.

Personally, I appreciate that we have a bunch of smart people building difficult things, as opposed to dedicating their lives to Slack Automation, gambling apps, and Ad-tech. That has to stand for something.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Fine-tuning is the industry’s favorite blunt-force instrument. It is expensive, computationally heavy, and — more often than not — it breaks as much as it fixes. In our collective quest for more efficient model control, Activation Steering promised a surgical alternative: an inference-time “nudge” that costs nothing and changes everything.

Yet, in production, steering often feels like fighting a ghost. You push for a specific concept, and the model’s distribution leaks into generic prepositions and hallucinations. You try to steer a model to be “more professional,” and suddenly it starts obsessing over the word “the” or “to,” losing the very nuance you were trying to preserve.

The problem isn’t that steering is “weak” — it’s that we have been fundamentally miscalculating the “shape” of the space our models live in.

We tend to treat the internal representations of an AI like a flat, simple map where you can just draw a straight line from Point A to Point B. But the moment a model uses Softmax to turn raw numbers into a probability distribution, that map warps. Much like mountaineering, you want your AI to account for the curvature of your landscape to get the best outcomes (this was one the things that made Kimi’s MuonClip work so well).

The paper we are breaking down today, “The Information Geometry of Softmax: Probing and Steering”, introduces a fix called Dual Steering: “We prove that dual steering optimally modifies the target concept while minimizing changes to off-target concepts. Empirically, we find that dual steering enhances the controllability and stability of concept manipulation.”

However, instead of going deep into their algorithm (which is shared above), I want to use this research as a starting point to talk about some of the grounding concepts in the research around a model’s information geometry (how it organizes and navigates internal knowledge representations), so that you can go beyond this paper and start understanding the larger space around LLM Geometry and Activation Steering.

In this article, we’re going to walk through:

• Why Euclidean Math Lies to You: We’ll explain why treating a model like a flat grid causes “probability leakage,” and why a tiny nudge in the wrong part of the model’s “terrain” can flip the entire output in ways you didn’t intend.

• The “Two-Map” System of Softmax: You’ll learn how models actually use two different coordinate systems at the same time. One map is for “freedom” (where the raw vectors live), and the other is a “cage” (where the probabilities live). If you don’t know which map you’re using, you’ll accidentally crush the very ideas you’re trying to blend.

• The Difference Between Movement and Measurement: We will break down a common “type error” in AI research. We often try to “add” a measurement tool (like a probe) directly into a representation, which is mathematically as nonsensical as trying to physically add a thermometer to a room.

• Navigating the “Exit Nodes”: We’ll look at why this math currently only works at the final layers of a model and where the research needs to go next. This will help you make high-signal judgments on where to invest your developer attention and capital.

We’re moving away from bludgeoning models with scale and toward a more precise way of understanding the math of intelligence. I hope you’re as excited about this as I am.

Executive Highlights (TL;DR of the Article)

Our current failure to make Activation Steering — an efficient, inference-time intervention — as effective as expensive Fine-Tuning stems from a fundamental geometric “type error.” We are treating AI representation space as flat (Euclidean) when it is actually warped (Bregman).

The Euclidean Illusion vs. Bregman Reality

• The Problem: Standard steering assumes adding a vector moves a concept in a straight line without distortion. In practice, this “naive” addition causes probability leaks. Steering a model toward a specific verb might accidentally spike the probability of a random preposition like “to,” degrading the model’s overall intelligence.

• The Geometry of Softmax: Once a representation passes through a Softmax operation, Euclidean rules break. Softmax creates a Bregman geometry governed by the log-partition function A(lambda). In this space, distance isn’t a straight line; it is measured by KL Divergence.

• The “Type Error”: Researchers often treat the Linear Probe (a measurement tool/covector) as a Vector (a displacement). Adding a probe directly to a representation in the residual stream is mathematically akin to trying to “add a thermometer to a room.”

Dual Steering: A Geometric Fix

Two Coordinate Systems: Bregman geometry necessitates two systems: the Primal (lambda), which is the raw, infinite vector in the residual stream, and the Dual (phi), which is the probability-weighted “center of mass” of the model’s vocabulary.

• Primal Interpolation acts like a Logical AND, crushing unique traits to find a “safe,” generic consensus (often resulting in bland outputs).

• Dual Interpolation acts like a Logical OR, preserving the union of concepts and allowing for distinct mixtures without collapsing into prepositions.

The Solution: To steer effectively, one must map the primal vector to the dual coordinate, add the probe there, and translate back. This Dual Steering ensures a “KL Projection” — changing the target concept while minimizing the shift of the rest of the distribution.

The Bottleneck: Current math for dual steering only applies to exit nodes (where Softmax occurs, like final-token distributions or CLIP retrievals). It does not yet exist for the intermediate layers where most surgical steering is actually performed. This is a huge problem and will have to be addressed. That’s why we treat this work more as a jumping off point into the larger space as opposed to focusing most of our attention on discussing the algorithm.

Final Takeaway: we should stop trying to “bludgeon” models into submission with compute-heavy fine-tuning and start mastering the information geometry of the models themselves. Unlocking the math of how models represent knowledge is the path to efficient, precise control.

Things you might find interesting:

1. Our research on Latent Space Reasoning, which allows us to unlock new capabilities not present in the base models, without any training.

2. Our work on Fractal embeddings, which allows us to integrate a sense of hierarchy directly into embeddings, allowing for lossless architectural depth hierarchical classification.

Both streams of research are promising pointers to how exploring geometry can unlock low-cost, powerful solutions that improve the AI landscape.

PS: Personal update. A good friend of mine is hosting this event. Y’all might find it interesting to go to (I’ll be there as well, lmk if you want to come say hi).

Website: silsilasounds.org | IG: @silsilasounds

Hope I’ll see you here.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

What Does Activation Steering Actually Do to a Model’s Output Distribution?

Generally, when we want to impose a behavioral change in a model, we tend to rely on Fine-Tuning. When it works (and it never does), fine-tuning a 70B parameter model requires dataset curation, regression testing, and hundreds of GPU hours; thousands of dollars per behavioral tweak (at the end of which you’re hoping that your expensive lil trip hasn’t broken something else in the model’s ability (which it generally does)).

Activation steering, on the other hand, costs nothing. It is an inference-time intervention — a single vector addition during the forward pass. So why don’t we do it everywhere? It hasn’t had the kind of results we were expecting from it. The reason why might have been in the way we were thinking about the space our vectors occupy.

The standard approach to steering assumes the model’s representation space is Euclidean — a flat mathematical environment where adding a vector moves a concept in a straight line without distorting the rest of the distribution. Here, you find the vector direction for a concept and add it. Let’s look at an example:

• Feed Gemma “Under Al-Teta, Arsenal play…”.

• The model predicts base verbs like “haramball” or “set piece ball.”

• To steer it toward positive verbs like “exciting football,” you calculate the “exciting games” vector, magnify it (you have to gaslight the model a lot for this example) and add it to the active representation.

The naive mental model assumes probability mass shifts cleanly from the base to the exciting concept. In production, the probability leaks. At intermediate steering strengths, the preposition “to” might suddenly pull more mass than any verb in the distribution. You ask for a conjugation and the model might hallucinate a preposition. This is not unique to Language Models; we observe the same failure in vision models. Steer MetaCLIP-2 away from “cat” toward “dog,” and the top retrieval becomes an image containing both a cat and a dog. The target concept moves, but unrelated concepts get dragged with it.

This explains why steering consistently loses to fine-tuning in head-to-head benchmarks. Subspace patching creates illusions of control while actual model behavior degrades.

To reiterate, since this is an important point, standard steering commits a type error: Once a representation passes through a softmax operation, it no longer exists in a Euclidean space. Softmax creates a Bregman geometry — a space that is mathematically flat, but requires two different coordinate systems to map distances correctly. Standard steering collapses these systems into one.

The research we’re going to look at derives a fix they call dual steering, and proves that under a clean probe and a concept-factorization assumption, dual steering is the KL projection that changes the target concept while minimizing off-target distribution shift.

The exact geometry derived in the paper only applies to representations that directly parameterize a softmax distribution. This covers final-token distributions, CLIP retrievals, and attention layers. It does NOT cover arbitrary intermediate layers deep inside the network. Most production steering targets those intermediate layers to intercept concepts before they propagate. The math for Bregman geometry at intermediate layers does not yet exist. So, as we get into this research, consider this more the start of an interesting discussion as opposed to the final say. My hope is that by surfacing this research (and others like this) we can encourage more contributors in our open source community to start exploring the geometry/math of AI, instead of purely looking at the standard axes of scale, tweaking, and model tuning.

To explore this research in-depth, we must first ask ourselves a very fundamental question — 

How Does a Representation Become a Distribution, and What Should “Close” Mean?

When we want to know if two representations are “close,” we naturally measure the straight-line distance between their vectors. Why? Because torch.dist() is easy to type, and we like to pretend AI happens in a clean, flat space (the Euclidean assumption we just about).

But as we saw in the last section, Euclidean distance lies to you. A microscopic nudge near a decision boundary flips the entire output, while a massive shove when the model is 99% confident does absolutely nothing.

The model doesn’t care about the geometric distance. It cares about the output distribution. To define “close” correctly, we have to look under the hood at how a vector actually becomes a distribution.

The model holds an active representation vector for the current context — let’s call it lambda (personally, I don’t love the use of Greek letters, but I’m keeping it here since most of the literature uses them). It also holds a massive lookup table of “unembedding” vectors for every possible token in its vocabulary — let’s call those gamma_y. To score a specific token, the model calculates the dot product of lambda and gamma_y.

Why a dot product? Because a dot product is fundamentally a measure of directional alignment. It asks the math: “How much does our current context vector point in the exact same direction as the ‘exciting football’ vector?” High alignment means a high raw score.

But these raw scores (logits) aren’t probabilities. To get probabilities, the model shoves them through a Softmax function. Softmax does two things

• It exponentiates the scores

• Then it divides by the sum of all the exponentiated scores so everything equals 100%.

Exponentiation is a bloodbath. Let’s say “haramball” scores a 10, “set piece” scores an 8, and “exciting” scores a 3. In raw score space, 3 is behind 10, but it’s in the same zip code. Once you exponentiate them (e¹⁰ vs e³), “haramball” shoots to roughly 22,000. “Exciting” is sitting at 20. Divide by the total, and “haramball” owns 88% of the probability mass. “Exciting” gets 0.08%. Softmax takes a mild preference and turns it into a blowout.

This extreme sharpness is exactly where standard mathematical tools break down — specifically, the covariance matrix.

All of this chaos is controlled by the denominator in that Softmax equation — the normalizer that divides everything. Because we usually work in log-space to keep our GPUs from throwing underflow errors, we take the log of that massive sum. This term is so fundamental it gets its own name: the log-partition function, written as A(lambda).

The actual probability of a token is just its exponentiated score minus this A(lambda) function. A handles all the normalization. This creates a very interesting outcome: the geometry of this space, the duality we need, and the steering fix we are about to build — all of it is hiding inside the derivatives of A(lambda).

To see why, we need a way to measure how different two softmax distributions are. And that leads us to our next point of exploration…

Why Does KL Divergence Measure the Change We Actually Care About?

We just established that the log-partition function A(lambda) controls the shape of our representation space. But before we can use it to fix our steering vectors, we have to solve a more immediate problem: how do we measure distance on this new terrain?

If Euclidean distance is a lie (yet another reason to not trust the Greeks)— if it treats a massive shove at 99% confidence exactly the same as a tiny nudge at a decision boundary — then what is the truth? We need a function that takes two probability distributions, compares them, and returns a single number representing how different they actually are in practice.

Our hero comes from Information Theoery: Kullback-Leibler (KL) divergence. If the true distribution is P, but your model assumes the distribution is Q, KL(P || Q) is the exact mathematical cost of that error. It calculates how surprised you will be when reality actually happens.

The formula is a sum over all possible outcomes x: P(x) * log(P(x) / Q(x)).

There are three moving parts here, and they each do a specific job to keep the math grounded in reality:

1. The Ratio: P(x) / Q(x) For any token x, how much more likely is it under the true distribution (P) than your steered distribution (Q)? If they agree, the ratio is 1. If P thinks the token is a sure thing and Q thinks it’s impossible, the ratio explodes.

2. The Logarithm Log converts multiplicative ratios into additive scores. A ratio of 1 (total agreement) maps to exactly zero. But more importantly, log makes KL sensitive to proportional changes, not absolute ones. A token dropping from 50% to 40% is only a 1.25x change. A token dropping from 0.1% to 0.01% is a 10x change. Log mathematically enforces the rule that relative probability determines model behavior.

3. The Weighting: P(x) The whole thing is multiplied by P(x) before summing. This is the “Do I actually care?” filter. KL only punishes disagreements where the true distribution P actually puts probability mass. If P thinks a token is garbage (P(x) is near 0), that term vanishes. What about Qs thoughts? Respectfully, who gives us a fuck what a grunt like Q thinks when a baller like P has already made up its mind.

Putting everything together, KL(P || Q) asks: “If P is the absolute truth, how surprised would you be if you had to navigate the world using Q?”

This weighting creates a profound asymmetry. KL(P || Q) does not equal KL(Q || P). Being wrong about P when Q is true costs a different amount than being wrong about Q when P is true. In the Euclidean space, the distance from New York to London is the same as London to New York. In information space, the penalty depends entirely on which distribution is actually generating your data. This asymmetry is exactly what causes forward and reverse KL to produce entirely different behaviors (which becomes the crucial AND-vs-OR distinction when we talk about interpolation later).

Why does this matter? Let’s dig into this math just a wee bit more. If we do this step-by-step, the geometry of the entire model falls out of the algebra.

What Happens When You Compute KL Between Two Softmax Distributions?

First, let’s put the pieces back on the board so we don’t lose track of what we are building:

• lambda is our original, unsteered context vector.

• lambda-prime is the steered vector (after we add our behavioral tweak).

• gamma_y is the unembedding vector for a specific token (the dictionary definition the model checks against).

• A(lambda) is the log-partition function — the brutal normalizer that forces everything to sum to 100%.

The KL formula asks us to compute a ratio for every token: log [ P(y | lambda) / P(y | lambda-prime) ].

In plain English: take the logarithm of the true probability divided by the steered probability.

How do we calculate those probabilities? Remember from the previous section: the probability of a token is just exp(score - normalizer).

Because we are taking the logarithm of an exponentiated number, the math simplifies beautifully. The log simply deletes the exp, leaving only the raw terms inside. Division inside a logarithm becomes subtraction outside.

So, taking the log of that probability ratio strips the math down to just the raw scores and the normalizers. For the top part of the fraction (the original state), we get: lambda * gamma_y - A(lambda)

For the bottom part (the steered state), we subtract it: minus [ lambda-prime * gamma_y - A(lambda-prime) ]

If you group the similar terms together, the log ratio becomes a clean, linear equation: (lambda - lambda-prime) * gamma_y + A(lambda-prime) - A(lambda)

Now, the final step of the KL divergence formula tells us to multiply that result by the true probability P(y | lambda) and sum it up over every token in the vocabulary.

When you do that, something fascinating happens to that gamma_y term. You end up calculating the sum of P(y | lambda) * gamma_y.

Stop and think about what that is physically. You are taking every single token vector in the model, weighting it by how likely that token is to be generated, and averaging them all together. It is the probability-weighted center of gravity for the model’s current state.

As it turns out (and we will prove exactly why in the next section), this center of gravity is exactly the mathematical gradient of our log-partition function A. Let’s just call it grad-A(lambda).

If we substitute that gradient back into our equation, the final KL formula reveals itself:

KL = A(lambda-prime) - A(lambda) - grad-A(lambda) * (lambda-prime - lambda)

Look at the physical architecture of this final result.

A(lambda-prime) - A(lambda) is exactly how much the normalizer actually changed when we steered the vector.

grad-A(lambda) * (lambda-prime - lambda) is how much a straight, flat tangent line predicted the normalizer would change.

The KL divergence is literally the gap between the true change and the linear approximation.

Why is this gap always positive? Because the normalizer function A is convex — it curves upward like a bowl. For any convex function, a straight tangent line will always sit below the curve. The true curve always bends up and overshoots the straight-line prediction. The gap only hits zero if the two vectors are exactly the same.

This gap — the error between a convex function and its linear approximation — has a formal mathematical name. It is a Bregman divergence.

KL(P || Q) between two softmax distributions isn’t like a Bregman divergence. It is the Bregman divergence induced by the log-partition function A.

Nobody chose this. The algebra forced it. It means Euclidean geometry is not the natural default once a representation passes through a Softmax distribution. Every LLM, every CLIP model, and every attention layer is living and breathing in a Bregman geometry that most researchers have never explicitly mapped out. We’ve been using Euclidean wrenches on Bregman bolts since 2017.

Information geometers have studied Bregman divergences since the 1980s. And the very first thing their toolkit tells you about a space governed by Bregman geometry is this: the representation space does not have one natural coordinate system. It has two. This has some very juicy implications.

What Are the Two Coordinate Systems, and Why Does the Duality Matter?

The first coordinate system is the one everyone already uses: lambda. This is the raw representation vector sitting in the residual stream. Let’s call it the primal coordinate.

The primal space is the Wild West. It is entirely unconstrained. You can take your lambda vector, multiply it by a million, and point it absolutely anywhere in that 4,096-dimensional space. The model won’t crash. Softmax will just take those massive numbers and turn the output into a brutal step-function where one token gets 99.999% of the mass. The Primal Space has that good ol’ Murican freedom, baby.

The second coordinate system is completely different. It comes directly from that gradient term we isolated earlier: the gradient of our log-partition function. Let’s call this new coordinate phi.

phi = grad-A(lambda)

Let’s walk through the actual derivative (in case you don’t remember, derivatives give us the rate of change of something with respect to something else) to see what phi is physically made of. Don’t skip this, because it is the most elegant piece of math in the entire architecture.

Remember that our normalizer function is A(lambda) = log [ sum of all exp(scores) ].

To find the gradient, we take the derivative. The chain rule in calculus tells us that the derivative of a logarithm is simply 1 / x multiplied by the derivative of whatever is inside the log.

• The bottom of our fraction (the x) becomes the inside of the log: the sum of all the exponentiated scores.

• The top of our fraction becomes the derivative of those scores. If a token’s score is lambda * gamma_y, its derivative with respect to lambda is just the token vector itself: gamma_y.

So, for any given token, the gradient gives us this exact fraction: exp(score) / [sum of all exp(scores)] ... multiplied by the token vector gamma_y.

Look very closely at the left side of that multiplication. That fraction is literally the exact formula for Softmax probability.

The algebra just handed us a massive gift. The gradient of the log-partition function is simply every single token vector in the model, multiplied by its Softmax probability, and added together.

phi = sum over y of [ P(y | lambda) * gamma_y ]

This is our dual coordinate. Physically, it is the probability-weighted center of mass for the entire vocabulary. If the model assigns 70% probability to “maintains”, 20% to “operates”, and 10% to random noise, then your dual coordinate (phi) sits exactly at 0.7 * (maintains) + 0.2 * (operates) + 0.1 * (noise). It is a physical coordinate telling you exactly where the model’s attention is currently hovering across the dictionary.

Because of how Bregman geometry works, lambda and phi are just two views of the exact same distribution. If you have the primal vector, you can calculate the dual center of mass. If you have the dual center of mass, you can reverse-engineer the primal vector.

But there is a massive physical asymmetry between them. We already established that the primal space (lambda) is infinite. The dual space (phi) is trapped in a cage.

Why? Because phi is built by multiplying token vectors by probabilities. Probabilities obey strict laws: they can never be negative, and they must add up to exactly 100%. Because of this, phi can never step outside the boundary drawn around your vocabulary. Imagine stretching a massive mathematical rubber band around every single token vector in the model’s embedding space. That rubber band is called a convex hull. You can move phi anywhere inside the hull by mixing different token probabilities, but you can never push it outside. If you try to steer the dual coordinate outside that hull, the math shatters, because no valid probability distribution could ever put you there.

What Do the Two Coordinate Systems Mean Semantically?

Why do we care that there are two systems? Because they answer the most basic question in geometry completely differently: how do you draw a straight line? This might seem like a silly questions, but drawing a straight line between two concepts is how we blend them. When you want to combine two ideas, you take their representations and find the midpoint. But on a warped Bregman surface, your midpoint completely depends on which coordinate system you use to draw the line.

Let’s look at a concrete example. You have two representations.

• Vector 0 is the model’s state after reading: “Q: What is the capital of France? A: It is” (Probability sits heavily on the token “Paris”).

• Vector 1 is the state after reading: “Q: What is the capital of Germany? A: It is” (Probability sits heavily on “Berlin”).

You want to find the exact 50/50 midpoint between these two concepts.

If you do a primal interpolation, you draw a straight line between the two raw lambda vectors in the unconstrained residual stream. Because of how the Bregman algebra shakes out, moving in a straight line in primal space mathematically forces the model to minimize the reverse KL divergence.

Think back to our KL section. Reverse KL is the “Do Not Hallucinate” penalty. It looks at the target distribution and says, “If the target thinks a token is garbage, you will pay a massive penalty for putting probability mass there.”

Look at what happens to the math when you stand at the primal midpoint. The France endpoint looks at the word “Berlin.” It sees that “Berlin” has near-zero probability in the France distribution, so it slaps the model with a massive penalty for including it. Simultaneously, the Germany endpoint looks at the word “Paris,” sees near-zero probability, and slaps the model with a massive penalty for including it.

What survives? Only the tokens that both endpoints agree are mathematically harmless — generic words like “The”, “is”, or “called”. Primal interpolation operates exactly like a logical AND. It crushes everything that makes a context unique and leaves only the safest possible intersection.

If you do a dual interpolation, you draw a straight line between the two phi vectors (the centers of mass) inside the convex hull. Moving in a straight line in dual space minimizes the forward KL divergence.

Forward KL operates under the exact opposite philosophy. It is the “Do Not Forget” penalty. It says, “If the target thinks a token is highly likely, you will pay a massive penalty if you fail to cover it.”

Look at the midpoint now. The endpoints are no longer allowed to veto each other. France demands you keep “Paris”. Germany demands you keep “Berlin”. Instead of crushing them, dual interpolation forces the model into a compromise. It creates a mixed probability distribution that holds both truths simultaneously, allocating roughly 50% mass to Paris and 50% mass to Berlin.

In other words, Dual interpolation operates exactly like a logical OR. It preserves the union of the two concepts.

This behavior is a fundamental law of any model that uses Softmax. If you take a vision model like CLIP and try to interpolate the concept of a “black dog” with a “white dog”, you get the exact same split.

• Primal interpolation (AND logic) searches for shared traits. The colors fight each other, the model panics, and it spits out a single dog with black and white spots.

• Dual interpolation (OR logic) holds both truths. It spits out an image that literally contains two distinct dogs — one black, one white.

Primal finds the boring consensus. Dual preserves the contradiction. Two completely different philosophies of blending concepts, arising purely from which ruler you picked up.

This is why the geometry actually matters for your pipeline. When you blindly subtract two vectors in the residual stream to build a steering direction, you aren’t just doing niche math. You are accidentally making a product decision. By calculating your vector in the raw residual stream, you are locking yourself into the primal coordinate system. You are forcing the model into that destructive AND logic. You are explicitly telling the model to crush anything unique about the prompt and only keep the safe consensus. This is exactly why standard steering leaks probability mass to generic prepositions like “to” — it’s abandoning specific concepts to find the mathematical middle ground.

If you actually want to add a behavioral concept without destroying the original context — if you want the OR logic — you cannot just add vectors in the residual stream. You have to translate the vectors, do the addition in the dual space, and translate them back.

Isn’t exploring the math of intelligence so much cooler than being a training grunt? Imagine learning about some of the coolest topics in the world only to be forced to debug GPU crashes and benchmark tests 24/7.

Interpolation shows that the two coordinate systems produce different behaviors when you blend representations. Steering is a related but sharper operation: instead of blending two representations, you’re modifying one to change a specific concept. The question is the same — which coordinate system are you operating in? — but the stakes are higher, because steering with a probe means adding a specific mathematical object to the representation. What kind of object the probe is determines which coordinate system it belongs to.

What Kind of Mathematical Object Is a Linear Probe?

Before we fix the steering math, we have to look closely at the tool we are using to steer: the linear probe.

When you train a linear probe to detect a concept — say, “third-person verb” — you are building a very specific mathematical object. It takes your 4,096-dimensional representation vector (lambda), runs a dot product against its own weights (beta_W), and spits out a single scalar score.

In linear algebra, an object that eats vectors and spits out scalars is a linear functional. A covector. Covectors live exclusively in the dual space.

Most of us learned the difference between vectors and covectors in undergrad and immediately dumped it from RAM. Why? Because if you live in flat Euclidean space, you don’t need to care. Flat space lets you cheat. You can add them, subtract them, and treat a covector exactly like a normal vector.

But they are physically different objects.

A vector is a displacement — a physical direction you can move. A covector is a measurement — a tool that assigns numbers to states. Think of a thermometer. A thermometer measures a room and returns a temperature. The thermometer itself is not a room. You cannot take a physical location and mathematically “add” a thermometer to it.

In the warped, twisted Bregman geometry of Neural Network Information Spaces, we can’t get away with conflating them. Technically, we can, but that is why so much of the activation steering research has sucked for so long and all of the attention has been to the fine-tuning tards.

What Does Standard Activation Steering Get Horribly Wrong?

Look at the formula the entire open-source community currently uses for activation steering: lambda_t = lambda_0 + t * beta_W

Take the original context vector (lambda_0), add the probe (beta_W) multiplied by some steering strength (t).

As an array operation, it runs perfectly. PyTorch will execute the addition without throwing a warning because both objects are just float32 arrays of the same shape. But PyTorch doesn’t know geometry.

As a geometric operation, this equation is a disaster. It blindly mashes a measurement tool (the probe) into a physical displacement (the representation). Because Softmax forces the model into Bregman space, vectors and covectors are not interchangeable.

But Dev Dev, you can say it sucks, but I don’t understand why. What does this type of error cost you in production?

When standard steering executes that addition, it is mathematically asking the model to find the closest point that satisfies the new concept. But because it used Euclidean math, which we already proved has zero relationship to the model’s actual output distribution. Put two and two together and we see that — that gap (the distance between the Euclidean guess and the true Bregman reality ) is exactly where your probability leaks. It is the physical reason the preposition “to” steals all the mass from your verbs. It is the reason your vision model spits out a dog with black and white spots instead of two distinct dogs.

The fix is one line of math.

phi(lambda_t) = phi(lambda_0) + t * beta_W

Map your primal vector to the dual coordinate system (phi). Add the probe (beta_W) in the dual space, where measurement tools actually belong. Then translate the result back to primal space to hand off to the next layer.

So what does this actually accomplish? A lot, actually.

The Payoff: What Actually Happens in Production?

What happens to Gemma-3–4B?

The hallucination dies. When you run the dual steering vector to change a base verb to a third-person verb, the probability mass shifts directly from “maintain” to “maintains”. That generic preposition “to” that spiked out of nowhere in the primal space? It stays completely flat. The probability leak is sealed.

What happens to the vision models?

Steer MetaCLIP-2 away from “cat” toward “dog” in dual space, and the model directly transfers probability to images of dogs. The off-target “cat and dog sitting together” image never spikes.

Why Does Dual Steering Actually Work?

When you steer in the primal space, you are just adding raw numbers to the model’s logits. Softmax then exponentiates those inflated numbers. Because exponentiation amplifies differences, the artificially massive logits crush the model’s original context. To make everything sum to 100%, the model abandons the specific context and falls back on the safest, most frequent tokens it knows. As with most things in life, an overabundance of playing it safe leads to bland, generic output (tokens like to).

The dual space (phi) operates under different laws. Because it is built entirely from probabilities, it is strictly zero-sum. The total mass is locked at exactly 100%. You cannot inject raw, unconstrained numbers here.

If you want to push the center of mass toward the third-person verb “maintains”, the math forces a direct trade. This trade is the physical reality of a “KL projection.” The projection takes your target concept and solves for a new probability distribution using two strict rules: the new distribution must satisfy your steering target, and it must change the original probabilities as little as physically possible.

To satisfy both rules, it steals the required probability mass directly from the base verb (“maintain”) and hands it to the target verb (“maintains”). The generic token “to” stays flat because the distribution was never shattered.

That’s not to say this is without flaws. Dual steering fixes the geometric type error, but it hits a hard wall in production: it only works at the exit nodes.

The Bregman geometry derived by Park et al. relies entirely on the log-partition function. That means it only applies to representations passing directly through a Softmax distribution. In modern architectures, that restricts you to the final-token unembedding layer, CLIP retrievals, and attention matrices.

In production, almost nobody steers at the final layer. By the time a concept hits the unembedding matrix, the model has already made its decision. Surgical steering happens deep inside the network — say, layer 15 of a 32-layer model — to intercept a concept before it fully forms. But intermediate layers don’t have a Softmax attached to them. They are just unconstrained states floating in the residual stream. For intermediate layers, the mathematical map of Bregman geometry simply stops.

So…. we can never really apply the math we just talked about to hit real-time steering. That is a slight problem.

So where does this leave us? Do we just retard-max and let fine-tuning handle all the work? Let’s bring our little exploration to a close.

Conclusion: Where Does Dual Steering Go From Here.

Let’s end our journey with a trip down memory lane, back to the days of classic ML. Think about how LLMs changed production. We used to build massive, brittle scaffolding — wiring ten specialized ML classifiers to two generators just to complete one basic workflow. LLMs replaced all of that with a single general-purpose model, collapsing infrastructure costs overnight. That drop in cost didn’t happen because we optimized the classifiers. It happened because we completely changed the interaction pattern.

We are at the exact same threshold with model internals.

Right now, the industry’s default reflex for bad model behavior is to treat it as an infrastructure problem. When a model fails, we throw compute at it. We run brittle fine-tuning jobs and try to bludgeon the network into submission using scale.

But as more research is showing, the model layer might be the wrong layer to solve these problems. Perhaps the true solution is deeper. By changing how the space of how we represent knowledge in the language models, and then how we navigate it, we might be able to unlock capabilities that our current paradigm deems unrealistic.

And even if it doesn’t work, isn’t the idea so much more fun? Do you really want to spend the rest of your career profiling GPUs, fiddling with random seeds, and writing 20 skills/agent templates so that your manager can show your teams AI readiness? Does your heart (and brain) not ache to do more, to push humanity’s knowledge forward?

Think that over.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

I spoke to Ryan Estes about Legal AI, Open Source Research, and why access to Legal Services needs to be more accessible over here. The conversation was very well received, so I’m sharing it here with Ryan’s permission.

I hope you enjoy it.

Companion Guide to the Livestream:

This guide expands the core ideas and structures them for deeper reflection. Watch the full stream for tone, nuance, and side-commentary.

1. The Three Hallucinations Hiding Under One Word

The Event — I broke hallucinations into three categories on the stream. Category one: the AI makes up a case that doesn’t exist. Category two: the case exists and the quote is real, but it’s from the wrong jurisdiction or doesn’t apply to your domain. Category three: the AI gives you an argument that looks correct on its own, but somewhere else in your documents there’s something that contradicts it, and the system never connected the two.

Why this matters — Everyone talks about category one because it’s the most obvious. A lawyer cites a fake case, gets sanctioned, it makes the news. But category one is also the easiest to fix. You just check whether the case exists. A basic validator catches almost all of it. Categories two and three are where the real damage happens, and they’re much harder to catch. The lawyer reads the brief, everything looks right, the citation is real, the quote checks out. They file. Then opposing counsel tears them apart because the cited authority was overturned in their jurisdiction, or because a deposition transcript on page 723 contradicts the whole argument.

Most legal AI tools are RAG wrappers. You upload documents, the system cuts them into chunks, turns the chunks into vectors, and when you ask a question it finds the chunks that look most similar to your question. This works fine for simple retrieval like “find the indemnification clause.” It does not work for “is this argument actually supported across all my documents.” Cosine similarity doesn’t know what a contradiction is. It doesn’t know about jurisdictions or whether a ruling is still valid. Two laws from different states will sit right next to each other in vector space even if they say opposite things. And “document 47 invalidates the claim in document 12” is a logical relationship that vector search can’t represent at all.

Every category-two and category-three hallucination is a potential malpractice claim. The tool makes you faster, you trust it, you file work that has buried contradictions, and the first time it costs a client real money your insurance situation changes permanently. The speed improvement means nothing if the work product carries hidden liability. This is why legal AI has to move to architectures that handle context across documents natively. The wrappers will be fine for boilerplate. They’ll fail at everything that actually matters.

2. Why Irys Doesn’t Wrap, It Rebuilds

The Event — Ryan asked whether Irys’s “infinite context” works the same way Harvey’s does. It doesn’t. Harvey chunks your documents and uses vector similarity to find relevant pieces. Irys builds a knowledge graph that links entities, propositions, assertions, and contradictions across all your documents, and updates that graph every time you add a new file. When you ask a question, the system walks the graph instead of doing similarity search.

Why this matters — Vector search became the default approach to “my documents don’t fit in the context window” because it was cheap and it worked for simple use cases. Then people treated it as a permanent solution. It was always limited. Chunks are independent of each other. There’s no cross-document reasoning. There’s no way to bind a claim in one document to evidence in another. None of this mattered when the use case was “summarize this PDF.” It matters a lot when the use case is “build me a litigation strategy across 100,000 pages.” I was writing about these structural limits in 2022, before RAG was even a common term.

The fix isn’t bigger context windows or better embeddings. The fix is a system that explicitly tracks entities like parties, dates, jurisdictions, and claims, that maintains contradiction edges between propositions, and that reorganizes itself when new documents arrive. That’s how you catch “document 47 contradicts document 12” before the LLM ever starts drafting. That’s what Irys does. And this is why the wrappers can’t close the gap by adding features. Their entire stack assumes chunks are independent and retrieval is similarity. To move to graph-native context, they’d have to throw it all away and start over.

3. Why GitHub Copilot Couldn’t Become Cursor, And Cursor Couldn’t Become Claude Code

The Event — Ryan asked what stops someone from copying Irys. This is the answer Harvard Business School reached out about for one of their courses. Software doesn’t just have features. It has assumptions baked into it about what the user controls, what the AI controls, where data lives, what gets automated, and what gets left to human judgment. GitHub Copilot had Microsoft’s distribution and money. It couldn’t become Cursor. Cursor had the developer tool category to itself. It couldn’t become Claude Code. Each product was built around a different set of assumptions, and those assumptions aren’t portable. They’re in every layer of the code and they compound with every release.

Why this matters — When founders talk about moats they usually talk about data, brand, and distribution. Those are visible from the outside. Architectural assumptions are not. If you build assuming the AI is an autocomplete assistant, you get Copilot. If you build assuming the AI is an autonomous agent that the user occasionally interrupts, you get Claude Code. The surface features can look the same but the products are completely different underneath. You can’t retrofit one into the other because every API, every state machine, every piece of the UX was built on the original assumption. Unwinding it costs more than starting over.

This is the moat for Irys. We assumed in 2022 that vector search would not be enough for legal context. So every layer of the system was built around graph-based context aggregation. Harvey assumed RAG was enough. They have a year of rebuilding before they can even start the conversation we finished three years ago. And by the time they’re done, we’ve shipped two more iterations on top.

There’s a second moat on the product side. The more you use Irys, the more it learns how you work. It doesn’t train on your data. But it learns which argument structures you prefer, what memo formats you use, how you weigh jurisdictions. All of that accumulates in your private workspace. If you leave, you lose all of that embedded knowledge and have to rebuild it somewhere else from scratch.

4. Why Open Source Is Cheaper Than Marketing

The Event — Irys is free to sign up. We’ve open-sourced major pieces of our reasoning infrastructure, including a lightweight version of the latent space reasoning engine. By normal startup logic, this makes no sense. We have a defensible product, real funding, well-funded competitors, and we’re giving away the technical work for free.

Why this matters — The usual way to think about open source is as a cost. Every piece of IP you publish saves your competitor some R&D time. That’s true if the game is static. In a fast-moving technical field, the thing that actually matters is who has access to information about what’s coming next. Who’s in the room when the labs are deciding the next generation of model capabilities. Who knows what’s shipping in six months.

Published work is what buys access to those rooms. NVIDIA’s senior engineers don’t take meetings with random startups. They take meetings with people whose technical work they’ve already seen. DeepMind doesn’t share roadmap previews with companies that haven’t contributed anything back to the field. The newsletter and the open-source reasoning engine aren’t marketing. They’re what get us the partnerships with the labs. That’s how we know what’s coming before it’s announced, and that’s how we make architecture decisions that our competitors don’t know they need to make yet.

The competitor who copies our open source saves maybe a quarter of engineering time. But they’re still nine months behind on the decisions that actually matter because they don’t have the relationships that tell them where the field is going. We didn’t lose value by publishing. We traded code for visibility into the technical horizon.

There’s also a recruiting benefit. The engineers who read the reasoning paper, run the lightweight implementation, and reach out about it are exactly the kind of people you can’t find through normal hiring channels. You can’t buy that pipeline. You can only earn it.

5. The Newsletter Is Peer Recruitment, Not Customer Acquisition

The Event — Ryan assumed the Chocolate Milk Cult newsletter (250,000+ subscribers, about 1.5M monthly reach) was the lead generation engine for Irys. I corrected him. Lawyers don’t read deep dives on quantization math or GPU pricing curves. The newsletter doesn’t sell legal seats. It serves a completely different purpose.

Why this matters — Most founder content advice treats your audience as a sales channel. That works when your audience and your customer are the same person. A fitness creator selling a fitness app. A finance creator selling a finance tool. When your audience is a different group from your buyer, the sales-channel framing leads you to measure the wrong things. You track conversion and click-through, you report the wrong wins, and eventually you conclude the audience isn’t worth the effort because the numbers don’t show a return.

What the audience actually does, when it’s decoupled from the customer base, is give you information you can’t get any other way. Lab researchers DM me about unpublished work. Engineers send me preprints. Investors share data they wouldn’t put in a pitch deck. None of that shows up as a Substack metric, but all of it changes what Irys ships next quarter.

The point for founders is simple. Be honest about who your audience is and what they’re actually for. If you’re building vertical SaaS for accountants, an audience of ML researchers won’t sell seats. But it might give you a technical edge that makes the product worth buying when your actual sales channel starts working. The two things serve different purposes and you can’t optimize both with the same content.

6. The Eat-Shit Theorem of Legal Access

The Event — Ryan asked why I picked legal when I could have pointed the technical foundation at any industry. The answer is the moral case. India has a ten-year backlog on civil cases. If your employer steals your wages and you go to court tomorrow, you won’t get a hearing for a decade. In NYC, tenants have landlords who let buildings rot and overcharge rent, and they can’t do anything because they can’t afford a lawyer. Indian farmers get oversold pesticides, fall into debt, and end up dealing with loan sharks. I get hit with baseless defamation suits over newsletter coverage, designed not to win but to bleed me on legal fees. The pattern is always the same. The legal system is gated by money, and ordinary people pay the price.

Why this matters — Democratization here is not a marketing word. It’s the reason every other decision gets made the way it does. Irys is free to sign up because if we charged what the market would bear, the people who need access most would never get in. We open-source the reasoning infrastructure because the category needs to advance whether we win or not. We’re building Irys Lite because the full product doesn’t reach a wage-theft case in rural India or a tenant in the Bronx.

For founders, the closing point from the stream is the one I’d keep. Be honest with yourself about whether you’re building a business or a mission. Both are fine. The question is which one survives the next downturn, the customer churn, the eighteen months where nothing works. If it’s a business, say so to yourself, your team, and your investors. Don’t dress it up as world-changing because that confusion is what burns founders out around year three. If it’s a mission, then it has to be the thing that’s still motivating you when a competitor with twenty times your funding announces the same product. The mission is whatever’s still there when it stops being fun. Everything else in the company is downstream of that.

Full conversation with Ryan is on AI for Founders, here. Try the platform free at iqidis.ai. The open-sourced latent space reasoning work is on the Chocolate Milk Cult archives. If your attorney is still billing you in fifteen-minute increments for things a knowledge graph does in fifteen seconds, send them this guide.

Subscribe to support AI Made Simple and help us deliver more quality information to you-

Subscribe now

Flexible pricing available—pay what matches your budget here.

Thank you for being here, and I hope you have a wonderful day.

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Last year, I compared the major AI web apps: ChatGPT, Gemini, and Claude. The question was simple: which one was worth paying for?

At the time, the answer was also simple. ChatGPT was the best general-purpose app. Gemini had strong models trapped inside a messy Google product. Claude had a good underlying model, but the app was too limited, too forgetful, and too frustrating for sustained work.

That rubric is now old. The major AI subscriptions are no longer just chatbots. They are memory systems, research tools, coding agents, file processors, image generators, project workspaces, and billing systems. The model still matters, obviously. But the model is no longer the whole product.

So I’m redoing our comparison with an updated question: Which subscription will let you get the most done for the least cost? I’m keeping the comparisons limited to Gemini, OpenAI, and Anthropic for 3 reasons:

1. Performance: They still have the best models available for use.

2. Reliability: All 3 tend to have much higher uptime and usage.

3. Paperwork: Some of the subscriptions (like the Chinese models) are typically not approved in most orgs and require a longer review process. By the time it is approved, we will likely see 10 new model releases and updates.

For the ranking, we will review them over the following points:

• workflow coverage. Can the subscription handle writing, research, coding, files, images, planning, review, and daily work without forcing you into three different products?

• memory and state. Does the system remember useful context without becoming weird? Good memory helps. Bad memory overfits to stale preferences. Additionally, there is ease of use/effortlessness (Anthropic natively looks at memories, Gemini needs to be prompted explicitly) and the cost (Anthropic burns a lot of tokens on memory)

• instruction stability. Can it follow complex instructions across long, messy work? Not one prompt. That is easy. The real test is whether it preserves constraints after ten turns, two corrections, and a change in direction.

• judgment and taste. Does it know what to cut? Does it know when the technically correct answer is still bad? Does it update proportionally when you push back?

• research quality. Search is not research. A good research workflow finds sources, weighs them, notices conflicts, and tells you what remains uncertain.

• coding and agentic execution. This is now central. Codex and Claude Code are not side features. They are main reasons to pay.

• output quality. Can it produce writing, visuals, reports, artifacts, and polished work that is actually usable?

• economics and billing trust. Can you predict what you are paying for? Does the plan scale with heavy use? Does the product punish you for doing real work?

• product friction. Does the app fight you? Does it hide limits? Does it make you restart? Does the best model live somewhere other than the product you are paying for?

• Ecosystem: Does it work well with other tools in the ecosystem?

Let’s see where you should be spending your money.

To access the full article—and all premium breakdowns going forward/written prior—upgrade to a premium subscription below.

Subscribe now

If you believe deep insight deserves support, become a premium subscriber to allow me to keep doing the same.

Flexible pricing available—pay what matches your budget here.

Most companies offer learning or professional development budgets. You can expense this subscription using the email template linked here.

Read more

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Most of the AI infrastructure boom isn’t actually buying compute. It’s buying memory bandwidth.

Look at the market right now: SK Hynix and Micron are completely sold out of their 2026 HBM capacity. Cerebras is pushing a $22 billion IPO built almost entirely around high-memory inference chips. Google just split its 8th-generation TPUs into completely separate training and inference lines.

Hundreds of billions of dollars are being deployed based on a single, massive assumption: that AI generation will always be bottlenecked by the exact same things it is today.

But if you look closely at the research, that assumption is already starting to rot. Diffusion models are rapidly rising out of the image-generation niche and seeing aggressive, mainstream use across text and reasoning workloads.

(PS: if you want to dig into why Diffusion Models are the future, from an engineering/technical perspective (including why companies are already investing in them, we covered it in this article (Prediction 2 and 3). Since I don’t want to repeat stuff, this article will take the rise of Diffusion Models as a given. The image below is a good illustration of how diffusion models unlock a deeper category of reasoning. .

This is not a trivial algorithmic swap. Diffusion fundamentally alters the AI paradigm. It aggressively rewrites the underlying math of how models are served, flipping the hardware bottlenecks and turning the established inference market on its head.

In this article, we will cover:

• The Dependency Graph: Why standard autoregressive LLMs are structurally trapped by memory constraints.

• The Bottleneck Inversion: How diffusion models shatter the sequential lock and shift the hardware paradigm.

• The Algorithmic Timeline: The math required to make language diffusion financially viable.

• Inference as Search: How diffusion makes complex, inference-time verification cheap, threatening the $2 billion base-model moat.

• The Vendor Breakdown: Which hardware architectures (NVIDIA, AMD, Groq, Apple, etc.) are actually positioned to survive the shift to compound diffusion workloads.

Let’s play together.

Executive Highlights (TL;DR of the Article)

• The Autoregressive Memory Trap (Section 1): The entire AI infrastructure supercycle is a desperately expensive patch for data movement. Because autoregressive models generate sequentially, operations have an arithmetic intensity of roughly 1 FLOP per byte. A modern tensor core needs nearly 300 FLOPs to stay fed. This means your $40,000 GPU executes at under 1% of peak compute while waiting for gigabytes of history (the KV cache) to stream across the chip.

• The Bottleneck Inversion (Sections 2 & 3): Diffusion models refuse left-to-right generation, refining entire sequences in parallel. This changes the underlying math from bandwidth-starved matrix-vector operations to massive matrix-matrix multiplications. The KV cache tax vanishes, and the bottleneck flips to raw compute. This is a massive tailwind: hardware generations like Blackwell and Rubin have compounded FLOPs at 3x every two years, while memory bandwidth has stalled. The industry accidentally built the perfect silicon for diffusion workloads.

• The Algorithmic Timeline & Step-Count Constraints (Section 4): Perfect hardware is irrelevant if the math requires too many steps. Image generation collapsed its step count using continuous math, but language operates in discrete jumps, trapping text diffusion in a 4-to-16 step regime. However, software scheduling is proving cheaper than raw parameter scale. Simply forcing a model to unmask logic premises before conclusions (LogicDiff) spiked reasoning scores by near 40 points without changing base parameters, threatening the hyperscaler gigawatt-compute thesis.

• Inference as Search & The Verifier Moat (Section 5): Because early denoising steps just build coarse shapes, multiple search trajectories can share the exact same early path. Running 4 candidate branches that share 40 of 50 steps yields a 4x quality search for a 1.6x compute tax. Branching AR is financially ruinous; branching diffusion is cheap. This unbundles the AI market: value migrates from the $2B base model to the companies building elite, proprietary verifier suites.

The Vendor Landscape (Section 6): Production diffusion is not a clean loop; it’s a chaotic compound pipeline (Denoiser + Verifier + VAE).

• NVIDIA: Retains its moat because CUDA is the only environment flexible enough to handle this “swamp” of switching models and search orchestration.

• AMD: Massive HBM capacity acts as the perfect hedge against the massive activation memory spikes required for video diffusion and DiT workflows.

• Groq & Pure ASICs: Brilliant for streaming deterministic AR tokens, but deeply vulnerable to the inter-chip latency required to orchestrate complex, backtracking search trees.

• Apple & Qualcomm: Positioned to completely strip-mine edge volume the second local step-counts collapse, separating the volume from cloud revenue.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

Section 1: Why Large Language Models Struggle with Decoding

Pick any fast frontier model — Claude 4.5 Haiku, GPT-5 Mini, Gemini 2.5 Flash. Look at their tokens-per-second at batch size one. These are the fastest autoregressive LLMs in the world, running on $40,000 silicon with peak FP8 throughput in the petaflop range. Yet they generate text at roughly the speed an undergraduate types.

To understand why, you have to look at the dependency graph.

Autoregression is strictly sequential. Token 200 cannot exist until token 199 is sampled. Because of this lockstep dependency, the model cannot process its layers in parallel across the sequence. To produce a single token from a 70B-parameter model, all 140 GB of weights must stream from High-Bandwidth Memory (HBM) into the chip’s SRAM, execute a tiny matrix-vector multiplication, and leave.

This operation has an arithmetic intensity of roughly 1 FLOP per byte of data moved. A modern Hopper or Blackwell tensor core needs nearly 300 FLOPs per byte before it stops starving. So when an LLM generates text for a single user, the GPU executes at under 1% of its theoretical peak compute. The other 99% of the silicon waits for data to arrive. The constraint isn’t how fast the chip can multiply. It is how fast the chip can be fed.

Once you understand that data movement is the true bottleneck, the entire AI infrastructure stack reveals itself as a series of patches for this exact flaw.

Take the KV cache. If you are already memory-bound, the logical fix is to avoid recomputing the past. The KV cache stores the attention states of previous tokens so each new token only adds to the sequence. But this trade creates a massive residency tax. The longer the context, the more memory capacity it eats.

To offset that capacity tax, the system has to amortize the weight-streaming cost across multiple users. That is why PagedAttention and continuous batching frameworks like vLLM exist. They batch requests so the 140 GB of weights stream through the chip once to serve 64 users instead of one. Throughput scales linearly — until the KV cache for those 64 concurrent users blows out the HBM capacity entirely.

When the software patches hit a wall, the hardware roadmap panics. HBM4 is a bandwidth upgrade. NVLink 5 at 1.8 TB/s per GPU is an interconnect upgrade. Multi-billion-dollar photonic interconnect acquisitions are not compute upgrades. They are data-movement upgrades designed exclusively to keep the tensor cores from starving.

But the physical reality of chip manufacturing is asymmetric. Adding denser compute units and low-precision formats like FP4 is relatively cheap. Scaling memory bandwidth and capacity is economically brutal.

Autoregressive inference sits on the wrong side of that divide. You can pour infinite FLOPs into a server, but if the workload requires dragging gigabytes of history across the chip for every single token, the math will never work. This is why Diffusion Models have been attracting so much attention.

Section 2: How Diffusion Models Break the Sequential Lock in AI

Diffusion language models escape the memory trap by simply refusing the premise of left-to-right generation.

Instead of predicting the next word, they start with a block of noise scaled to the target output and refine the entire sequence in parallel across a handful of forward passes. Every position is in flight at once. Token 200 never waits for token 199.

That single architectural shift shatters the dependency graph. Because a diffusion forward pass operates on hundreds or thousands of tokens simultaneously, the underlying math changes from bandwidth-starved matrix-vector operations to massive matrix-matrix multiplications. The arithmetic intensity spikes into the hundreds of FLOPs per byte — easily clearing the threshold required to actually keep a modern tensor core fed.

This is the bottleneck inversion worth noting: autoregression is memory-bound, while diffusion is compute-bound.

(worth stressing again — High Bandwidth Memory, HBM, is extremely expensive to build, hence the memory wall that has been choking AI. More on this in the next section).

Obviously, this means that the model does more raw work per output token, and it repeats that work across multiple refinement steps. But it pays that tax using the exact resource the hardware has in massive surplus.

We already established that Diffusion Models are fast, and they have some structural advantages with the hardware paradigm. But what about their performance? It’s still a bit early to tell, but the results look extremely promising:

Assuming this will continue to scale (and we’ve already broken down the research for this in previous works), we should see them close the gap in the next 2 years (Gemini 3 already started incorporating Diffusion in its model, so they’re already touching the frontier). This means more and more hardware providers will have to start accounting for diffusion models and their more matmul-heavy workloads.

Let’s explore that sentence next in more detail.

Section 3: The Hardware Diffusion Needs (and Why the Industry Already Has It)

If diffusion is going to take over a meaningful share of inference, we have to look at what it actually demands from the silicon.

To understand what this architecture needs to work well, we look at the math.

• A forward pass processes hundreds or thousands of tokens simultaneously, creating massive matrix-matrix multiplications. Compute becomes the primary axis of growth: the faster the chip multiplies, the faster the model runs.

• Given the size of the MatMuls, we also need Diffusion hardware to be good with quantization. Most modern hardware handles this fine, and it will be helped by the fact that, because the generation process is an iterative refinement loop, it naturally absorbs precision noise across steps.

• When it comes to memory, Diffusion drops the bandwidth tax. There is no KV cache to stream and no per-token residency penalty. It does require memory capacity for activations — which becomes a real wall when scaling to video — but for text, that is a far weaker constraint than the bandwidth trap autoregressive decode hits at batch size one.

The net result: the ideal architecture for this workload is heavy on compute, native in FP4, modest on bandwidth, and generous on activation memory. Surprisingly, this maps wonderfully to the hardware the industry has been building.

Every accelerator generation since Hopper has been an escalating compute story. Blackwell hits 20 PetaFLOPS of FP4. Rubin is targeting 50. Node shrinks and new precision formats have led to FLOPs compounding at 3x every two years. The only problem? The memory bandwidth needed to feed it has grown at half that rate.

The moment your model’s data spills over the tiny pool of super-fast on-chip memory, the system punishes you. Hard. “When data size exceeds GPU memory capacity, the data must be migrated repeatedly between the CPU and GPU, either manually or automatically. However, manual migration can be laborious for programmers, and it is infeasible for irregular workloads because the data access pattern is unpredictable. On the other hand, demand paging approaches (e.g., NVIDIA Unified Memory [120]) can automatically manage data movement, but it can significantly degrade performance due to high page fault-handling latency and limited PCIe BW [46, 121, 52]. This overhead can be particularly severe for irregular workloads since prefetch/eviction policies become ineffective [14]. For example, the runtime of bfs can increase by ∼4.5× with only 125% oversubscription (i.e., exceeding memory capacity by 25%) compared to when the GPU is not oversubscribed.”

Since my brother also reads this newsletter, here is a summary so that he can follow along: Compute go up fast. Good. Memory go up, but not so fast. Bad. Low Memory make algorithms run slow. Much money lost to GPUs.

Why has memory been scaling so slowly? HBM is pinned by physics, supply consolidation, and packaging constraints. This is not a hardware newsletter, so here is a quick summary below:

This is why increasing growth in compute haven’t scaled hardware utilization. The memory wall Pepe’s the fuck out of any deployment.

For diffusion, this set of conditions is a massive tailwind for all the reasons mentioned earlier (faster compute, good w/ quantization, and less hti by memory). The billions in capex committed through 2028 — from Blackwell to Rubin, MI400 to MI500 — is a roadmap of compute-axis improvements that will stall on AR decode but compound beautifully on diffusion.

All this means is that the blockers for diffusion going forward aren’t in the silicon since the chips and manufacturing constraints are pretty friendly to Diffusion. The real bottlenecks are upstream: maintaining frontier reasoning quality, and solving activation memory at video scale.

But most importantly, diffusion’s economics aren’t gated by how fast each step runs on the hardware. They are gated by how many steps you have to run in the first place.

Let’s look at the math behind that next.

Section 4: How the Algorithmic Layer Dictates the Hardware Timeline

Let’s pause and anchor exactly where we are in the stack, because this is where the money actually changes hands.

We know the architecture fixes the memory trap. We know the hardware industry accidentally built the perfect silicon for it.

But let’s be blunt: having the perfect chip is completely irrelevant if the math running on it is a nightmare. Hardware is just a dumb, incredibly expensive rock. The algorithm — the actual math telling the rock what to do — is the gatekeeper.

If diffusion requires 1,000 forward passes to generate a single image, nobody cares how perfectly it aligns with the silicon. At 1,000 steps, the unit economics are dead. For diffusion to actually threaten autoregression and shift billions of dollars in hardware spend, the step count — the Number of Function Evaluations (NFE) — had to collapse.

It already happened for images.

In 2020, diffusion ran on stochastic math (SDEs). Because random noise accumulated at every step, the model had to take tiny, excruciatingly slow steps to keep the image from blowing up. But a year later, the math flipped. Researchers realized they could use deterministic Ordinary Differential Equations (ODEs) instead, stripping out the random noise and allowing massive jumps across the generation path. The 1,000 steps collapsed to 50 for free, and the Midjourney and Stable Diffusion API economies were born overnight.

But language is a completely different financial reality.

Images solved the step-count collapse because the math is forgiving. You can draw a smooth, continuous mathematical line between two pixel colors. You cannot draw a smooth curve between the word “Apple” and the word “Murder.” Masked language models operate in discrete, harsh jumps across a vocabulary space.

Because of this, the continuous distillation techniques that made image generation real-time fail spectacularly on text. When researchers tried porting consistency distillation to LLaDA-8B, the model bled 6 full points of accuracy on the GSM8K math benchmark.

Because discrete distillation is struggling, language diffusion is currently stuck in the 4-to-16 step regime. That is fast enough to beat autoregression in a server rack, but it fundamentally delays the “frontier reasoning on an iPhone” narrative. Until researchers crack discrete space distillation (or we get a way to make language concepts operate in the continuous space, which imo is more useful) — which is likely 18 to 36 months out — Apple and Qualcomm are stuck waiting. The compute required for those 4 to 16 steps ensures that the hyperscalers will maintain absolute, walled-garden dominance over text generation. The datacenter monopoly holds.

But inside those datacenters, the bottlenecks are shifting in ways that threaten the current capex consensus.

For a while, we assumed diffusion language models were bottlenecked by raw parameter scale. We thought they just needed bigger training runs to reason better. It turns out they just had terrible time management.

Standard masked diffusion unmasks tokens randomly. This means the model routinely commits to a conclusion before it has even unmasked the premises it needs to get there. It’s trying to solve the end of the maze before looking at the start. In 2025, the LogicDiff paper fixed this with a microscopic, 4.2M-parameter classification head that simply forced the model to unmask premises first and conclusions last.

The result: LLaDA-8B jumped from 22.0% to 60.7% on GSM8K. A near 40-point spike without changing a single base parameter.

If you can buy 40 points of reasoning with a dirt-cheap scheduling hack instead of a $2 billion training cluster, the entire hyperscaler capex thesis starts looking very fragile. Software efficiency is quietly undercutting the necessity of building gigawatt, nuclear-powered compute clusters just to achieve frontier logic.

Lastly, it’s worth studying LLaDA-MoE, which proved that routing tokens through expert subnetworks works beautifully for diffusion text models. But MoE inherently requires streaming different expert weights from memory for every routing decision. It isn’t the relentless KV-cache tax of autoregression, but it is an expert-loading tax that brings the memory bandwidth constraint back into discussions.

Section 5: How Diffusion Turns Inference into a Search Problem

Once diffusion eats a meaningful slice of inference, the serving stack needs to be rethought deeply.

We aren’t just swapping out the model weights and keeping the same infrastructure. We are dealing with a structural mutation in what inference actually is. We are moving from a clean, sequential generation pass to a chaotic, branching search problem.

To understand why, look at what just happened to language models.

Inference-Time Scaling Arrives in Diffusion

The most important paradigm shift of the last two years was the realization that throwing compute at inference beats just training a fatter model. OpenAI’s o1 and DeepSeek-R1 proved that the model that “thinks longer” wins.

Diffusion is about to do the exact same thing, but the mechanism is entirely different. Instead of “thinking longer” by generating more internal reasoning tokens, diffusion searches wider.

You don’t just run the denoiser once, roll the dice, and hope for the best. You generate a swarm of candidate trajectories, and you use a secondary model — a “verifier” — to judge them mid-flight. This was shown to us in Google’s “Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps”. By investing in verifiers and better search, they were able to mog simplistic NFE scaling.