About a week ago, I was offered a multimillion-dollar role by a very wealthy investor. They had been reading this newsletter and were impressed by the range of subjects we cover. Model architecture, infrastructure, hardware, economics, company strategy, and technical areas I had sometimes never worked on before writing about them (such as this deep dive into Weka and the memory bottleneck in AI)—

More than anything else, they were interested in the speed. How was I able to read so much about very different fields, and build a deep enough map to write deep dives on them so regularly? Yes AI helps, but they wnated to know how to use it most effectively to be able to assess multiple emerging industries and make predictions accordingly.

This article is a breakdown that process.

Most people waste enormous amounts of time when they try to understand a new technology. They begin with papers, terminology, architectures, benchmarks, and implementation details. Weeks later, they know a lot more, but they may still be unable to answer the basic questions that matter.

To me, knowledge is about decision making power: it’s most useful when you can use it to make meaningfully decisions to move your agenda. This means tha you have to be able to answer questions such as

• Why does this technology exist now?

• What constraint does it remove?

• Who benefits from it?

• What could prevent it from succeeding?

• Does it change anything important enough for you to act?

and many others quickly so you can make the important decisions. Such questions tell me which technical details deserve attention and which ones can be ignored for now.

This creates an iterative process. You learn enough to form an initial view. That view tells you where the uncertainty is. You then go deeper only where resolving that uncertainty could change your decision. This process builds a much memorable (and more actionable) understanding of what you’re studying.

Let’s break it down in depth.

To access the full article—and all premium breakdowns going forward/written prior—upgrade to a premium subscription below.

If you believe deep insight deserves support, become a premium subscriber to allow me to keep doing the same.

Flexible pricing available—pay what matches your budget here.

Most companies offer learning or professional development budgets. You can expense this subscription using the email template linked here.

Read more

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

(Help me out by sharing this article on LinkedIn here)

Thousands of engineering teams are burning margin, forcing heavy frontier models to handle basic production tasks because smaller, cheaper open-weight alternatives can’t deliver reliable outputs. And on the surface, they’re completely right.

A model like Qwen3–4B — one of the best small open-weight architectures on the market — gets basic arithmetic wrong almost all the time, landing a correct answer just 32% of the time. With odds like that, you might as well hit the roulette and gamble like a proper degenerate.

But if you dig into the failure data, an interesting stat emerges — it actually computes the correct answer inside its internal states an impressive 80% of the time. Somehow, the model gets to the correct answer and then veers off track. This tells us that the issue isn’t in the base intelligence (the model has the knowledge somewhere), but in navigation (discovering the knowledge reliably and then stopping). In fact, when we tracked whether models finished their responses naturally versus hitting the token limit, every single naturally completed answer was correct (across 500 generations). Wrong answers are destinations that our model failed to reach, not destinations that our model lacked in its map (full EOS analysis in the appendix).

This means that models get trapped in an optimization loop — formatting, restructuring, and elaborating indefinitely — until they brick the run before ever stating the final result. What we need is a system that reliably find and unlocks the knowledge already hidden in the language model.

Last time I wrote about Latent Space Reasoning, the system we broke down required a pretty sophisticated setup requiring trained judges, specialized aggregation, and a universal embedding projected space. Since then, our team at Irys has been exploring simpler ways to unlock the hidden knowledge in models that bring the power of latent space reasoning to less technical teams. And our results have been very compelling — 

• Injecting just two random vectors into the model’s embedding space before it starts generating forces it to break the lock-in, spiking that 32% success rate straight to 51.6%. Re-read that number — we almost doubled accuracy by adding 2 random tokens.

• Bump that to ten random vectors with a basic plurality vote, and the success rate hits 72%. (Use plurality, not majority voting — majority actually performs worse than baseline when individual seed accuracy is below 50%. Details in the appendix.)

• At ten vectors, every single task in the benchmark gets solved by at least one seed — 100% oracle coverage extracted purely from random noise.

• On a Redis debugging task, the baseline model spit out 14 incoherent words. With two random tokens, every single seed produced a complete 650-word diagnostic plan.

• On 12 legal reasoning tasks, the top seed beat the baseline on 11 of them.

• Smaller models with perturbation beat or matched larger models of the same family more than half the time, proving that this technique unlocks a new axis of capabilities w/o needing increased hardware or training investment.

These results allow us to get better performance from smaller models than from bigger models AND for lower cost (small model + multiple runs is much cheaper than big models).

^^Experiment details on GitHub. Once again, our system unlocked performance that parameter scaling did not.

Given how impactful our findings are for democratizing intelligence for everyone, we’ve open-sourced what we could on our latent space reasoning repo — github.com/dl1683/Latent-Space-Reasoning. While the effects of our research are statistically significant (McNemar p < 0.001 across two separate models), we recognize how small our sample size is. This is where we’d love to collaborate with the rest of our open community to scale up the experiments and look into different outcomes.

Executive Highlights (tl;dr of the article)

The finding: A Qwen3–4B model computes correct arithmetic answers 80% of the time but only states them 32% of the time. The model isn’t failing to reason — it’s failing to finish. It gets trapped formatting answers it already computed.

The fix: Injecting 2 random vectors into the model’s embedding space before generation breaks the lock-in. No training, no fine-tuning, one line of code. Accuracy jumps from 32% to 51.6% on a single seed, 72% with plurality voting across 10 seeds, and 100% oracle coverage — every task solved by at least one seed.

It generalizes. On a Redis debugging task, baseline output was 14 incoherent words. With 2 random tokens, every seed produced a 650-word diagnostic plan. On 12 legal reasoning tasks, the best seed beat baseline on 11. On incident response, evolved vectors surfaced honeypot deployment, MITRE ATT&CK analysis, and HSM credential rotation from a 4B model that baseline could only get “rotate credentials, check logs” out of.

Direction doesn’t matter. Random noise and carefully optimized projections produce identical results (Mann-Whitney p = 1.000). The perturbation provides energy, not information. Leading candidate explanation: stochastic resonance. Three independent groups (Kim et al., Shi et al., Pfau et al.) reached the same conclusion through different methods.

Small models + perturbation beat bigger models. Scaling from 4B to 14B bought 4 percentage points on our arithmetic benchmark. Changing how we interrogated the 4B model bought 40. The 4B runs on a $429 GPU at 200 tokens/sec. The 14B needs a $1,999 GPU at half the speed. At cloud rates, 10 perturbation seeds cost $0.009/query versus $0.45 for GPT-5.6 Sol thinking tokens — 56x cheaper.

Practical guardrails. Use 8-bit quantization, not 4-bit (4-bit nearly eliminates the effect). Use plurality voting, not majority (majority performs worse than baseline). If your model already scores 75%+ on a task, perturbation can hurt mean accuracy — it helps stuck models, not cruising ones.

What doesn’t work. Our original verbosity claim was a code bug. The scorer is barely trained (broken on 9/12 legal tasks). DeepSeek showed a negative mean effect at high baseline. The data breach task exposed hallucination when the model lacked the knowledge entirely. Sample sizes are modest: 25 arithmetic, 5 planning, 12 legal tasks.

What’s next. Temperature comparison (the test that determines if this is a new technique or a mechanistic insight), gated attention probe, position-shift ablation, scorer improvement. Full pipeline preregistered and open-source.

The thesis: The industry’s default answer to “my model isn’t good enough” is a bigger model. Our results say that for a meaningful class of tasks, the capability was already in the small model — the bottleneck was the inference path. Lifting the ceiling gets the headlines. Raising the floor is where the new value gets created.

Everything is open-source: github.com/dl1683/Latent-Space-Reasoning

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

How Standard Formatting Might Be Making Small Models Fail

The standard narrative (small open-weight model hallucinations/mistake are a result of their lower intelligence) is true quite often, but as we’ve derstand why, let’s quickly remind ourselves of what models are doing when they do inference.

Language models generate text autoregressively — one token at a time. Because each token becomes part of the hard context window for every subsequent decision, the first token constrains the second, and the first two dictate the third. In our research across multiple models and sizes across multiple domains, we found that the first 20 generated tokens tend to disproportionately impact the AI context generation (even in very long context generations; research shared in our comprehensive proprietary AI research map available to all founding members of the Chocolate Milk Cult over here). This leads to the “autoregressive lock-in” phenomenon we have discussed extensively in the past.

Image Source

In practice, this will manifest in several ways:

1. Models will spend tokens on the formatting and run out of their token budget, even when they know the answer.

2. Models might follow a reasoning path, realize it’s no good (either themselves or by your injection), but the false path will STILL influence the outputs.

3. The models you pay for by token are occasionally setting their token budgets on fire to format answers they’ve already computed. We can call this structural failure formal presentation mode. Left to its own devices, the model burns through its footprint producing immaculate headers, enumerated steps, and clean LaTeX expressions. In other words, it commits to a visual template and mindlessly fills it in — performing the aesthetic of math rather than actually doing the math (anyone getting shades of pulling up a terminal in front non coders to seem “hackery”).

Knowing that, we decided to compute the geometry of the model’s search space (nerd stuff, not the most relevant here; check next section, GitHub experiments or gimme a shout if you want deets) and then used that geometry to disrupt that default trajectory by prepending a couple of random, noisy tokens to the embedding space. This pushed the models into exploratory computation mode — a messy, stream-of-consciousness scratchpad.

Look at the text execution on the arithmetic problem (45 + 23) * 17 - 89:

Without prefix (Wrong — bricks on the formatting template):

Let me solve this step by step.
Step 1: Calculate 45 + 23 = 68
Step 2: Multiply by 17: 68 x 17 = ...

With prefix (Correct — hits the exact payload in half the footprint):

ok so first 45+23 thats 68, then times 17... 68*10=680, 68*7=476, so 680+476=1156, minus 89, 1156-89=1067

The random prefix doesn’t inject any new knowledge into the models (it can’t). Instead, it seems to bypass a lot of the hacking behaviors injected from the alignment process and forces the system to compute directly.

Ever since Wei et al. dropped their foundational Chain-of-Thought paper, the consensus has been that intermediate steps boost accuracy. But the ecosystem completely missed the real bottleneck: chain of thought worked best when the ideas occur in overlapping knowledge regions—

This proved that intelligence was a “search through the latent space” issue (in hindsight, this paper had much bigger implications than any of us realized). It also gives us an interesting framing for the ubiquity of the formal presentation mode: the alignment hacking present in modern post-training pushes form over substance. The random token perturbation allows us to attack this gap by undercutting the formatted alignment spaces and leads to a system that wastes 800 tokens on beautifully formatted, incomplete garbage (base LLM), losing to a model that spends 400 tokens on a messy, complete solution (our perturbed variant).

Take one second to appreciate how insane this is. The most common engineering band-aids either operate way too late after the trajectory is already committed (like temperature sampling or best-of-N), require heavy gradient updates (fine-tuning), or try to extract diversity during token selection rather than the computation step (self-consistency). We avoid all of that by directly attacking the root of this issue, and are able to get significantly better results out of this with a solution that’s practically free.

In simple words, change where the model starts computing and how it searches a predefined space, and you can dramatically update its results. So how exactly does that work? We will look at that next.

What If You Shift Where the Model Starts Computing?

In the vein of data augmentation techniques like RandAugment, we take two vectors of random numbers, scale them to match the model’s native embedding magnitude (an RMS of ~0.022 for Qwen3–4B), and prepend them as positions 0 and 1 before your prompt tokens. The intuition here was similar to what made RA so effective — the randomness should inject a level of diversity into your search space, ensuring better performance.

Our thrifty twosome now sits at the start of every model call, shifting the computational trajectory through every subsequent layer from the very first real token onward. Apparently, this randomness is all you need to start unlocking model performance. We tested this approach against a simple, uncalibrated version of the latent space reasoning pipeline we had shared earlier:

• Optimized projections vs. Random noise: Mann-Whitney p = 1.000. The distributions are identical.

• Euclidean vs. Hyperbolic mutations: p = 1.000. Same result.

In other words, this setup provided the same level of geometric exploration as a more complicated exploration framework. This leads to our next interesting question — why would random tokens make LLMs better?

Why Pure Noise Unlocks AI Model Reasoning

Truth be told, no fucking idea. That’s one of the reasons I’m open-sourcing this, so that one of you can figure this out while I take all the credit.

Until that happens, I’ve had some researchers from Irys and the larger Chocolate Milk Cult looking into this. Our leading candidate explanation is stochastic resonance — the physical phenomenon where adding noise to a non-linear system amplifies weak signals that would otherwise fall below an activation threshold. This is an idea we explored over here when looking into why Brains are so much more efficient than LLMs by using Sub-Landauer patterns (patterns that encode some useful information, but are not given “full energy” so they only activate when a bunch of Sub-Landauer patterns trigger together to collectively overcome the energy threshold) — 

Mapping this to NNs, standard transformers are like a non-linear threshold system: the correct reasoning path exists in the model’s parameters but sits just below the greedy selection boundary. Random embedding noise provides the activation energy needed to push the computation over that threshold. Because the noise provides energy rather than information, optimizing its direction adds zero value (which is why directional perturbations without tweaking magnitudes didn’t present much utility).

We are currently running experiments to isolate stochastic resonance from competing accounts (attention redistribution, positional offsets from shifted RoPE IDs, and attention-sink disruption). If you want in, just pull up and start running some experiments yourself.

In the meantime, we have some reason to be confident in our assessment of the situation since independent teams are reaching the exact same conclusion across different setups:

• Kim et al. (“Random Soft Prompts”): Showed random, unoptimized embedding vectors match the accuracy of heavily trained soft prompts on math tasks.

• Shi et al. (“Meaningless Tokens”): Found inserting sequence strings of meaningless discrete tokens consistently boosts reasoning via activation redistribution.

• Pfau et al. (“Let Me Speak Freely”): Demonstrated that simple repeated filler tokens (“…”) allow hidden computation in the residual stream.

All 3 seem to be pointing at the same thesis — the perturbation matters much more than specific tuning. At least until we have a more rigorous map of the latent space itself.

Why Perturbation Changes the Unit Economics of LLM Reasoning

Why go through this effort? For the vibes? Streetcred? To find some way to spend my Codex tokens (unfortunately, that’s not true for me since I don’t get paid to tokenmax)?

Nah, it’s coz I’m a gareeb, and I need to find ways to make AI cheaper. A lot cheaper so that Irys company lunches can go from two packs of Oreo shared b/w all of us, to at least 4 Chipotle bowls for the whole team. So watch your dearest Dev Dev fight his way to put more meat in his mouth.

Turns out that even accounting for the 10 extra inference passes of the smaller model (+ the oracle), latent-space reasoning extracts far more capability per dollar without ever forcing you onto a bigger model. This changes your business in two meaningful ways — 

1. You can route more work to smaller models safely and get the same threshold of performance. This opens up a whole new class of latency-sensitive work that couldn’t be done before, while also driving down the costs of the existing agentic loops where you might be worried about costs.

2. The magnitude of the performance shifts means that you can deploy the same level of intelligence in a much cheaper class of hardware.

Both will massively speed up the Jevon’s Paradoxification of AI, ultimately massively upping the value generated by AI.

Following are some rough markers of short-medium ROI I put together across various scenarios (image is a summary, we’ll break them down)—

Raw Performance

On our arithmetic benchmark:

• Qwen3–4B scored 32% under ordinary greedy inference.

• Prepending a two-token perturbation raised that to 51.6%.

• Ten perturbed trajectories with plurality voting hit 72%.

• Qwen3–14B — 3.5 times the parameters — scored 36% (the small delta b/w 4B base and 14B isn’t surprising here since the arithmetic isn’t going to be novel for even the 4B model).

In other words, scaling the model bought four percentage points. Changing how we interrogated the small model bought forty. The 14B model isn’t even competing here. This becomes much more impactful when you consider the macroeconomics and the memory supercycle (companies like Micron and Nvidia are committing highway robbery b/c of how expensive High Bandwidth Memory has become in GPUs) and how LSR (even single pass with perturbation) gets us higher gains than scaling, w/o increasing the hardware requirements (the implications of this will be made clear in the next sections).

The Hardware Class Gap

Model size dictates the minimum hardware required to run the system. From Qwen’s official AWQ benchmarks:

• Qwen3–4B: 2.9 GB VRAM, 199.7 tokens/sec

• Qwen3–14B: 10.0 GB VRAM, 96.5 tokens/sec

• Qwen3–32B: 19.1 GB VRAM, 47.7 tokens/sec

The 4B model sits in a completely different deployment class, meaning that we get cheaper cards, more room for context, and double the throughput of 14B. How much cheaper? Take a look at the difference:

• RTX 5060 Ti (16 GB): $429 at launch

• RTX 5090 (32 GB): $1,999 at launch — before the rest of the rig

On Runpod’s current serverless rates, a 16 GB worker runs $0.58/hour versus $1.58/hour for the 32 GB tier. That is a 2.7x hourly premium for a larger model that generates at half the speed.

Factoring in current cloud rates, power draw, and the May 2026 US residential electricity baseline (18.44 cents per kWh):

• $1,000 local machine (16 GB): pays for itself after ~1,830 GPU-hours

• $3,250 local machine (32 GB): pays for itself after ~2,205 GPU-hours

At 240 GPU-hours per month, you are looking at your investments paying back more than a month earlier — 

Worth stressing here is that the lower upfront hardware costs enable market creation since the costs lower the barrier for participation.

Unit Economics vs. Frontier Models

Let’s now look at the calculations from a cloud costing perspective. Using standard market baselines (Qwen3–4B at 4-bit, 2,048 token sequence, continuous prefix noise at positions 0 and 1):

• 10 Perturbation Seeds ($0.008 per query): 10 parallel generation passes = 20,480 tokens at $0.40/M. The 2 prefix vectors add 0.097% overhead — pure rounding error.

• 10 Seeds + Scorer MLP ($0.009 per query): A 300K-parameter MLP evaluates candidate trajectories in one pass on short sequence embeddings, adding ~$0.001.

• GPT-5.6 Sol Thinking Tokens ($0.45 per query): OpenAI’s flagship bills thinking tokens at the output rate ($30.00/M). Mid-range queries burn ~15,000 thinking tokens before answering ($30.00 × 0.015).

Computing the difference, we get Sol being 56x more expensive than our system.

Take a second to consider what that means at scale. At 10,000 queries per day, that is $90/day versus $4,500/day. Over a month, you are comparing $2,700 to $135,000. At 50,000 daily queries, the perturbation approach stays under $15,000/month while Sol crosses $675,000.

Now, this is not meant to say that our system will replace Sol/big models everywhere. Ultimately, no matter how much we might wish, latent space reasoning can’t make existing models more knowledgeable. So in many tasks, a heavier model will simply outperform a lighter model, no question. Similarly, if your model already scores well on a task (75%+), perturbation can actually hurt mean accuracy since it disrupts trajectories that were already efficient. The oracle still improves performance, but the average gets pulled down by bad perturbations.

However, this also leaves a large gulf of techniques where the simpler models have the base knowledge, but they get stuck in unhelpful attention sinks they can’t avoid. And your constraints limit you from hitting bigger models. In that case, we believe that Latent Space Reasoning with Perturbations can help your model punch above it’s weightclasses.

Conclusion: The Overlooked AI Revolution

The market is currently pricing in only one economic revolution: pushing the capability ceiling. Frontier labs are burning billions to maximize peak capacity. As economic pressures mount, this will lead to higher prices for consumers as the labs have to justify their massive capex buildouts. We’re already seeing this manifest with the higher token prices in recent models (both directly in PPM and indirectly with models consuming more tokens for their tasks).

Take a second to consider that there is a second revolution: raising the floor. The smartphone didn’t win by being a faster laptop. It won by delivering a fraction of a laptop’s compute at a hundredth of the cost, completely changing the deployment envelope. That cost collapse birthed entirely new economies — ride-hailing, mobile logistics, the app ecosystem — because adequate compute became cheap enough to integrate a billion new participants.

Delivering adequate intelligence at a radically lower cost changes who gets to participate in the system. It makes problems economically tractable that simply never pencil out at frontier pricing. You cannot integrate the broader economy when operators are bleeding out on $135,000/month inference bills.

APPENDIX: Full Experimental Detail

A1. Arithmetic: Dose-Response, Controls, and Voting

Setup. Prefix-length sweep from 0 to 8 tokens. Qwen3–4B at 4-bit quantization. 25 multi-step arithmetic tasks (nested expressions, 3–6 sequential operations).

Dose-response. Baseline (0 tokens): 32.0%. 1 token: 42.7% (+10.7pp, n=3 seeds). 2 tokens: 51.6% (+19.6pp, n=10) — — the peak. 3 tokens: 44.0% (+12.0pp, n=10). 8 tokens: 44.4% (+12.4pp, n=10). McNemar p = 0.000015 at 2 tokens. Performance peaks at 2 and drops at 3 and 8 because of token budget competition: correct runs average 718 tokens, wrong runs hit the 1,024 ceiling and get truncated. At 8 prefix tokens, the model branches into multiple partial strategies and exhausts its budget before concluding any of them.

Controls. Zero embeddings (adding positions only): 36.0%, +4pp. Mean embedding (identical values at both positions): 36.0%, +4pp. Random noise (diverse values): 51.6%, +19.6pp. No chain-of-thought: +0pp. The effect requires both diverse prefix values and step-by-step reasoning. Remove either and it vanishes.

Voting at n=10 seeds. Plurality (most common answer): 72% on 4B, 56% on 8B. Oracle (best of 10): 100% on 4B, 80% on 8B. Majority (>50% agreement): 40% on 4B, 12% on 8B — worse than the 16% 8B baseline. Majority fails because when individual accuracy is below 50%, the threshold demands agreement that doesn’t exist among mostly-wrong seeds. Correct answers cluster (different trajectories converge on the same right answer). Wrong answers scatter (each truncation produces a different spurious value).

Caveat. The n=3 scout runs overestimated effects: Qwen3–4B dropped from 60% (n=3) to 51.6% (n=10). 25 tasks establish significance but not precise effect sizes.

A2. EOS Completion

Setup. 250 perturbation responses on Qwen3–4B, 250 on Qwen3–8B. Tracked whether each response finished with a natural end-of-sequence token or hit the 1,024 max token limit.

4B results. Finished + correct: 94 (37.6%). Finished + wrong: 0 (0.0%). Truncated + correct: 35 (14.0%). Truncated + wrong: 121 (48.4%).

8B results. Finished + correct: 32 (12.8%). Finished + wrong: 0 (0.0%). Truncated + correct: 40 (16.0%). Truncated + wrong: 178 (71.2%).

P(correct | finished naturally) = 1.000 on both models. Zero exceptions across 500 responses. Same pattern holds for baseline greedy decoding on this benchmark. EOS responses average 718 tokens. Truncated responses hit 1,024. Perturbation increases the EOS rate: 37.6% vs 24% at baseline. Tested on 25 arithmetic tasks only.

A3. Planning Tasks

Setup. 5 tasks: fraud detection system design, incident response, healthcare data platform, Redis cache debugging, Oracle-to-PostgreSQL migration. 3-way comparison: greedy baseline, random perturbation (5 seeds), evolved latent vectors (5 seeds). Qwen3–4B at 4-bit, 2,048 max tokens, greedy decoding. LLM judge scored on coherence, correctness, completeness, specificity, actionability.

Attention sink rescue. On Redis debugging, baseline collapsed to 14 incoherent words — — stopped generating immediately in a degenerate attention pattern. Every perturbation seed produced 650–710 word complete diagnostic plans. Likely mechanism: attention sinks (Xiao et al.) cause earliest token positions to accumulate disproportionate attention; under greedy decoding, degenerate early-position patterns propagate through every subsequent token. Random noise at those positions disrupts the pattern before it forms.

Evolutionary search. A 300K-parameter MLP scorer evaluates candidate latent vectors and guides an evolutionary population. On incident response, baseline produced “rotate credentials, check logs.” Evolved vectors produced honeypot deployment, MITRE ATT&CK lateral movement tracking, tiered HSM credential rotation, and immutable container rebuilds from verified base images.

LLM judge tally. Perturbation won 3/5, evolution won 2/5, baseline won 0/5. Sample size: 5 tasks.

A4. Legal Reasoning

Setup. 12 tasks across 5 categories: FTC unfairness, GDPR classification, disparate impact, SaaS contract review, acquisition due diligence, data breach triage, IP risk portfolio, negotiation leverage, regulatory response, contractor misclassification, corporate veil piercing, whistleblower retaliation. Same 3-way comparison. Qwen3–4B at 4-bit, 2,048 max tokens. Blind evaluation — condition labels stripped, order randomized, scored on Legal Accuracy, Analytical Depth, Practical Utility, Structural Quality, Completeness (1–10 each).

Oracle results. Best of 5 seeds beat baseline on 11/12 tasks (92%). Average lift: +1.6 points on a 10-point scale. Largest lifts: negotiation leverage 2.0 → 5.4 (+3.4), contractor misclassification 2.2 → 5.6 (+3.4), IP risk 3.6 → 6.4 (+2.8), whistleblower 3.2 → 5.6 (+2.4), FTC unfairness 5.2 → 7.2 (+2.0).

Qualitative examples. Contractor misclassification: baseline gave surface analysis. Best perturbation seed applied state-specific tests (California ABC test, New York economic reality test, Texas common law) with per-jurisdiction liability estimates. Corporate veil piercing: one seed identified “SubCorp appears to be a real operating business, not just a sham shell” — a Delaware Chancery Court doctrine element other outputs missed.

Boundary condition. GDPR controller/processor classification (the 1 loss): exceeded the model’s knowledge. Perturbation seeds consumed full 2,048-token budgets in thinking loops without substantive analysis.

Mean vs oracle gap. Most random seeds produce lateral moves or slight regressions. The gap between 92% oracle and mixed mean results is the scorer problem. Sample size: 12 tasks.

A5. Cross-Model and Quantization

Cross-model results on the 25-task arithmetic benchmark:

Qwen3–4B at 4-bit: 32.0% → 51.6% (+19.6pp), 100% oracle. 80% answer-anywhere at baseline, 32% stated. Perturbation barely changes answer-anywhere (80% → 82%) but jumps stated accuracy. Convergence aid — helps finish, not compute.

Qwen3–8B at 8-bit: 16.0% → 28.8% (+12.8pp, p = 0.000177), 80% oracle. 32% answer-anywhere at baseline, raised to 50%. Exploration aid — finds answers the model couldn’t compute before.

DeepSeek-R1-Distill-Qwen-1.5B at 4-bit: 76.0% → 74.4% (-1.6pp), 100% oracle. Near ceiling already. Perturbation disrupts efficient trajectories more often than it finds better ones.

phi-2 unquantized: 12.0% → 18.7% (+6.7pp), 28% oracle. Low ceiling limits gains.

Quantization. Qwen3–8B at 4-bit: +1.3pp (null result). Same model at 8-bit: +12.8pp (p = 0.000177). Working hypothesis: 4-bit rounds to 16 distinct weight values, too few for perturbation to create meaningfully different trajectories. 8-bit (256 values) preserves enough diversity. Based on one model comparison — may not generalize.

A6. Limitations, Failures, and Next Experiments

Verbosity bug. First analysis claimed perturbation increases verbosity. Artifact of a field-ordering bug (commit 6c69284). Perturbation actually produces 3% fewer words on Qwen3–4B.

Data breach hallucination. All 11 outputs (baseline, perturbation, evolution) fabricated legal content — invented statutes, fictional agencies, incorrect deadlines. Task exceeded the model’s knowledge.

Broken scorer. 300K-parameter MLP was non-deterministic on 9/12 legal tasks. Fix applied (deterministic projection with fixed seed), needs clean re-run. The 48pp gap between 100% oracle and 51.6% mean accuracy on arithmetic is the scorer’s failure to select the correct seed.

Sample sizes. 25 arithmetic, 5 planning, 12 legal. Statistically significant (McNemar p < 0.001 on two models) but effect sizes will shift with scale.

Next experiments (preregistered Phase A pipeline):

Temperature comparison. Head-to-head: prefix perturbation vs temperature sampling vs prompt rephrasing, same tasks/models/compute budgets. If temperature at n=10 matches perturbation’s oracle and mean, the contribution is mechanistic insight rather than a new technique. If embedding-level perturbation accesses trajectory classes token-level sampling can’t, the technique claim holds.

Gated attention probe. Qwen3.5–4B uses hybrid gated DeltaNet + gated attention layers that eliminate attention sinks via post-softmax sigmoid gating. If perturbation disappears on gated architectures, the effect depends on attention sinks specifically. If it survives, the mechanism operates below the attention layer.

Position-shift ablation. Soft prefixes shift RoPE position IDs for downstream tokens. Decomposes the total effect into: position offset, embedding diversity, and their interaction.

Scorer improvement. Domain-specific training, more sophisticated architectures, learned aggregation. The unresolved problem is selecting the correct candidate from among seeds.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

is one of the best people writing about modern AI research. He has a rare ability to take a fast-moving technical field, read virtually everything happening inside it, and explain the important ideas without either dumbing them down or burying the reader beneath acronym soup. Cameron is a PhD and Senior Research Scientist at Netflix, and his newsletter, Deep (Learning) Focus, is consistently worth reading for anyone who wants to understand what is actually changing underneath the weekly flood of model releases.

Deep (Learning) Focus

I contextualize and explain important topics in AI research.

By Cameron R. Wolfe, Ph.D.

This article is an excellent example of why. Reinforcement learning with verifiable rewards has produced enormous gains in areas like mathematics and coding, where answers can be checked automatically. The only problem is that most valuable human work does not come with unit tests (the verification problem we’ve talked about a few times over here). Research, medicine, writing, strategy, and general helpfulness involve several dimensions of quality, legitimate disagreement, and answers that may be partly right in very different ways. Rubric-based rewards are one of the most serious attempts to bridge that gap by turning vague human judgment into explicit criteria that models can be evaluated—and eventually trained—against. Cam walks through the entire emerging field, from basic LLM judges and static checklists to grounded, evolving rubrics that adapt as models discover new behaviours and new ways to game their rewards. Read this not merely as a survey of several recent papers, but as a guide to a much larger question: how do we specify what “good” means clearly enough for an AI system to learn it without simply learning to fool us?

As you read, consider:

1. What does “good” actually mean for the AI task you are building, and can you decompose it into separate criteria?

2. Which requirements are flexible preferences, and which failures should invalidate an answer regardless of everything else it does well?

3. What grounds your rubric: expert knowledge, reference answers, retrieved evidence, human preferences, or simply another model’s judgment?

4. How would you distinguish genuine capability improvement from a model becoming better at exploiting its judge?

5. Which desired behaviours conflict with one another, and should they be trained together or through separate stages?

6. When should a rubric remain fixed for consistency, and when should it evolve in response to new model behaviours and failure modes?

7. Is the real bottleneck in your AI system model intelligence—or your inability to clearly specify, measure, and reward the behaviour you actually want?

8. How would you rebuild AI to make verification of quality a first class primitive instead of a nice to have?

If you dig this work, make sure you share it, or follow Cam on X and LinkedIn!

Many of the recent capability gains in large language models (LLMs) have been a product of advancements in reinforcement learning (RL). In particular, RL with verifiable rewards (RLVR) has drastically improved LLM capabilities by using rules-based, deterministic correctness checks (e.g., passing the test cases for a coding problem) as a reward signal. Deterministic verifiers allow RLVR to provide a reliable reward signal that is more difficult to exploit compared to the neural reward models that were traditionally used for RL with LLMs. Such improved reliability has made stable RL training possible at scale, enabling the creation of powerful reasoning models with extensive RL training. Despite these benefits, verifiable rewards also have limitations—the same properties that make RLVR reliable confine it to domains with clean, automatically-checkable outcomes.

“While lots of efforts have been paid on RLVR, many high-value applications of LLMs, such as long-form question answering, general helpfulness, operate in inherently subjective domains where correctness cannot be sufficiently captured by binary signals.” - from [3]

Many important applications (e.g., creative writing or scientific reasoning) are not verifiable, making RLVR difficult to apply directly. To address this gap, we need reward signals that preserve RLVR’s scalability and reliability while still working in non-verifiable settings. Rubric-based rewards are a promising step in this direction: they decompose desired model behavior into structured, interpretable criteria that an LLM judge can evaluate and aggregate into a multi-dimensional reward. By creating prompt-specific rubrics that specify the evaluation process in detail, we can derive a more reliable reward signal from LLM judges and, therefore, use RL training to improve model capabilities even in highly subjective domains. For this reason, rubric-based RL training, which we will cover extensively in this overview, has become one of the most popular topics in current AI research.

Join 60,000 others who use Deep (Learning) Focus to understand AI research. Consider a paid subscription if you would like to help support the newsletter.

From LLM-as-a-Judge to Rubrics

Before learning about how rubrics can be used for RL training, we need to build a background understanding of LLM-as-a-Judge and the different setups that can be used to evaluate open-ended problems with an LLM. At the end of the section, we will connect these ideas to rubrics and RL training by overviewing existing RL training techniques and how they are being extended to non-verifiable domains.

LLM-as-a-Judge

Prior to the LLM era, many evaluation metrics used for generative tasks (e.g., BLEU or ROUGE) were quite brittle. These metrics use n-gram matching (or embedding-based matching as in BERTScore) to compare a model’s output to a golden reference answer. Though this approach works relatively well, there are some fundamental problems that arise with reference-based metrics:

• We always require a reference answer in order to perform evaluation.

• Our output must be similar to this reference answer to perform well.

As we know, LLMs are capable of solving many different tasks, and most of these tasks are open-ended in nature. For example, we can use the same LLM to do creative writing or to answer medical questions. Although these problems are quite different, they do have a fundamental similarity: there are many ways to answer a question correctly. Traditional reference-based metrics struggle to handle such nuanced scenarios where divergence from a chosen reference answer does not imply that an output is bad. As a result, we have seen from several papers that reference-based metrics tend to correlate poorly with human preferences.

“LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain.” - from [7]

LLM-as-a-Judge is a reference-free metric that prompts a foundation model to perform evaluation based upon specified criteria. Although it has limitations, this technique shows high agreement in many settings with human preferences and is capable of evaluating open-ended tasks in a scalable manner (i.e., minimal implementation changes are required). To evaluate a new task, we simply need to create a new prompt that outlines the evaluation criteria for this task. LLM-as-a-Judge was originally proposed after the release of GPT-4. This metric quickly gained popularity due to its utility and simplicity, culminating in the publication of an in-depth technical report [7]. Today, LLM-as-a-Judge is a widely-used technique in LLM evaluation; e.g., AlpacaEval, Chatbot Arena, Arena-Hard, and more.

Scoring setups. When performing evaluation with an LLM, there are a few different scoring setups that are commonly used (shown above):

1. Pairwise (preference) scoring: the judge is presented with a prompt and two model responses and asked to identify the better response.

2. Direct assessment (pointwise) scoring: the judge is given a single response to a prompt and asked to assign a score; e.g., using a 1-5 Likert scale.

3. Reference-guided scoring: the judge is given a golden reference response in addition to the prompt and candidate response(s) to help with scoring.

This list of scoring setups is not exhaustive, but most scoring setups for LLM-as-a-Judge use some variant or combination of the above techniques. For example, we can derive a pairwise score by scoring two responses independently and comparing their scores. In most cases, we also pair LLM-as-a-Judge with chain-of-thought prompting by asking the model to explain its evaluation process before providing a final score. Not only do such explanations make the evaluation process more interpretable, but they also improve the scoring accuracy of the LLM. Practically, implementing this change can be as simple as adding “Please provide a step-by-step explanation prior to your final score” to your prompt.

“We identify biases and limitations of LLM judges. However, we… show the agreement between LLM judges and humans is high despite these limitations.” - from [7]

Biases of LLM-as-a-Judge. Despite the effectiveness of LLM-as-a-Judge, this technique has several limitations of which we need to be aware. Fundamentally, the LLM judge is an imperfect proxy for human evaluation. By using a model for evaluation, we introduce several sources of bias into the evaluation process:

1. Position bias: the judge may favor outputs based upon their position within the prompt (e.g., the first response in a pairwise prompt).

2. Verbosity bias: the judge may assign better scores to outputs based upon their length (i.e., longer responses receive higher scores).

3. Self-enhancement bias: the judge tends to favor responses that are generated by itself (e.g., GPT-5 can assign higher scores to its own outputs).

4. Capability bias: the judge struggles with evaluating responses to prompts that it cannot itself solve.

5. Distribution bias: the judge may be biased towards certain scores in its scoring range (e.g., on a 1-5 Likert scale the judge may output mostly 3’s).

In addition to these biases, LLM judges are generally sensitive to the details of their prompt. Therefore, we should not simply write a prompt and assume proper evaluation. We must calibrate our evaluation process, collect high-quality human labels, and tune our prompt to align well with human judgment; see here.

There are several techniques we can adopt to combat scoring bias; e.g., in-context learning to better calibrate the judge’s score distribution, randomizing position and sampling multiple scores (i.e., position switching), providing high-quality reference answers, or using a jury of multiple LLM judges. For further details on LLM-as-a-Judge, a full overview of the topic is available at the link below—

Using LLMs for Evaluation

The prompts used for LLM-as-a-Judge in the above section are quite simple. We just describe the evaluation task at a high level and let the LLM judge output a score. However, scoring with a single, general prompt is not always the best approach. Prior work [15] has shown that we can significantly improve the reliability of LLM evaluation by:

• Creating several per-criterion scoring prompts.

• Providing a step-by-step description of the evaluation process.

Put simply, providing a granular scoring prompt is beneficial, and we need not stop here. We can create judge prompts targeted to each domain, task, or instance. Increasing the granularity of LLM-as-a-Judge in this way is where the idea of a rubric arises. A rubric is just a scoring prompt that provides a detailed set of criteria by which a response is evaluated; see below. In many cases, rubrics are prompt (or instance)-specific, meaning that a tailored rubric is created for each prompt-response pair being evaluated. These prompt-specific rubrics are often synthetically generated with an LLM—potentially with human intervention.

As we can see above, rubrics are usually checklist-style and separated into a list of distinct criteria. Each of these criteria captures a single quality dimension that can be evaluated with an LLM judge. Additionally, in many setups, weights are defined for each criterion to simplify the aggregation of criteria-level scores. Given the similarity of rubrics and vanilla LLM-as-a-Judge, the emergence of rubrics is hard to attribute to a single paper. Rather, the use of rubrics was a slow transition that occurred over time as LLM-as-a-Judge prompts became more granular.

“HealthBench is a rubric evaluation. To grade open-ended model responses, we score them against a conversation-specific physician-written rubric composed of self-contained, objective criteria. Criteria capture attributes that a response should be rewarded or penalized for in the context of that conversation and their relative importance.” - from [16]

In recent work, prompt-specific rubrics have become heavily used for evaluation in expert domains. For example, HealthBench [16] evaluates the quality of medical conversations according to physician-written rubrics that are specific to each conversation; see below. These rubrics focus on detailed and objective criteria—each associated with a weight—that can be verified with an LLM to yield a binary (pass or fail) score. MultiChallenge [17]—a multi-turn chat benchmark focused on tough edge cases like iterative editing, self-coherence, and instruction retention—develops prompt-specific rubrics to improve benchmark reliability, finding that rubrics improve agreement between expert humans and LLM judges.

In this overview, we will go beyond the use of rubrics for evaluation and instead focus on the application of rubrics for deriving a reward signal in RL training. One of the biggest risks when using LLM-as-a-Judge-derived rewards for RL training is reward hacking—LLM judges have known biases that can be exploited. However, we see above that detailed rubrics help to make the evaluation process more reliable, thus reducing risks associated with reward hacking.

RL with Verifiable (and Non-Verifiable) Rewards

Though RL training has long been used for LLMs, the role of RL in LLM training pipelines has become more central with the recent advent of reasoning models. In general, there are two common RL paradigms used for LLMs:

• Reinforcement Learning from Human Feedback (RLHF) trains the LLM using RL with rewards derived from a reward model trained on human preferences.

• Reinforcement Learning with Verifiable Rewards (RLVR) trains the LLM using RL with rewards derived from rule-based or deterministic verifiers.

The main difference between RLHF and RLVR is how we assign rewards—RLHF uses a reward model, while RLVR uses verifiable rewards. Aside from this difference, both are online RL algorithms with a similar structure; see below. For details on the inner workings of RL optimizers, please see prior posts on PPO and GRPO.

Impact of RLVR. Recent progress in reasoning models has been driven largely by reinforcement learning with verifiable rewards (RLVR), which derives a reward signal during RL training from deterministic (or programmatic) rules that can be reliably checked (e.g., passing unit tests for code or matching a known numerical answer in math). Using rules-based rewards lowers our risk of reward hacking because we are using a hard rule to derive our reward rather than an LLM-based reward model. As a result, we can run larger-scale RL runs (i.e., over more data and for a larger number of iterations) with less risk of training instability.

On the other hand, the same property that makes RLVR so powerful—the dependence on reliable, rules-based rewards—limits its applicability. Practically, we can only use RLVR on tasks with clean ground-truth labels¹ that can be checked automatically. Luckily, several important tasks fall into this category (e.g., math and coding). However, there are many other tasks we would like to solve but are subjective and difficult to verify. Due to this need for verification, we see that LLMs have advanced quickly in certain verifiable capabilities, while gains on non-verifiable tasks have been less uniform. To solve this issue, we need to develop an approach for extending recent advances in RL training to non-verifiable tasks.

“In RLVR, rewards are derived from deterministic, programmatically verifiable signals—such as passing unit tests in code generation or matching the correct numerical answer in mathematical reasoning. While effective, this requirement for unambiguous correctness largely confines RLVR to domains with clear, automatically checkable outcomes.” - from [2]

Open-ended domains. We typically turn to RLHF for training LLMs in open-ended settings. RLHF replaces deterministic verifiers with a learned reward model trained on preference data; see below. Preference data can be collected for any domain by simply sampling multiple completions for each prompt and having a human (or model) select the better of the two. We can drastically increase domain coverage by using RLHF. However, relying upon preference data and reward models introduces notable difficulties and failure modes:

• A large volume of preference data must be collected.

• We lose granular control over the alignment criteria—preferences are expressed in aggregate over a large volume of data rather than via explicit criteria.

• The reward model can overfit to artifacts (e.g., response length, formatting, etc.) and generally introduces more risk of reward hacking.

RLHF is a general technique, but it is usually used in practice for improving broad, subjective properties; e.g., helpfulness, harmlessness, or style. For complex, open-ended tasks, the reward signal tends to be multi-dimensional. Traditional reward modeling captures these quality dimensions via a single preference label, which eliminates our ability to specify quality dimensions at a more granular level. One could collect criterion-level preferences to solve this issue, but doing so requires training (and maintaining) separate reward models per criterion and increases the volume of data that must be collected. A natural alternative is to make evaluation dimensions explicit by using a rubric to ground the reward in structured, interpretable criteria rather than a single judgment.

Rubrics-as-Rewards. The idea of deriving a reward from a rubric-based LLM judge is one of the current frontiers of RL research—it presents an opportunity to extend RLVR to arbitrary open-ended tasks. Although this area of research is still nascent and evolving quickly, the idea of using rubrics for RL is not new! Similar ideas have already been proposed for better handling the safety alignment of LLMs. During LLM alignment, we have a detailed list of safety specifications that describe the desired behavior of the model. These specifications are changed frequently as new needs or failure cases arise in practice. The dynamic nature of safety criteria makes applying a standard RLHF approach difficult—the preference data must be adjusted or re-collected each time that our criteria change.

To avoid the need for constant data collection, methods like Constitutional AI [13] and Deliberative Alignment [14] show that a reliable reward signal can be derived directly from the safety specifications themselves. More specifically, we can provide safety criteria as input to a strong reasoning model that is used to generate data or evaluate model outputs according to these criteria. Due to the strong instruction following capabilities of frontier-level reasoning models, this approach is capable of providing a reliable reward signal for safety training.

This approach avoids the need to re-collect data as criteria change. Rather, we just maintain a clear, itemized list of safety criteria—basically a safety rubric—that can be provided as input to the alignment system. Instead of collecting data, we focus on creating a “constitution” that dictates the behavior of our model. Once this constitution is available, we rely upon an LLM judge to apply the necessary supervision for achieving this desired behavior. This approach is both dynamic and interpretable, but it can only be applied in domains where the LLM judge is known to perform well. Extending similar techniques to arbitrary domains, which we will explore for the remainder of this post, is a non-trivial research problem.

Using Rubrics for RL

We now have a detailed understanding of LLM-as-a-Judge, rubrics, and their application to RL training. Next, we will extend these ideas by overviewing a broad collection of recent papers that study the application of rubrics to RL training. Many papers have been written on this topic in quick succession. As we will see, however, much of this work shares a similar flavor. Slowly, rubric-based RL has become more effective across a wider variety of tasks, enabling powerful reasoning models to achieve impressive gains even in non-verifiable domains.

Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains [1]

“Rather than using rubrics only for evaluation, we treat them as checklist-style supervision that produces reward signals for on-policy RL. Each rubric is composed of modular, interpretable subgoals that provide automated feedback aligned with expert intent. By decomposing what makes a good response into tangible, human-interpretable criteria, rubrics offer a middle ground between binary correctness signals and coarse preference rankings.” - from [1]

RLVR is effective in verifiable domains with a clear correctness signal like math or coding, but there are many domains in the real world that are not strictly verifiable (e.g., science or health). For these domains, we need a more versatile reward mechanism—such as an LLM judge or reward model—that can handle open-ended problems that lack a clear or verifiable answer. Going beyond a vanilla LLM-as-a-Judge setup, we see in [1] that prompting the LLM judge with a rubric composed of structured, instance-specific—meaning unique to each prompt—criteria benefits the model’s performance in on-policy RL training.

Creating rubrics. Rubrics in [1] are checklist-style and cover multiple criteria that are specific to each prompt being scored. The checklist for a rubric contains K total criteria c_i, each with a corresponding weight w_i. A criterion is defined as a binary correctness check that can be validated using an LLM judge. We can also recover an RLVR setup by assuming K = 1 and letting c_1 be a deterministically verifiable reward signal with weight w_1 = 1.0.

We refer to this approach of using rubrics to generate a reward signal for RL as Rubrics-as-Rewards (RaR). There are two approaches we can use to evaluate a rubric and derive a reward for RL training (shown above):

• Explicit aggregation: each criterion is independently evaluated using an LLM judge, and the final reward is derived by summing and normalizing the weighted score of each criterion.

• Implicit aggregation: all criteria along with their weights are passed to an LLM judge, which is asked to derive a final reward that considers all information.

Explicit aggregation provides more granular control over the weight of each criterion, which can aid in interpretability but requires tuning and can be fragile. In contrast, the implicit aggregation approach delegates the reward aggregation process—including handling the weights of each criterion—to the LLM judge.

Generating rubrics. All instance-specific rubrics used in [1] are generated by an LLM; see above. When generating rubrics, guiding principles are provided to the model with respect to how rubrics should be created. Namely, rubrics must i) be grounded in guidance from human experts, ii) be comprehensive (i.e., span many dimensions of quality), iii) specify per-criterion importance (e.g., factuality is more important than style), and iv) use self-contained criteria (i.e., criteria should not depend on one another). Given these desiderata and a golden (expert-curated) reference answer for a prompt, the LLM then generates a rubric that includes:

• 7-20 self-contained criteria.

• A numeric or categorical (i.e., essential, pitfall, important, or optional²) weight for each of these criteria.

Numeric weights provide fine-grained control over criterion importance, but categorical weights, each of which are mapped to a numerical score, are more interpretable—both for humans and the LLM—which leads them to be used for experiments in [1]. Once generated, a rubric can be used as a reward function by passing it to an LLM judge and performing explicit or implicit aggregation.

“We generate rubrics using OpenAI’s o3-mini and GPT-4o, conditioning generation on reference answers from the underlying datasets to approximate expert grounding. The resulting collections—RaR-Medicine and RaR-Science—are released for public use.” - from [1]

Experimental settings. In [1], authors see rubrics as an opportunity to provide flexible, scalable, and interpretable reward signals for RL in real-world domains that go beyond verifiable problems like code and math. Moving in this direction, two non-verifiable domains are considered in [1]: medicine and science. Prompts and rubrics used for RL in [1] are sampled from a mixture of public datasets, such as NaturalReasoning, SCP-116K, and GeneralThought-430K. This data is further curated to create two datasets for RaR training in [1]:

• RaR-Medicine: ~20K prompts focused on medical reasoning with instance-specific rubrics generated with GPT-4o.

• RaR-Science: ~20K prompts curated to align with the problem categories from GPQA-Diamond with instance-specific rubrics generated by o3-mini.

All experiments use Qwen-2.5-7B as a base model and train with GRPO. Rewards are assigned using GPT-4o-mini with the instance-level rubrics described above. The proposed technique in [1], referred to as RaR-Implicit, uses LLM-generated, instance-specific rubrics with implicit aggregation as a reward signal. Several rubric-free and fixed-rubric baselines are also considered:

• Base models: Qwen-2.5-7B and Qwen-2.5-7B-Instruct models are evaluated with no additional training.

• Direct Assessment Judge: an LLM judge provides a direct assessment score for each response on a 10-point Likert scale—this is a standard LLM-as-a-Judge setup that does not use a granular, instance-specific rubric.

• Reference-Based Judge: same as above, but the LLM judge is given a golden reference answer as context when generating a score.

• RaR-Predefined: a fixed set of generic rubrics are used for all prompts with explicit aggregation and uniform criteria weights.

• RaR-Explicit: instance-specific rubrics are used, but all criteria receive fixed weights based on their categorical importance label.

All models are evaluated on the GPQA-Diamond (Science) and HealthBench (medicine) benchmarks. For some smaller ablation experiments, RL training is performed on the training set of HealthBench rather than RaR-Medicine.

Do rubrics provide useful rewards? Across all experiments in [1], we see that using structured, rubric-based rewards during RL training is beneficial. Rubric-based rewards are especially impactful when using smaller LLM judges for RL training and are found to reduce variance in reward signals across different sizes of LLM judges. As shown above, rubric-based approaches outperform all rubric-free methods aside from the reference-based LLM judge, relative to which we only see marginal gains from rubrics. However, rubrics are found to yield a more notable gain over reference-based LLM judge rewards in later experiments that train on HealthBench; see below. We also see that implicit aggregation tends to outperform explicit aggregation by a small (but consistent) margin.

“Rubric-guided training achieves strong performance across domains, significantly outperforming Likert-based baselines and matching or exceeding the performance of reference-based reward generation.” - from [1]

These experiments also highlight the necessity of expert-curated references for generating rubrics—performance noticeably deteriorates without references, indicating purely synthetic rubrics are suboptimal. Predefined or generic rubrics are also found to perform quite poorly, indicating that prompt-specific criteria are useful for deriving high-quality rubrics. These best practices for creating better rubrics are also evaluated beyond their impact on RL training. In [1], authors show that rubrics created via their proposed approach have noticeably higher levels of agreement with preference annotations from human experts; see below.

Reinforcement Learning with Rubric Anchors [2]

“The success / failure hinges tightly on the diversity, granularity, and quantity of the rubrics themselves, as well as on a proper training routine and meticulous data curation.” - from [1]

Authors in [2] continue studying the application of RL to open-ended tasks using rubric-based rewards. They scale the rubric creation process to produce a dataset of ~10K rubrics curated by humans, LLMs, or a combination of both. Building on this dataset, a practical exposition of rubric-based RL is provided, ultimately arriving at a functional RaR training framework called Rubicon. Interestingly, simply increasing the number of rubrics—whether generated synthetically or with human assistance—yields only marginal gains. Instead, we must carefully curate high-quality rubrics, suggesting that the success of RaR heavily depends upon both rubric quality and the quality of the underlying training dataset.

Rubric system. Instead of using strictly instance-level rubrics, multiple scopes are considered in [2], including instance, task, and dataset-level rubrics. When generating data, the system in [2] (shown below) starts by constructing the rubric first. Data is synthesized only after the rubric is created so that it explicitly matches the rubric. Then, the combination of rubric and data is used for both RL training and evaluation. Tasks in [2] are selected according to the asymmetry of verification—verifying a candidate output should be much easier than generating it.

To ensure rubric quality, authors run dedicated ablation experiments for every set of rubrics that is generated to measure their impact on the training process. Each rubric is comprised of K criteria C = {c_1, c_2, …, c_K}. An example of a rubric created for evaluating open-ended or creative tasks is provided below. After evaluating each of these criteria, we are left with a multi-dimensional reward vector that can be aggregated to yield a final reward.

As a baseline, criteria-level rewards can be aggregated via a weighted average, but non-linear dependencies may exist between criteria that make a weighted average suboptimal. For this reason, authors in [2] consider the following advanced strategies for criteria aggregation:

• Veto Mechanisms: failing on a critical dimension overrides any reward from other dimensions.

• Saturation-Aware Aggregation: over-performing on a single dimension yields diminishing returns relative to a balanced reward across dimensions.

• Pairwise Interaction Modeling: criteria are modeled together to capture inter-criteria relationships (i.e., synergistic or antagonistic effects).

• Targeted Reward Shaping: rewards in high-performance regions are amplified to better capture differentials and avoid scores becoming compressed.

Training strategy. The data used in [2] is derived from a proprietary post-training corpus with ~900K examples. Prior to any training, offline difficulty filtering is performed to remove any examples on which the base model performs too poorly or already performs well³. From here, RL training progresses in two phases, each with a different curriculum:

• The first phase focuses on instruction-following and programmatically-verifiable tasks to teach the LLM how to properly handle constraints.

• The second phase extends the training process to more open-ended and creative tasks with a higher level of subjectivity.

While the first phase primarily relies upon static rubrics and verifiers, we must use reference-based rubrics—often with instance-specific criteria—for the second phase. Granular rubrics help to provide a more reliable reward signal on tasks that are highly subjective. This multi-stage training framework aims to progressively cultivate the capabilities of the model. When training jointly on all tasks, authors observe a “seesaw effect”—joint training actually reduces model performance relative to forming a multi-stage curriculum; see below.

Reward hacking is one of the biggest risks in a RaR setup. Whereas verifiable rewards are deterministic, neural reward models can be exploited, and the likelihood of our policy finding such an exploit increases in large-scale RL runs⁴. The Rubicon approach proposed in [2] combats reward hacking by performing an offline analysis of rollout data. After the first phase of RL training, authors examine rollouts that yield abnormally high rewards and create a basic taxonomy of recurring reward hacking patterns that are discovered. From this taxonomy, a specific rubric is created for preventing reward hacking—this rubric can also be iteratively refined over time. The addition of a reward hacking rubric improves training stability (i.e., avoids collapse into a reward-hacked state) and allows RL training to be conducted for a much larger number of training steps.

Rubicon-preview is a Qwen-3-30B-A3B base model that is finetuned in [2] using the Rubicon framework. This model excels in open-ended and humanities-related benchmarks. For example, we see below that Rubicon-preview achieves an absolute improvement of 5.2% compared to the base model on various instruction following, emotional intelligence, and writing benchmarks. Notably, Rubicon-preview also outperforms DeepSeek-V3-671B on most of these tasks, where an especially significant performance boost is observed on writing tasks.

The performance benefits of Rubicon-preview are also achieved with shocking sample efficiency—the model is only trained on ~5K data samples. By using an RaR approach, authors are also able to granularly control the style or voice of the resulting model. More specifically, a few case studies are presented in [2] that demonstrate the use of rubrics to guide the LLM away from the didactic tone that is common of chatbots and towards a human-like tone with more emotion.

Going further, creatively-oriented RaR training does not seem to damage the LLM’s general capabilities. As shown above, Rubicon-preview performs on par with or better than the original base model across a wide scope of benchmarks. Such a result should not come as a surprise given the natural ability of RL to avoid forgetting and retain the prior knowledge or skills of an LLM; see here.

OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment [3]

We’ve seen several papers that study the use of rubrics for RL training, where rubrics are generated—possibly with human intervention—and evaluated by an off-the-shelf LLM. Instead of focusing upon the downstream application of rubrics in RL, authors in [3] specifically analyze the rubric generation and evaluation process. To facilitate this study, an open dataset of prompt-rubric pairs, called OpenRubrics, is created for training both rubric generation models and rubric-based reward models. As we learned in [2], RaR training is highly dependent upon rubric quality. Creating better rubrics—and reducing the amount of human supervision in this process—makes RaR training more scalable and effective.

The rubric structure used in [3] is consistent with prior work. Namely, each rubric is comprised of K criteria, where each criterion is a rubric description that specifies one aspect of response quality. Two types of criteria are considered:

1. Hard rules: explicit or objective constraints (e.g., length or correctness).

2. Principles: higher-level qualitative aspects (e.g., reasoning soundness, factuality, or stylistic coherence).

Unlike prior work, rubrics in [3] do not use per-criterion weights and are used for pairwise comparison of two completions—as opposed to direct assessment. For a rubric R = {c_1, …, c_K} and two responses y_1 and y_2 to the same prompt x, we want our rubric-based reward model to provide a binary preference label (i.e., y_1 > y_2 or y_1 < y_2) by reasoning over the rubric criteria.

Building OpenRubrics. The prompts and preference labels used for creating OpenRubrics are sourced from several public datasets (e.g., UltraFeedback, MegaScience, Medical-o1, instruction following data from Tulu-3, and more). For each of these datasets, preference data is obtained via domain-specific post-processing of the existing data. For example, the highest and lowest scoring responses form a preference pair for UltraFeedback, while for MegaScience and Medical-o1 completions are generated with a pool of LLMs and scored via a jury of different reward models to obtain preference pairs; see below.

Once this preference data is available, rubrics are generated using two key strategies proposed in [3] (shown above):

1. Contrastive Rubric Generation (CRG): an instruction-tuned LLM is provided both a prompt and a preference pair and asked to produce discriminative evaluation criteria by contrasting the chosen and rejected responses.

2. Rubric Filtering: rubrics are filtered by prompting an LLM to choose the preferred response given a preference pair and rubric as input and only retaining rubrics that yield agreement with human-provided preference labels (i.e., preference label consistency)⁵.

CRG and rubric filtering aim to create rubrics that are both prompt-specific and aligned with human preference examples, allowing them to serve as useful anchors for reward modeling. The result of this rubric generation and filtering approach is OpenRubrics, the key statistics of which are summarized in the plots below.

“After collecting the rubrics-based dataset, we proceed to develop a rubric generation model that outputs evaluation rubrics and a reward model Rubric-RM that generates final preference labels.” - from [3]

Rubric-RM. OpenRubrics provides a high-quality dataset of preference pairs and rubrics. In [3], this data is used to train two kinds of models (both of which are based upon Qwen-3-4B or Qwen-3-8B):

1. A rubric generation model—trained via SFT—that, given a prompt, can produce a discriminative rubric for predicting preference labels.

2. A reward model—also trained via SFT⁶—called Rubric-RM that can predict rubric-guided, pairwise preferences.

At inference time, these two models are used in tandem. Given a prompt, we first use the rubric generation model to produce our rubric. Then, Rubric-RM ingests this rubric, the prompt, and a pair of completions to generate a final preference prediction. We can also use majority voting (i.e., running this pipeline several times and taking the most frequently outputted score) to improve accuracy. Although using a two-stage pipeline increases inference costs, authors mention that costs can be decreased significantly by caching generated rubrics.

Comparison to other reward models. Rubric-RM is compared to a wide variety of other reward models and LLM-as-a-Judge approaches on several key evaluation benchmarks; see above. Rubric-RM tends to outperform similarly-sized baselines; e.g., the 8B variant gets 70.1% average accuracy, whereas the strongest 7B-scale reward model (RM-R1-7B) has an average accuracy of only 61.7%. These results are made even stronger with the use of majority voting. When comparing to the Qwen-3 base models, we see a noticeable uplift in preference scoring accuracy for Rubric-RM, highlighting the effectiveness of the finetuning strategy in [3].

“Rubric-RM excels on benchmarks requiring fine-grained instruction adherence… This demonstrates that rubrics capture nuanced constraints better than scalar reward models.” - from [3]

The gains from Rubric-RM are most pronounced on instruction-following tasks, which means that the rubrics in [3] work well for explicit evaluation criteria. On the other hand, this finding indicates less impact for subjective criteria, revealing that improving rubric supervision for open-ended tasks is still an open problem.

Application to post-training. Beyond evaluating Rubric-RM on reward modeling benchmarks, we can also measure the model’s downstream impact by using it as a reward signal in LLM post-training. Downstream evaluations in [3] only consider instruction following tasks (i.e., IFEval, InfoBench, and IFBench)—likely because this is the domain on which Rubric-RM excels—and use DPO for preference tuning. Rubric-RM is found to yield a boost over other reward models; see below.

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [4]

“Deep research (DR) models aim to produce in-depth, well-attributed answers to complex research tasks by planning, searching, and synthesizing information from diverse sources” - from [4]

Rubrics are studied in the context of deep research (DR) agents in [4]. A DR agent is an LLM that is taught to perform multi-step research and produce long-form answers—or surveys—that answer a query with detailed information and citations. This idea was popularized by Gemini DR and followed shortly after by DR agents from OpenAI, Anthropic, and more. Though many closed models support DR mode, open models are behind in this area: most open DR models are either prompt-based or trained on short-form, search-intensive QA tasks (i.e., not reflective of frontier DR agents) with RLVR. To solve this, authors in [4] train Dr. Tulu-8B—a fully-open⁷ LLM agent for long-form, open-ended DR tasks—using a novel online RL technique that evolves instance-level rubrics alongside the policy throughout training.

Definition of DR. Before describing Dr. Tulu, we need to understand the basic mechanics of DR agents. Details of closed DR agents are not publicly disclosed, but we can discern from using these agents that they:

1. Heavily rely on search tools to ground their answers in external knowledge.

2. Output long answers (i.e., basically survey papers) with many citations.

Authors in [4] use these observations to formalize an action space for DR agents; see below. In this formulation, a DR agent has the ability to i) think, ii) call a set of search tools, iii) provide a final answer, and iv) insert citations into the final answer. For all actions, any context that is output (e.g., thinking traces or tool outputs) is just concatenated to the sequence being processed by the DR agent. The DR agent itself is just an LLM that performs tool use in this action space.

Rubrics for DR. Evaluating a DR agent is a tough task. These agents generate lengthy outputs with detailed information, so there are many ways that an output could be good or bad—a static or predefined set of rubrics will not capture the detailed quality dimensions required for this task. Additionally, evaluation varies depending on the query (e.g., asking for a vacation plan versus an AI research survey).

Given that most DR queries are knowledge-intensive, we must also verify key information against known world knowledge. For this reason, synthetically generating instance-specific rubrics with an LLM—as in [1, 3]—is insufficient. This approach relies upon the parametric knowledge of the LLM rather than grounding on external knowledge that can be used to verify correctness. Ideally, we should ground the evaluation process in knowledge retrieved via search tools rather than relying on the (incomplete) parametric knowledge of an LLM.

Evolving rubrics. To address the unique considerations of DR tasks, Dr. Tulu is trained using a modified rubric-based RL technique, called Reinforcement Learning with Evolving Rubrics (RLER), that derives a reward from instance-specific rubrics that i) evolve alongside the policy during training and ii) are grounded in knowledge from the internet; see above. Similarly to prior work, rubrics are defined as a set of weighted criteria. Each of these criteria can be scored with a separate LLM judge to derive a final score as shown below. This formulation matches the explicit aggregation strategy proposed in [1].

During training, we have a buffer of rubrics for each prompt that stores a set of evolving rubrics specific to that prompt. Within this buffer, we designate certain rubrics as active, and these active rubrics are used for deriving the reward in the current training iterations. To initialize the buffer, we first create a set of search-based rubrics using an LLM with access to search tools. These initial rubrics are used persistently—meaning they are always included in the active set of rubrics—throughout training. At each training step, we prompt an LLM to generate a set of new (or evolving) rubrics given a prompt, a group of corresponding rollouts, and the set of active rubrics for that prompt as context; see below. Specifically, there are two types of rubrics that can be created by the LLM:

1. Positive Rubrics: capture strengths of new relevant knowledge explored by the current policy but not yet present in any rubric.

2. Negative Rubrics: address common undesirable behaviors of the current policy (e.g., reward hacking).

During RLER, the number of evolving rubrics can become large. To avoid this, we maintain a subset of active rubrics—always containing the initial persistent rubrics—via an explicit management strategy that filters and ranks rubrics based on their discriminative power. To measure a rubric’s discriminative power, we rely upon the group of completions created for advantage computation in GRPO. During each policy update, the group of completions for a given prompt is scored using all active rubrics for that prompt, and rubrics with zero reward variance (i.e., no discriminative value) are removed. Remaining rubrics are ranked in descending order based on the standard deviation of rewards across the group. Only rubrics with the top K standard deviations—and persistent rubrics—remain active.

The evolving rubrics in [4] are grounded in external knowledge and allow the reward for RL to adapt to the current state of our policy. As the model discovers new behaviors (e.g., a reward hack), these changes can be identified and captured in a new or modified rubric to maintain training fidelity. For this reason, we do not need to create a rubric a priori that exhaustively captures all desiderata for evaluation, which is difficult for DR tasks. Rather, this system can observe policy behavior and automatically incorporate key trends into new rubrics.

The rubric evolution process is found in [4] to have interesting characteristics, such as producing rubrics with measurably higher levels of specificity or even negative rubrics that penalize specific behaviors within the LLM; see above.

Dr. Tulu-8B is trained using a two-stage approach that includes a cold start SFT phase and online RL with GRPO. The Qwen-3-8B base model used in [4] does not yet possess the necessary atomic skillset (e.g., proper planning or citations) for solving DR tasks. If we were to begin RL training directly from this model, most rollouts would be of low quality, and the training process would likely struggle to efficiently discover high-reward solutions via exploration. To solve this issue, a cold start SFT phase is performed in [4] prior to RL training by sampling DR trajectories from a strong teacher model—in this case GPT-5 with a detailed system prompt describing the DR task—for supervised training. By finetuning the Qwen-3 base model on these trajectories, we allow the model to quickly learn a better initial policy for searching, planning, and citing sources prior to online RL. Given that most open DR agents are trained on short-form QA tasks, these supervised trajectories, which are openly available, are by themselves a useful artifact.

After cold start SFT, we perform online RLER using GRPO (with token-level loss aggregation) as the RL optimizer. Efficiently generating rollouts for online RL with a DR agent is a non-trivial systems problem due to output length and the frequency of tool calls. Rollouts are already the largest bottleneck in RL. Adding tool calls into the mix (i.e., “agentic” rollouts) makes this problem even worse. To improve efficiency, authors in [4] use one-step asynchronous RL training. Rollout generation and policy updates are performed at the same time, but policy updates are performed on rollouts from the prior training step. Additionally, tool calls are executed immediately to overlap generation and tool calling as much as possible.

“Tool requests are sent the second a given rollout triggers them, as opposed to waiting for the full batch to finish… Once a tool call is sent, we place that given generation request to sleep, allowing the inference engine to potentially continue to work on generating other responses while waiting for the tool response. This results in the generation and tool calling being overlapped wherever possible.” - from [4]

One other difficult aspect of RL training with a DR agent is the output lengths—generating long outputs (obviously) increases the time taken to produce a rollout. Plus, there can be high variance in output lengths. To mitigate this issue, sample packing is adopted during RL training, which improves efficiency by combining multiple outputs into a single, fixed length sequence. Finally, a few additional sources of heuristic rewards are used on top of RLER to encourage correct formatting and sufficient usage of search and citation tools by the agent.

Performance and efficiency. Dr. Tulu-8B is evaluated on several DR benchmarks (ScholarQA, HealthBench, ResearchQA, and DeepResearchBench), where we see that it substantially outperforms other open DR agents—even those that are larger (e.g., Tongyi-DR-30B-A3B)—and frequently matches the performance of the top proprietary systems. Additionally, Dr. Tulu-8B is smaller and cheaper compared to other systems. Notably, Dr. Tulu-8B is up to three orders of magnitude cheaper than OpenAI DR in some cases; e.g., costs are reduced from $1.80 per query to $0.0019 per query on ScholarQA⁸. Much of these savings come from the ability to call the correct tools and avoid excessive tool usage that drastically increases API costs. Not only does Dr. Tulu-8B generally make fewer tool calls, but authors observe in [4] that the model heavily calls free paper search tools for academic benchmarks while only using paid web search tools for more general queries.

Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training [5]

Rubrics are helpful for performing granular evaluation, assuming that the rubric we are using is of high quality. To curate a high-quality rubric, we rely upon human annotators or synthetic generation. Relying on human oversight would make it difficult to scale rubric curation. On the other hand, synthetic rubrics are scalable, but static models are often used to generate and evaluate these rubrics, which limits adaptation to new domains. To make this process more dynamic, a joint training procedure for rubric generation and evaluation is proposed in [5].

“Rubric-ARM [is] a framework that jointly optimizes a rubric generator and a judge using RL from preference feedback. Unlike existing methods that rely on static rubrics or disjoint training pipelines, our approach treats rubric generation as a latent action learned to maximize judgment accuracy. We introduce an alternating optimization strategy to mitigate the non-stationarity of simultaneous updates.” - from [5]

Rubric-ARM. There are two models being trained in this framework: a rubric generator and an LLM judge. These models are trained with an alternating RL framework that switches between training each model. This approach, called Rubric-ARM, jointly optimizes the generator’s ability to create a rubric and the judge’s ability to predict human-aligned preference scores given a rubric as input. By learning these components together (i.e., instead of using separate training pipelines), we allow them to co-evolve and reinforce each other throughout training.

A rubric is defined in [5] as a set of evaluation criteria that are conditionally generated given a prompt as input—no explicit per-criterion weights are defined. Given a rubric sampled from the rubric generator, the objective—for both the rubric generator and the judge—is to maximize the preference accuracy of scores output by the judge. Notably, Rubric-ARM only considers preference data. The LLM judge is trained to predict a preference label (i.e., instead of performing direct assessment) given a prompt and two possible completions as input.

Training pipeline. Prior to RL, Rubric-ARM performs a cold-start SFT phase that trains both the rubric generator and the judge over a synthetic dataset curated from a variety of open data sources (e.g., UltraFeedback, Magpie, and more). From here, we begin the alternating RL procedure that switches between training the rubric generator or judge while keeping the other fixed. Alternating the learning process gives each component a clear training signal by keeping the other fixed.

At each training iteration t, we sample a batch of preference data. A rubric is then sampled—and cached for future use—with the rubric generator for each prompt in the batch. First, the rubric generator is kept fixed, and we perform RL training (with GRPO) to update the judge. The reward is defined as a sum of:

• Preference accuracy: a binary score indicating whether the predicted label matches the ground-truth label.

• Correct formatting: a heuristic that checks the judge’s trajectory for expected components (i.e., addressing each rubric criterion, providing per-criterion explanations, and finishing with an overall justification and decision).

Rubrics are generally sampled once and used for multiple judge optimization steps. After training the judge, we then freeze the judge’s weights and update the rubric generator. The rubrics used during this phase are cached, as the rubric generator was not trained during the prior phase. To train the rubric generator, we only use a preference accuracy reward based on whether the fixed judge is able to predict a correct preference label given the generated rubric. We learn from experiments in [5] that the optimization order is important. Training the rubric generator before the judge leads to noticeably degraded performance.

“Early-stage exploration by the rubric generator can dominate the learning dynamics. To mitigate this, we first stabilize the reward model under fixed rubrics before optimizing the rubric generator. This alternating schedule reduces variance and ensures robust optimization.” - from [5]

Application to post-training. The rubric generator and judge obtained from Rubric-ARM can also be applied to LLM post-training. Beginning with a set of prompts, we do the following:

1. Sample a rubric for each prompt with the rubric generator.

2. Sample two completions for each prompt using our current policy.

3. Score the completions using the judge with the above rubric⁹.

4. Perform DPO using preference data created with the above steps.

We are not restricted to offline training either! The above steps can easily be generalized to a semi-online DPO setup by regularly sampling new, on-policy completions and performing DPO training in phases to increase the freshness of preference data. We can even perform fully-online RL by modifying the above steps with a pairwise RL approach [6]. More specifically, we do the following for each prompt:

1. Sample a deterministic (baseline) completion with greedy decoding.

2. Sample a group of rollouts using a normal sampling procedure.

Once we have these completions, we use them to derive a direct assessment reward from the pairwise comparisons predicted by the LLM judge. To do this, Rubric-ARM creates preference pairs between each rollout in the group and the baseline completion. Then, our reward is defined as whether Rubric-ARM correctly predicts the greedy baseline as the rejected completion; see below.

“Rubric-ARM outperforms strong reasoning-based judges and prior rubric-based reward models, achieving a +4.7% average gain on reward-modeling benchmarks, and consistently improves downstream policy post-training when used as the reward signal.” - from [5]

How does this perform? Rubric-ARM is trained on the general-domain portion of OpenRubrics [3]. Both the rubric generator and LLM judge use Qwen-3-8B as a base model, and a two-stage rubric judging process—including generating and evaluating the rubric—is used at inference time. Rubric-ARM is compared to several open and closed LLM judges, as well as an SFT baseline trained on the same data (i.e., the Rubric-RM model [3]). Metrics on a wide variety of alignment-related reward modeling benchmarks are provided below. As we can see, Rubric-ARM outperforms all other open models and matches or exceeds the performance of most closed judges. Additionally, Rubric-ARM improves the performance of the SFT baseline by 4.8% absolute, indicating that alternating RL is helpful for discovering more discriminative rubrics and improving judge performance.

Rubric-ARM is also tested on WritingPreferenceBench, an out-of-distribution benchmark, where we see that the system generalizes well to other domains and continues to outperform baselines even on a very open-ended task (i.e., creative writing). Authors also run several ablation experiments, where we learn that:

• The optimization order for alternating RL is important; i.e., training the rubric generator first (instead of the judge) degrades preference accuracy by 2.4% with the largest regressions seen on instruction-following tasks.

• Removing the format reward used for the judge is harmful; i.e., LLM judges trained with only correctness rewards perform 2.2% worse than those trained on a combination of correctness and format rewards.

Similar results hold true when Rubric-ARM is used for LLM post-training. Rubric-ARM yields a boost in policy performance in both online and offline alignment scenarios, and policies trained with Rubric-ARM outperform those trained with other open models. Of the methods that are considered, iterative DPO with Rubric-ARM yields the best results, indicating that Rubric-ARM excels in creating high-quality preference data for LLM post-training; see below.

Further Reading

Although we have already covered a variety of papers, RaR is a particularly active and popular topic. To give a more comprehensive picture of the current research landscape, we close with high-level summaries of several more related works.

RL from Checklist Feedback (RLCF) [8] proposes a rubric-based approach for aligning language models to follow complex instructions. Instead of deriving rewards from a reward model trained on a static preference dataset, RLCF uses an LLM to generate instruction-specific checklists that outline the requirements of the instruction as a series of itemized steps. Each component of the checklist is an objective yes or no question that can be evaluated to derive a reward signal.

Rule-based rewards [9] propose an approach to LLM safety alignment that derives a reward signal from an explicit set of rules. Safety alignment is usually handled via RLHF-style preference tuning. However, this process requires collecting preference data, which is expensive, scales poorly as requirements evolve, and offers limited fine-grained control. As an alternative, the authors in [9] explore a hybrid setup in which an LLM evaluates responses against a specified set of safety rules, enabling fine-grained control over refusals and other safety-related behavior. This rule-based reward model is combined with a standard reward model for general helpfulness, allowing the model to undergo a standard alignment procedure with rule-based rewards guiding safety behavior.

Context-Aware Reward Modeling (CARMO) [10] attempts to mitigate problems with reward hacking in human preference alignment with RLHF. Going beyond static evaluation rubrics, an LLM first dynamically generates evaluation criteria for each prompt. Then, these criteria are used by the LLM to score the response, and the score can be directly used as a reward signal for preference alignment.

Reinforcement Learning with Adversarial Critic (RLAC) [11] proposes an adversarial approach for training LLMs on open-ended generation tasks. This framework has three components:

• Generator: the LLM being trained.

• Critic: another LLM that identifies potential failure modes.

• Validator: a domain-specific verification tool.

For each prompt, the generator produces multiple outputs, the critic proposes validation criteria—or a rubric—for each output, and the validator provides binary feedback based on correctness. Preference pairs can be formed between outputs that are validated and those that fail, naturally providing data to update the generator with DPO. At the same time, the critic is actively trained to identify criteria that the generator is unable to satisfy. This creates a dynamic in which the generator constantly improves its outputs as the critic finds weaknesses.

Auto-Rubric [12] aims to avoid the need for extensive preference data collection in LLM alignment by extracting generalizable evaluation rubrics from a minimal amount of data with a training-free approach. These rubrics are transparent and interpretable, unlike standard reward models that are trained over large volumes of preference data. To derive these rubrics, authors adopt a two-stage approach:

1. Query-Specific Rubric Generation focuses on creating rubrics that agree with observed preference data. After proposing an initial rubric set, we can check whether these rubrics yield correct preference scores and, if not, propose a set of revisions to derive an improved rubric set. This process repeats until the rubrics correctly predict human preference labels.

2. Query-Agnostic Rubric Aggregation eliminates redundancy and unnecessary complexity in the resulting rubric set. With an information-theoretic approach, the rubric set is narrowed to a subset of rubrics that maximize evaluation diversity without introducing redundancy.

Using this approach, Auto-Rubric can extract underlying general principles from preference data, allowing smaller LLMs to outperform large and specialized LLMs on reward modeling benchmarks with minimal training data.

Conclusion

Rubrics decompose desired LLM behavior into self-contained criteria that an LLM judge can score and then aggregate into an overall evaluation or reward. Put simply, rubrics are a practical middle ground between deterministic verifiers and preference labels that allow us to extend RLVR beyond verifiable domains while retaining granular control over output quality. The work we have studied suggests rubric rewards are most reliable when criteria are specific (often instance-level), grounded (via references or retrieval), and carefully curated (usually with human oversight). In more advanced setups, rubrics can also be updated based on on-policy behavior, allowing the rubric to adapt instead of becoming stale or exploitable. Despite promising results, key challenges remain; e.g., reducing reliance on human supervision and improving robustness in highly subjective domains. As reasoning models and LLM judges become more capable, however, rubric-based RL is becoming a viable and general tool across a wider variety of domains.

Bibliography

[1] Gunjal, Anisha, et al. “Rubrics as rewards: Reinforcement learning beyond verifiable domains.” arXiv preprint arXiv:2507.17746 (2025).

[2] Huang, Zenan, et al. “Reinforcement learning with rubric anchors.” arXiv preprint arXiv:2508.12790 (2025).

[3] Liu, Tianci, et al. “Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment.” arXiv preprint arXiv:2510.07743 (2025).

[4] Shao, Rulin, et al. “Dr tulu: Reinforcement learning with evolving rubrics for deep research.” arXiv preprint arXiv:2511.19399 (2025).

[5] Xu, Ran, et al. “Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training.” arXiv preprint arXiv:2602.01511 (2026).

[6] Xu, Wenyuan, et al. “A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization.” arXiv preprint arXiv:2504.04950 (2025).

[7] Zheng, Lianmin, et al. “Judging llm-as-a-judge with mt-bench and chatbot arena.” Advances in neural information processing systems 36 (2023): 46595-46623.

[8] Viswanathan, Vijay, et al. “Checklists are better than reward models for aligning language models.” arXiv preprint arXiv:2507.18624 (2025).

[9] Mu, Tong, et al. “Rule based rewards for language model safety.” Advances in Neural Information Processing Systems 37 (2024): 108877-108901.

[10] Gupta, Taneesh, et al. “CARMO: Dynamic Criteria Generation for Context Aware Reward Modelling.” Findings of the Association for Computational Linguistics: ACL 2025. 2025.

[11] Wu, Mian, et al. “Rlac: Reinforcement learning with adversarial critic for free-form generation tasks.” arXiv preprint arXiv:2511.01758 (2025).

[12] Xie, Lipeng, et al. “Auto-rubric: Learning to extract generalizable criteria for reward modeling.” arXiv preprint arXiv:2510.17314 (2025).

[13] Bai, Yuntao, et al. “Constitutional ai: Harmlessness from ai feedback.” arXiv preprint arXiv:2212.08073 (2022).

[14] Guan, Melody Y., et al. “Deliberative alignment: Reasoning enables safer language models.” arXiv preprint arXiv:2412.16339 (2024).

[15] Liu, Yang, et al. “G-eval: NLG evaluation using gpt-4 with better human alignment.” arXiv preprint arXiv:2303.16634 (2023).

[16] Arora, Rahul K., et al. “Healthbench: Evaluating large language models towards improved human health.” arXiv preprint arXiv:2505.08775 (2025).

[17] Deshpande, Kaustubh, et al. “Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms.” Findings of the Association for Computational Linguistics: ACL 2025. 2025.

9 More specifically, authors in [5] score each example twice, where the order of completions are flipped when generating the two scores. Then, only data that yields the same score for both orderings is retained for training.

Deep (Learning) Focus

I contextualize and explain important topics in AI research.

By Cameron R. Wolfe, Ph.D.

Thank you for being here, and I hope you have a wonderful day.

Dev <3

---

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

I’ve been digging into LLM geometry recently to answer questions like how models arrange concepts internally, how that structure changes inside a prompt, and how it affects what they can reason through. While going through some older work, I found this 2023 article I wrote as a breakdown of why chain-of-thought prompting works. Reading it now, the more interesting question was hiding underneath: what must the model’s knowledge look like for step-by-step reasoning to help at all?

The paper, Why Think Step by Step?, begins from a simple fact. Models rarely see every relevant variable together. They see local pieces of the world: a few related concepts in one example, another overlapping group elsewhere. The researchers found that intermediate reasoning helped when these local groups reflected the real dependencies between the variables. The model could make one accurate local inference, use it to reach the next, and eventually connect things it had never seen together. When the local structure was wrong, the extra steps stopped helping. There was no useful path to follow.

That makes this more than a result about prompting. Reasoning worked because the model had learned local relationships that could be composed. The intermediate text did not create new information. It helped the model move through information it already had.

More recent research has started examining that process inside the model. In-Context Learning of Representations found that examples in a prompt could reorganize the model’s internal representations around a newly defined graph—

Fluid Representations in Reasoning Models showed those representations becoming more abstract and aligned with the true structure of a task as reasoning continued. Another recent paper models reasoning itself as a trajectory through representation space, with correct and incorrect solutions following similar paths before eventually separating (which is similar to our findings with latent space reasoning, the reason that motivated this version to begin with).

Together, these point toward a better explanation for chain-of-thought. The extra computation may give the model time to build a better representation of the problem and move through a sequence of useful local relationships. If the concepts are arranged badly, or the model chooses the wrong neighborhood, more reasoning will not rescue it. It will simply take a longer route to the wrong answer.

This is the driving force behind the research I’m engaged in now. By understanding how how knowledge is organized geometrically inside language models, we can deliberately make useful inference paths easier to find and make high quality AI training much more efficient. We are running some experiments around this, which I’ll share as the results become clearer.

Before getting into those, I wanted to reshare this older research breakdown. It studies a simpler version of the same problem from the outside: when can a model connect locally learned relationships to reach something it was never taught directly?

As research into Large Language Models become more and more mainstream, we have seen a lot of research into how they can be used more effectively. One of the techniques that have unlocked the performance of LLMs at a higher level is chain-of-thought prompting. Instead of asking an LLM for an answer directly, we instead prompt the models to generate a series of intermediate steps. This leads to better performance in certain kinds of tasks. According to the paper, "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" -

This leads to a few interesting questions about why this technique works so well, and how we can leverage it effectively. The paper- Why think step by step? Reasoning emerges from the locality of experience- brings some very interesting insights about Chain of Thought prompting and Language Models. The authors explore Why Chain of Thought Prompting works (and when it will help). Their insights have some interesting implications in designing better datasets for language models. In this article we will be breaking down the findings from the Why Think Step by Step paper in more detail in order to explore these implications. If you’re interested in working with Autoregressive Large Language Models, this is not an article you want to miss.

The basic hypothesis that the authors put forth is relatively straightforward-

We posit that chain-of-thought reasoning becomes useful exactly when the training data is structured locally, in the sense that observations tend to occur in overlapping neighborhoods of concepts

The Setup

To test their hypothesis, the authors use Bayesian networks (super underutilised tool imo). The goal for an AI agent is to estimate learner conditional probabilities from the Bayes net andneeds to accurately estimate . The twist is that the learner may not see all variables together only the locally structured observations. By giving them access to locally structured observations, we can see if the agent can hippity-hop through the connected pieces to get to the final result.

To those of you that have difficulty sleeping until you see the formal definitions, you can catch it below-

There are 3 kinds of predictors used by the authors-

1. Direct prediction: Simply use the model to directly estimate the probability of the target variable given the value of the observed variable. This serves as a baseline.

2. Scaffolded generation: The scaffolded generation estimator represents ideal reasoning if we knew the best set of steps to work through. A scaffold is an ordered set S consisting of variables that were each observed with another scaffold variable and collectively d-separate the observed variable from the target variable. In the case of a chain, the scaffold consists of all variables between Yi and Yj . Variables are ordered by their distance from the observed variable in the Bayes net. We estimate each variable given the observed variable and previously-generated scaffold variables using q before estimating the target probability.

   

3. Free generation: This is like scaffolded generation but free generation uses the model to also choose which variables to instantiate rather than just to estimate their values. The authors sample variable indices and values from q until it generates the index of the target variable. Now, the probability of the target variable is computed averaged over M such samples. This estimator tests whether trained models spontaneously generate useful intermediate variables.

The training data is generated by using the following psuedocode-

Once BayessNet’s have been generated and selected according to certain criteria, we can select the sampled variables. The variables are selected according to three important criteria-

1. Locality- Observed samples contain only variables from a local neighborhood, consisting of a central variable along with all variables of distance at most k from it. To sample from the observation distribution, we sample the central variable uniformly randomly and then sample k from some distribution that controls the local neighborhood size.

2. Variable dropout- Even within a local subset of the world, we may not see everything at once. Certain variables may be missing or go unnoticed. We formalize this intuition with variable dropout. With some probability (0.2 in our experiments), variables are dropped from a local neighborhood and the learner does not see them. I really like the use of Variable dropout because it may also help a model generalize with more unseen pairs. Multiple research papers, including this one that we broke down, have shown, that the integration of dropout in models can be a game-changer for performance.

3. Held-out pairs Finally, target pairs of variables are held out across all training data. Performance at matching conditional probabilities for these pairs is our main performance metric. If a local neighborhood, after variable dropout, would include a pair of variables we decided to hold out, we randomly remove one of the two variables in the pair from the sample.

This is a fairly comprehensive way to account for the limitations of perception in learning. The authors combine this with control conditions-

We also create two control conditions to demonstrate the importance of a local observation distribution. As one control, we consider training data made up of local neighborhoods from the wrong Bayes net. This maintains the co-occurrence distribution structure, but the co-occurrences do not reflect the structure of which variables influence each other. As another control, we use a fully-observed condition where each sample contains almost all of the variables in the Bayes net. One of the two variables in each held-out pair is randomly dropped, but all other variables are included. These controls enable us to test whether local structure in the training data drives the value of reasoning.

-The researchers were very thorough with this one. One of benefits of reading high-level research is the exposure to well designed experiments.

and even a test to see how irrelevant variables influence the results-

These were the major components that stood out to me. Let’s move on to evaluating some of the results of their experiment-

The Results

The researchers had some interesting results that are worth paying attention to-

Firstly, we see that step-by-step prompting works when the observation distribution has the correct locality structure. When the training data is structured locally with respect to strong dependencies, both scaffolded and free generation perform significantly better than direct prediction—the reasoning gap. Scaffolded and free generation also perform significantly better than negative scaffolded generation, indicating that relevant intermediate variables help in predicting the target variable, but irrelevant intermediate variables do not.

Take a look at the image below.

It has some interesting implications-

1. More intermediate variables don’t seem to correlate with accuracy. This is somewhat counter-intuitive because I would assume that longer traces would lead to worse results.

2. The most wrong paths are the ones with the wrong local structure. This implies that training on local clusters of variables is valuable because it helps free generation produce intermediate variables that are relevant to the relationship between the observed and target variables.

3. Local training produced fewer intermediate variables than fully observed training (another surprise to me). This combined with the performance, implies that training on local training data might just a more efficient training approach than fully observed training.

We will now be exploring the last part. Take a look at the following analysis by the researchers

This has great potential for training LLMs in an efficient better. When I worked on changing English statements (written by business users) to SQL queries that had to be executed (potentially joining multiple tables), one thing I quickly learned was that AI could only do so much. I was able to build a somewhat working prototype by instead using a relatively basic AI (compared to the monstrosities we see these days) and focusing all of my efforts on restructuring the datasets in ways that made it easier for AI to interact with the datasets. This seems to be a similar principle.

The authors also discovered something very interesting about when Step by Step Prompting does not work- the worse a training condition does at matching the true conditional probability, the better it matches the marginal. The language models trained on data with the wrong locality structure generated estimates that were particularly close to the marginal probabilities. When the variables that co-occur with each other frequently are not local in the Bayes net, they often have very little influence on each other. This means that the joint distribution over co-occurring variables is usually very close to the product of the marginal probabilities, i.e. P(X1, X2, X3) ≈ P(X1)P(X2)P(X3) for non-local X1, X2, X3. Without the ability to estimate conditional probabilities accurately, there are no reliable ‘steps’ for step-by-step reasoning to use.

These results combine well to support the main hypothesis of the authors-

As an interesting aside, the authors noted that learning from local structures resembled human learning. This was fairly interesting because it reminded me off the chess master experiment. In an experiment, chess masters and noobs were asked to look at the configuration of pieces on a chess board and recreate that board from memory on a fresh one. The masters were able to recreate that board using way fewer glances than the noobs. However, what was interesting is that this same experiment was repeated, but this time the pieces were placed at random (creating configurations that would never exist in a chess match). This time there was no difference in performance between noobs and masters.

This experiment was used to show that the superior performance in the first task was not an inherent superiority in Chess Master mental abilities, but rather greater familiarity with chess boards and configurations which leads to superior pattern matching. Pattern matching is the key to expert-level performance, and local structuring might behave enable this pattern matching in LLMs. We covered this experiment in my article, How to Learn and Master Skills on my other publication Tech Made Simple here.

Subscribe to support AI Made Simple and help us deliver more quality information to you-

Flexible pricing available—pay what matches your budget here.

Thank you for being here, and I hope you have a wonderful day.

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

The AI market is going through a correction. Not a crash. We are correcting what we thought we understood about how this market works.

For the last three years, benchmarks stood in for capability, run rates for durable revenue, backlogs for future cash flow, and token prices for cost. June showed how unreliable those shortcuts had become.

The same question appeared everywhere: how do you verify what you are buying? Washington could not independently test the risk that led it to recall Anthropic’s models. Investors still cannot compare OpenAI and Anthropic’s economics. Credit markets can see Oracle’s capex and debt, but not the margins buried inside its $638B backlog. CFOs can see token bills, but not whether agents completed useful work. Open weights, despite inferior performance, gained volume because buyers can test, host, and keep the exact model they deploy.

All this tells the story of a market is not deciding struggling to figure out which claims can be trusted, and discounting the ones that cannot. In this article we will talk about:

• Frontier model releases became permissioned. The Department of Commerce disabled Anthropic’s newest models using an export-control theory designed for controlled technology transfers. The evidence was contested, the process bypassed normal regulatory channels, and the legal theory was never tested in court. Every major US lab now has an incentive to negotiate before launching.

• The frontier labs moved toward public markets. Anthropic and OpenAI filed confidentially for IPOs, which means audited margins, losses, customer concentration, and compute commitments will eventually replace private marks and selective leaks. SpaceX showed what can happen when an enormous private valuation meets public price discovery.

• Credit markets started pricing the buildout’s leverage. Oracle reported record growth and a $638B backlog, but also negative free cash flow and another enormous financing requirement. The demand is real. The question is whether the companies building the infrastructure can survive the wait for that demand to pay.

• CFOs started metering the agents. GitHub moved more usage costs onto customers. Salesforce offered to charge only for completed outcomes. Enterprises imposed caps, consolidated vendors, and started asking what an agent actually produced for the money it consumed.

• Open-weight models took the volume. Chinese models increasingly handled the cheap, repetitive layer of AI usage while proprietary US models retained the most demanding workloads. Lower prices helped, but so did the ability to inspect, host, reproduce, and retain the model without depending on a US vendor or government.

June did not resolve any of these questions. It showed that nobody is willing to leave them unanswered anymore. And that’s not something we can overlook.

Executive Highlights (TL;DR of the article)

• Frontier releases moved behind a checkpoint in 24 days. On June 2, Executive Order 14409 created a voluntary covered-frontier-model process with up to 30 days of government access before release, while explicitly denying that this was a licensing regime. Anthropic launched Fable 5 and Mythos 5 on June 9. Commerce issued its directive three days later, and Anthropic disabled both models globally within roughly 90 minutes because it could not screen every API user by nationality. The complete global block lasted 14 days, until limited Mythos access returned on June 26; the directive remained active for 18 days, until June 30. OpenAI launched GPT-5.6 to roughly 20 trusted partners on the same day the block partially lifted. The lesson was obvious: asking permission before launch is now safer than risking a recall afterward. This also makes model evaluations, independent audits, severity standards, and eventually data-provenance requirements unavoidable, because the government cannot run a permission regime using reports it cannot independently check.

• The frontier labs moved onto a disclosure clock, while SpaceX showed what happens when a private valuation meets public trading. Anthropic filed confidentially on June 1 and OpenAI followed on June 8. The filings are not public, so investors still cannot compare audited inference margins, customer concentration, compute commitments, revenue quality, or losses. That is why the filings matter. The current debate relies on leaks and private-round marks; the IPO process will eventually force both companies into a common accounting framework. SpaceX priced at $135, opened at $150, reached roughly $225 in three sessions, and fell below its opening price within three weeks. The company entered public markets with only 4.2% of its shares trading and much larger lockup releases still ahead. Nasdaq’s revised methodology also allows qualifying IPOs into the Nasdaq-100 after only 15 trading days, which means passive capital can arrive before the market has had much time to test the valuation. Anthropic’s first audited inference gross margin will do more than price Anthropic. It will reset how the entire sector is valued.

• Credit markets priced the gap between AI demand and the cash required to serve it. Oracle reported $638B in remaining performance obligations, $55.7B in annual capex, and negative $23.7B in free cash flow. Its five-year CDS spread had already reached a record of roughly 198 basis points in March, almost five times its level one year earlier; the June results explained why. Oracle must finance and build the infrastructure before most of the revenue arrives, leaving it exposed to delays, utilization, refinancing costs, depreciation, and customer risk. CoreWeave carries the same problem without Oracle’s software cash flows. Micron showed the other side of the market: when three suppliers control memory capacity and outsiders cannot verify their real allocations or production flexibility, scarcity produces extraordinary margins. The Bank for International Settlements then warned that debt-funded AI investment and infrastructure bottlenecks could end in overinvestment followed by a prolonged bust. Demand does not need to disappear. It only needs to pay later, or at lower margins, than the financing assumes.

• Agent demand stayed strong, but buyers started setting limits. GitHub moved Copilot to AI Credits on June 1, transferring the costs of long autonomous sessions, retries, failed searches, and tool calls to customers. Salesforce took the opposite position by charging only when its Help Agent resolves an issue without human escalation, leaving the vendor to absorb the cost of failure. Both models exist because flat-rate seats no longer work when agents can consume compute continuously. Enterprises also began imposing budgets and consolidating standalone tools into platforms where they controlled procurement, data, security, and telemetry. Sonnet 5 made the billing problem harder by introducing a tokenizer that produces roughly 30% more tokens on average than Sonnet 4.6, with one English test producing 1.42 times as many. A lower price per million tokens means little when the vendor can change how many tokens the same work contains. The useful metric is becoming cost per completed task, including retries, routing, tool use, human intervention, and failures.

• Open-weight models took the volume because buyers got lower prices and more control. OpenRouter found that Chinese models had overtaken US models in token volume by early June. Vercel’s production data showed open-weight models handling 29% of tokens while accounting for less than 4% of spending, with DeepSeek alone reaching 22.6% of volume. This does not mean proprietary frontier models are being replaced everywhere. The market is splitting. US models retain the workloads where the final increment of capability justifies the premium, while cheaper open-weight models absorb the repetitive, high-volume layer underneath. Buyers can also test the exact model version, host it privately, control the surrounding infrastructure, and retain access regardless of future pricing or policy changes. Washington can restrict an American API. It cannot remotely recall weights already running on private infrastructure abroad.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Section 1. AI Models Releases are getting very complicated

Anthropic has spent a long time begging the government to regulate open-weight models because they’re “too dangerous”. Turns out the government bought their fearmongering and ended up regulating Anthropic. Karma’s a bitch, ain’t it?

Schedeanfruede aside, the way it happened is actually worth studying.

1.1. What Actually Happened Versus What the Press Reported

The mainstream narrative says Anthropic shipped a dangerous model, a jailbreak proved the risk, and Washington stepped in to protect the public. The primary record destroys this version.

Our timeline here moves fast:

• June 1, 2026: Anthropic files confidentially for its IPO.

• June 2, 2026: Executive Order 14409 creates a voluntary “covered frontier model” path. It gives the government 30 days of pre-release access to evaluate cyber capabilities but explicitly disclaims any formal licensing or preclearance authority.

  June 9, 2026: Anthropic launches Claude Fable 5 publicly alongside a limited release of Claude Mythos 5 through Project Glasswing. Both models clear GPT-5.5 by five points on the Artificial Analysis index.

• June 12, 2026, 5:21 PM ET: The Commerce Department issues an urgent export-control directive signed by Secretary Howard Lutnick and managed by the Bureau of Industry and Security (BIS). Both models go dark globally inside 90 minutes.

• June 26, 2026: The block partially lifts. Mythos 5 access opens strictly for roughly 100 vetted US critical infrastructure operators.

• June 30, 2026: Commerce officially revokes the directive.

• July 1, 2026: Fable 5 returns globally with a newly appended safety classifier designed to block the flagged codebase exploit.

The complete global block lasted 14 days, until limited Mythos 5 access returned on June 26. The directive remained active for 18 days, until the Department of Commerce revoked it on June 30. Neither involved a hearing or court review.

The model recall here is so and so, but how it happened is a much bigger deal than people are pricing. The crisis began with an external jailbreak report showing a prompt path that allowed the model to map and surface exploitable software vulnerabilities across large codebases. Anthropic publicly called the technique “narrow and non-universal,” noting it belonged to an exploit class already present in other deployed models. The exploit arrived entirely outside standard bounty channels after Anthropic’s internal bug bounty had run for over 1,000 hours without producing it.

The escalation path bypassed standard regulatory channels. Reports indicate the chain ran from Amazon CEO Andy Jassy to Treasury Secretary Scott Bessent, then directly to Commerce. Amazon is Anthropic’s largest investor, its dominant compute vendor, and—via its Nova models—a direct competitor.

Take a second to consider what that means. The precedent for recalling new models is set— the evidence required to pull a rival’s frontier model from global distribution is roughly one returned phone call. Every lab now knows its competitors hold this leverage, and this will likely trigger a massive lobbying arms race to ensure that every lab has countermeasures against this.

The legal mechanism also creates a larger industry risk. Commerce invoked the “deemed export” theory. This doctrine treats a foreign national’s API session as a controlled technology transfer, a framework originally built for physical blueprints and missile components. Because real-time, per-user nationality screening across AWS, GCP, and Azure is impossible, the restriction is applied globally to all foreign nationals, including Anthropic’s own overseas employees. The minimum enforceable compliance unit for models is currently everyone.

Finally, it is worth noting that the legal theory died untested when the directive was lifted on June 30. This might seem like a win, but this is actually the worst outcome for the industry. Since the theory was never actually struck down, it is infinitely reusable.

The behavior of the labs shows where the leverage sits.

1.2. Why Every Lab Now Asks Permission First

On June 26, the same day Anthropic’s block partially lifted, OpenAI launched GPT-5.6. They did not release it to the public. They shipped it to roughly 20 government-coordinated “trusted partners,” gating general availability behind a review framework that finalizes on August 1. OpenAI watched a rival lose its flagship model for two weeks over an after-the-fact objection and converted recall risk into permission by asking first. Google’s delayed Gemini 3.5 Pro is now described in single-sourced reporting as “cleared for July.” Cleared is a word that did not exist in launch coverage two months ago.

In 24 days, the US technology ecosystem shifted from a voluntary framework with an explicit no-licensing disclaimer to a de facto licensing regime. No legislature voted on it, and no court reviewed it. It is enforced entirely by the industry’s memory of the week Fable went dark.

This emerging architecture has three parts:

1. Pre-release access: The non-negotiable price of a stable launch.

2. Partner lists: The state selects the initial 20 companies for distribution, turning frontier access into an allocated scarcity subsidy.

3. The recall: A permanent backstop that sits behind the arrangement, requiring no further enforcement because the industry saw it fire once.

The Great American AI Act (GAAIA) discussion draft would formalize this exact structure with mandatory third-party audits and $1 million-a-day civil penalties traded against state preemption.

1.3. Why AI Standards and Audit Infrastructure Are Now Inevitable

The focus on the political drama missed something crucial: a permission regime requires verification standards, and right now nothing checkable exists.

In June, Commerce recalled the most capable model ever deployed based on a jailbreak report it could not independently evaluate. The claim came from a source with competitive cross-interests against a model whose own 1,000-hour safety testing failed to reproduce it. The state had no internal evaluations to confirm or refute the risk, no accepted severity scale to categorize it, no independent auditor to consult, and no definition of “dangerous capability” beyond political consensus. The intervention was epistemically improvised. The regulators know an arbitrary process cannot survive sustained litigation or a changing administration. Improvised power either institutionalizes or evaporates. The August 1 framework signals they have chosen institutionalization.

This shift forces a missing verification industry into existence. If the state claims the right to check models before release, it needs evaluation standards independent of vendor claims, auditors who do not work for the labs, reproducible severity metrics, and chain-of-custody tracking over training inputs. You cannot certify what you cannot trace.

(somewhat poetically, we first predicted the rise of the verifier economy almost exactly an year ago, something we elaborated on in our predictions for AI in 2026. The market is starting to validate this)

The regulatory sequence follows a specific order:

• Capability evals arrive first: The immediate data layer the permission decision consumes.

• Data provenance and transparency mandates follow: The necessary evidence layer required when an audit finding faces litigation.

This follows the historical path of every major regulated industry. Securities trading generated GAAP and the accounting profession; pharmaceuticals forced GLP and clinical registries; aviation established type certification. AI simply compressed this transition into 14 days. Concurrently, the European Union’s General Purpose AI (GPAI) enforcement goes live on August 2, threatening fines up to 3% of global turnover. Two independent permission regimes are coming online simultaneously. Dual-compliance is now table stakes for frontier distribution.

The critical engineering problem is that the measurement layer underneath this entire architecture is broken. As Section 4 documents, benchmark scores are regularly distorted by fallback routing to secondary models, tokenizer adjustments break price-performance comparisons, and orchestration layers mask unbounded compute scaling behind single-model performance metrics. Regulators have mandated strict verification on a statutory timeline before evaluation science can reliably deliver it.

This vacuum will be filled quickly. Whichever entities define the evaluation parameters—whether standards bodies, a consolidated network of professional audit firms, incumbent labs protecting their position, or specialized evaluation startups—will dictate the market. In an audit regime, the entity that controls the test controls the commercial landscape. The AI standards-and-audit infrastructure market will form in earnest inside four quarters. By this time next year, we will analyze evaluation-standards politics the way we analyze hardware memory allocation.

Talking about numbers, the direct financial cost of the recall to Anthropic was bounded—estimated in the tens of millions against an abstract baseline of $129 million in daily compute and operational run-rate. The lasting structural repricing landed on the architecture of the industry:

1. Multi-model failover moved from an infrastructure cost-optimization to a strict compliance requirement, structurally weakening vendor lock-in.

2. Non-US enterprises learned that access to US-hosted frontier models can be revoked in 90 minutes. Mistral capitalized on this lesson immediately, leveraging a sovereign, on-premise deployment pitch to anchor its recent 3-billion-euro capital raise.

I expect every US frontier launch through the end of 2026 to ship with direct government coordination attached. At least one major lab will market its trusted-partner regulatory status as an enterprise security feature by Q4 (IBM on the comeback boizzz).

Regulation was only one part of the correction. The same demand for verifiable numbers reached the capital markets.

Section 2. What Do the Anthropic and OpenAI S-1s Show and Why Did SpaceX Stock Crash?

The AI sector’s center of financial gravity just moved from private rounds to public registration statements in thirty days. Anthropic filed confidentially on June 1, four days after closing a $65 billion Series H at a $965 billion post-money valuation. OpenAI confirmed its own confidential filing on June 8. SpaceX priced the largest IPO in history on June 11 and started trading on June 12, raising $75 billion at a $1.77 trillion valuation with the merged xAI riding inside the wrapper.

The timing isn’t a coincidence or a pagan ritual to honor the summer solstice. Late-stage venture capitalists and sovereign wealth funds are out of dry powder. When you need $60 billion or more per raise, you have exhausted the private crossover universe. The massive compute commitments have forced these labs public. Seen from that lens, the IPO wave isn’t a victory lap; it’s a desperate refinancing to use your grandma’s pension fund as liquidity, while providing the increasingly anxious tech investors an exit opportunity.

2.1. Why Are Both Lab Valuations Currently Unverifiable?

An interesting fact about the filings is that comparing both companies’ filings isn’t as one ot one as people are pretending. OpenAI and Anthropic both have weird wrinkles in their filings, just in different directions.

OpenAI’s problem is disclosed and quantified: a leaked $14 billion projected loss for 2026 against a consumer-heavy subscriber mix. It is ugly, but at least it is a written-down metric. Anthropic’s problem is quieter and much harder to price because it is a question of whether its revenue holds. A run rate built on metered agentic consumption is high-quality revenue only if customers keep consuming voluntarily. June was the month enterprise customers started saying no:

• Microsoft canceled Claude Code internally at 84% to 95% adoption.

• Enterprises spent the quarter swapping horror stories about token misconfiguration.

• The pattern of an engineering team discovering they burned an absurd sum through a misconfigured agent pipeline is now its own genre of corporate post-mortems, with major law firms like Kirkland & Ellis surfacing in these accounts.

We saw a pretty general push back against the tokenmaxxing trend, something we covered here—

Anthropic’s own behavior through the spring shows they know this. They raised Claude Code capacity limits three times in five weeks between mid-April and mid-May—an intervention most convincingly explained as trying to stop subscriber bleed toward Codex, which consumes roughly four times fewer tokens for equivalent tasks and has climbed back to around 6 million users.

On May 14, Anthropic announced that programmatic usage would split onto separately metered credits starting June 15. On June 15, they canceled the change the day it was due to take effect. You do not pull a billing change at the eleventh hour unless your customers threaten to mutiny. Then, on June 22, thirteen days after launch, Fable 5 exited flat subscription plans entirely to move to usage credits. A vendor metering its own flagship against its own subscribers two weeks into the product’s life is telling you everything you need to know about its unit economics. Even now, in July, Anthropic has constantly pushed back its dates for taking Fable away because of the pressure it’s feeling right now.

So the takeaway isn’t that either lab is winning. It is that nobody can currently rank these companies, and the S-1 process exists precisely to end that confusion. The first audited inference gross margin that prints will re-anchor the valuation of the entire sector. If Anthropic prices at or above its $965 billion private mark in October, every private AI asset re-rates against a public benchmark. If it breaks below the Series H price, the down-round signal will cascade through mutual-fund marks, employee RSUs, and every vendor-financing chain using lab equity as implicit collateral. October’s Anthropic debut is the single most consequential dated event of late 2026. OpenAI leaning toward a 2027 listing doesn’t look like patience; it looks like letting someone else test the ice.

Speaking of going public, there was a massive event in June that we need to talk about. SpaceX showed what happens when a private valuation finally meets a public market.

2.2. How Did SpaceX and Nasdaq Structure the Ultimate Retail Liquidity Dump?

The SpaceX IPO was certainly eventful. The prospectus claimed a $28.5 trillion total addressable market, a figure NYU’s Aswath Damodaran publicly ridiculed, writing that the document read like it was “written by Grok” and pointing out the IPO price sat 27% above his own discounted cash flow value. The structure fused a cash-generating satellite business to a deeply unprofitable AI lab. The company posted a $4.94 billion net loss for 2025, followed by a $4.28 billion net loss in Q1 2026 alone alongside $7.7 billion of quarterly AI capital expenditure.

The stock ran from its $135 IPO price to roughly $225 in three trading days, then gave all of it back. It traded below its $150 opening price within three weeks and fell roughly 25% off its post-IPO high by early July. Its first-ever bond issuance—$25 billion announced in late June—is investment-grade on paper but trading like absolute junk in the secondary market.

And the real supply has not even arrived yet. The float at IPO was only 4.2% of shares outstanding. Major lockup releases are expected to begin after the company’s first earnings report, with further tranches becoming available through the end of the year. Some releases depend on the stock meeting price targets, so a weak share price can delay part of the supply. It cannot remove the broader problem. A stock trading at 101 times sales is approaching a large, published increase in tradable shares.

As we covered before, this structure was built to give insiders exit liquidity at the expense of index-fund holders who never chose the exposure. Nasdaq’s revised methodology took effect on May 1, allowing qualifying IPOs to enter the Nasdaq-100 after 15 trading days without a minimum public-float threshold. Nasdaq separately announced five quarterly additions on June 11, effective June 22. The two events were separate, but together they showed how quickly newly public AI companies could be routed into passive portfolios. The same rules could allow Anthropic to qualify within weeks of an October debur.

The public listing helped SpaceX acquire Cursor for 60 billion USD, while giving away almost nothing thanks to their massive valuation and appreciation. We might see similar consolidations happen as the “AI will eat everything, so invest in distribution” thesis biases towards incumbents and bigger players.

The S-1s will eventually reveal the labs’ real margins. Credit markets did not wait. They started pricing the financing risk already visible underneath the buildout.

Section 3. How Credit Markets Started Pricing the AI Buildout

In March, this was still a thesis. We argued that the Western AI buildout was being financed through hyperscaler debt, vendor financing, infrastructure-backed loans, and increasingly circular investment loops. The system worked as long as demand arrived on schedule. Miss the ROI timeline, and the same leverage that accelerated the buildout would accelerate the repricing.

In April, the problem became operational. Power constraints, permitting delays, and speculative gigawatt announcements exposed the gap between announced capacity and capacity that could actually be funded and built.

Credit markets had started repricing the risk by March. June’s earnings showed why.

The question was no longer whether AI demand was real. It was who would finance the gap between today’s spending and tomorrow’s revenue.

3.1. Oracle’s CDS Spread Was the Audit Opinion Nobody Asked For

Oracle reported fiscal Q4 on June 10. Revenue reached $19.2B, up 21%; Oracle Cloud Infrastructure grew 93%; and remaining performance obligations reached $638B, up 363% year over year. On the old enterprise software scorecard, this was ridiculous. Oracle had produced what looked like the greatest earnings print in the industry’s history. But times are changin’ pretty fast.

So instead, the stock fell roughly 12% the next day.

Why?

Oracle spent approximately $55.7B on capital expenditure in FY26 and generated negative $23.7B of free cash flow. It raised $43B in debt and $5B in equity during FY26, then told investors it expected to raise another $40B through debt and equity in FY27.

Now, there is an important correction to the pure doom version of the story. Oracle said $75B of its large AI contracts involved customers either prepaying for GPUs or supplying the GPUs themselves. The entire $638B backlog is not an unfunded liability (which coincidentally is what my parents called me before I got a job; how proud Indian parents are of their son writing his newsletter instead of working for a respectable company is left to your imagination).

The problem is timing.

Oracle must build the infrastructure before most of the revenue arrives. It carries the financing, construction, utilization, depreciation, and counterparty risks in between. This is something that the credit market had already noticed. In March, Oracle’s five-year credit default swap spread reached a record of roughly 198 basis points, up from around 40 basis points one year earlier. The market was charging nearly five times as much to insure Oracle’s debt before the June results explained why.

Investors cannot independently verify the ultimate margins, utilization, cancellation risk, or counterparty durability embedded in $638B of long-dated contracts. They can verify $55.7B of capex, negative free cash flow, and another enormous financing requirement. So credit priced what it could see.

(July update: On July 9, S&P downgraded Oracle from BBB to BBB-, one notch above junk. S&P said Oracle’s expanding AI infrastructure business was weakening its traditional business-risk profile and admitted that it had underestimated how much investment the buildout would require. One of June’s proposed confirmation signals arrived within a month.)

The point here isn’t that Oracle is about to default. It is that Oracle is using its investment-grade balance sheet to bridge the gap between AI companies’ current cash flows and their future compute commitments.

OpenAI is the obvious concentration risk. They OpenAI burned $3.7B during the first quarter of 2026 against $5.7B of revenue. Reuters could not independently verify the internal documents, but the mismatch is large enough to show what Oracle is financing around. Oracle equity is therefore becoming a leveraged bet on more than AI demand. It depends on customers converting compute commitments into sufficiently profitable products before financing costs, depreciation, and infrastructure margins consume the upside.

If I am to put all of that simply, the 2-line takeaway is that while a backlog is traditonally seen as an asset, a backlog that requires tens of billions in financing before it pays is a …problem.

CoreWeave shows the same structure without Oracle’s software cash flows. It ended Q1 with $99.4B of revenue backlog and $2.08B of quarterly revenue, but recorded $536M of net interest expense and a $740M net loss. The company also signed a new $21B commitment from Meta and a multi-year agreement with Anthropic.

Again, demand is not the problem.

The company must finance GPUs, power, and data centers today against contracts recognized over several years. If delivery slips, utilization falls, customers internalize capacity, or refinancing costs rise, the backlog becomes less valuable. That is not good.

On June 28, the Bank for International Settlements finally named the structure. Its annual report warned that AI investment was increasingly being supported by debt and complex financing arrangements, while intense competition and infrastructure bottlenecks increased the risk of overinvestment followed by a prolonged investment bust. Reuters summarized the warning as a growing financial vulnerability created by high valuations and increasingly complex debt-financing structures.

The circularity became almost comically visible through SpaceX (god bless Elon). After SpaceX acquired xAI in February, Anthropic took the full capacity of the 300MW Colossus 1 data center. Google then agreed to pay SpaceX $920M per month from October 2026 through June 2029 for capacity including roughly 110,000 Nvidia GPUs.

The Anthropic and Google agreements carried a combined headline value of more than $70B over their stated terms. But headline value was not guaranteed revenue. Google could terminate after December 31 with 90 days’ notice and could exit earlier if SpaceX failed to meet its delivery obligations.

And that is AI financing in 2026. An AI lab rents compute from the parent company of a competing AI lab. The supplier records a massive contract. The customer records access to future compute. Outside investors see the headline value but cannot observe future utilization, contract durability, or the economics supporting either side.

Everyone gets a backlog.

The GPUs still have to earn the money. But that isn’t something we need to talk about. After all, who cares about profits when our benevolent tech oligarchs will solve all our society’s problems by making us dependent on their platforms? At that point, what is the alternative?

3.2. The Memory Oligopoly Learned What Unverifiable Capacity Is Worth

Everywhere else in June, verification arrived and prices corrected.

In memory, the margins showed what happens when verification remains weak.

Micron reported fiscal Q3 revenue of $41.46B, up from $9.30B one year earlier. Its non-GAAP gross margin reached 84.9%, compared with 39% one year earlier. The company then guided to $50B of revenue and approximately 86% gross margin for the following quarter. Those are not ordinary commodity-cycle numbers. Micron’s core data-center business reported an 87% gross margin.

Micron also announced strategic agreements covering 16 customers and approximately $22B of commitments. The agreements included deposits, purchase obligations, and pricing protections intended to make the current shortage more durable and predictable for Micron.

The shortage itself is real. AI systems require enormous quantities of high-bandwidth memory, while new fabrication plants and production lines take years to build and qualify.

What outsiders cannot verify is how much of the scarcity is unavoidable. Customers and analysts cannot independently observe manufacturers’ true wafer allocations, production yields, conversion schedules, or how aggressively each supplier could expand capacity at a lower margin. The manufacturers control the supply and most of the useful information about that supply. When supply is concentrated, capacity is opaque, and customers are desperate, suppliers can capture extraordinary scarcity rents without outsiders being able to distinguish physical constraint from commercial discipline.

Micron’s 85% margin is the receipt.

(July update: On July 10, SK Hynix’s chief executive said 2027 could bring the worst memory shortage in the industry’s history and that demand may continue exceeding supply beyond 2030.)

The consequence spreads through every AI capex forecast. A growing portion of infrastructure spending now goes toward higher memory prices rather than proportionally more usable compute.

The third repricing in the industry came through the macro data.

3.3. The Fed Discovered It Was Underwriting the Buildout

On June 25, the Bureau of Economic Analysis raised first-quarter real GDP growth to 2.1% annualized. Investment, information services, professional and technical services, and durable-goods manufacturing were among the major contributors. Meanwhile, real final sales to private domestic purchasers grew only 1.7%.

The BEA does not publish an official “AI contribution to GDP.” One private decomposition estimated that AI-related spending accounted for approximately 76% of first-quarter headline growth. That number depends heavily on which investment categories are labeled AI, so it should be treated as an estimate, not a national-accounting fact.

This means that AI infrastructure has become one of the main engines of American business investment at the same moment that underlying domestic demand was weakening.

Then on June 17, the Federal Reserve held its policy rate at 3.50% to 3.75%. But its projections turned sharply more hawkish. The median projected year-end 2026 rate rose to 3.8%, while the median 2026 PCE inflation forecast rose to 3.6% and core PCE to 3.3%.

Why are we talking about this? Simply put, this creates another kind of loop— AI capex supports economic growth. Stronger growth gives the Fed less room to cut. Higher rates then raise the financing cost of the infrastructure producing that growth.
The companies most exposed are not the cash-rich hyperscalers. They are Oracle, CoreWeave, and the rest of the leveraged layer being paid to build capacity for them.

The Nasdaq demonstrated the sensitivity on June 5. After the US economy added 172,000 jobs, more than twice the consensus estimate, expectations of another rate increase surged and the Nasdaq fell approximately 4.2%.

July update: The June inflation report, released on July 14, weakened the most aggressively hawkish version of this argument. Headline CPI fell 0.4% during June and rose 3.5% year over year, while core CPI was flat for the month and rose 2.6% over the year. That reduced the immediate probability of another increase, but it did not remove the underlying sensitivity of debt-funded infrastructure to interest rates.

There is also one necessary correction before the bears become drunk on their own seriousness.

The viral claim that half of planned 2026 US data-center capacity had been canceled was wrong. found that its forecast for genuine hyperscaler self-build capacity had changed by only around 1% over six months, while its colocation forecast moved by less than 5%.

The apparent collapse came from counting speculative announcements as real projects. Many lacked financing, tenants, equipment orders, interconnection agreements, or construction. In other words, they weren’t really serious projects to begin with.

Demand remained enormous during the same month the credit warnings arrived. Broadcom forecast $16B of AI-chip revenue for fiscal Q3, more than triple the year-earlier level, although the forecast still fell slightly below Wall Street’s expectations. OpenAI had also committed to purchasing up to 750MW of Cerebras capacity, with the expanded agreement reportedly valued above $20B.

Whether this buildout produces enough revenue to justify the capital depends on whether customers continue paying for the agents consuming all this compute. June gave us plenty to examine there.

Section 4. How CFOs Started Metering AI Agents

We’ve been talking about the breakdown of flat-rate agent pricing for a while now. June showed the two replacements: charge customers for the compute agents consume, or charge for outcomes and make the vendor absorb the cost of failure.

4.1. Input Metering Versus Outcome Pricing

On June 1, GitHub replaced Copilot’s premium-request system with GitHub AI Credits, priced at $0.01 per credit and calculated from the input, output, and cached tokens consumed at each model’s API rate. GitHub’s explanation was blunt: under the old system, a short chat request and a multi-hour autonomous coding session could cost the user the same amount, leaving GitHub to absorb the difference, which agentic usage had made “no longer sustainable.”

The base subscriptions remained at $10 per month for Copilot Pro, $19 per user for Business, and $39 for Enterprise, but usage above the included allowance became the customer’s problem, including failed searches, repeated test runs, bad tool calls, and retries. Some users reported projected bills up to 100 times higher, although these were individual reports rather than audited platform-wide data. GitHub added per-user credit consumption to its usage metrics API on June 19 and, by July 2, introduced cost-center credit pools and separate spending caps.

OpenAI moved in the same direction, then partially reversed course. It introduced pay-as-you-go Codex seats for Business and Enterprise customers in April, then stopped offering new pay-as-you-go seats to ChatGPT Business customers on June 24. Existing seats remained active and normal Business subscriptions still included baseline Codex access, but the self-serve path from experimentation to metered deployment lasted less than three months.

Finally, Salesforce took the opposite side of the risk. Its Agentforce Help Agent, launched on June 25 with pay-per-resolution billing scheduled for July, charges only when an issue is resolved autonomously from beginning to end, with no charge when the customer gives negative feedback or requests a human. Salesforce did not publish the per-resolution price in June, but the structure matters more than the number: under input pricing, the customer pays while the agent tries; under outcome pricing, the vendor absorbs the inference cost and gets paid only when it can defend the result as successful.

Salesforce says its own Help site has processed 4.3 million inquiries and autonomously resolved 70% of them, but outcome pricing does not remove verification; it relocates it. Every disputed resolution becomes a billing argument over whether the issue was genuinely closed, whether the user returned, whether a human corrected the answer, and who defines success. Watch every dispute require a 46-page submission of proof, so that most unhappy customers are annoyed into not submitting claims. Such is the nature of progress after all.

Both approaches exist because agents broke the core SaaS assumption that consumption per seat is bounded by human attention. Ramp’s June transaction data showed the resulting spread: across more than 70,000 US businesses, median AI spending was only $11.38 per employee per month, while the top 10% spent roughly $611 and the top 1% spent $7,449. Flat-rate pricing works on the median customer and collapses around the most aggressive users (and as more expensive models become common + more people start using agents, the cost percentiles will shift a lot).

4.2. The Month Demand Learned to Say No

The stronger signal came from buyers:

• Uber reportedly exhausted its full-year Claude Code budget during the first four months of 2026 and responded by imposing a $1,500 monthly limit per employee for each agentic coding tool, including Claude Code and Cursor.

• Microsoft offered the more revealing case. Its Experiences and Devices division, which includes Windows, Microsoft 365, Teams, Outlook, and Surface, began canceling most internal Claude Code licences and instructed thousands of employees to move to GitHub Copilot CLI by June 30. Claude models remained available through Copilot; Microsoft removed the separate Claude Code product and consolidated usage onto infrastructure, procurement, and telemetry it controlled.

The decision shows why enterprise buying isn’t as output-dependent as originally thought. A product can improve output and still lose the contract if the buyer owns a close substitute, can integrate it more deeply, and prefers to keep the data, spend, and bargaining power inside its own stack. A Microsoft study released in July found that engineers adopting command-line coding agents merged roughly 24% more pull requests over four months, although the authors warned that merged pull requests are only a proxy for value. Microsoft had internal evidence of measurable productivity gains and still removed most Claude Code licences because it did not need to rent Anthropic’s full product to capture them.

Once costs become visible, enterprises optimize across performance, integration, security, procurement, observability, switching costs, and vendor leverage rather than simply buying whichever model tops a leaderboard. The model can win the benchmark while the vendor loses the budget.

The spending data shows the same pattern. A January survey of 100 Global 2000 executives found that average enterprise LLM spending had risen from about $4.5M to $7M over two years, with respondents expecting another 65% increase during 2026, while 81% were testing or using at least three model families and 65% still preferred incumbent products when available. Budgets were growing, but they were also becoming fragmented, governed, and negotiable. Falling token prices did not produce infinite demand because cheaper intelligence still had to pass through procurement.

4.3. The Tokenizer Became a Pricing Weapon

Once buyers began auditing the bill, the billing unit itself became unstable. Anthropic launched Claude Sonnet 5 on June 30 at an introductory $2 per million input tokens and $10 per million output tokens through August 31, rising to $3 and $15 afterward. Sonnet 5 also introduced a new tokenizer that, by Anthropic’s own estimate, produces approximately 30% more tokens than Sonnet 4.6 for the same input, depending on the content.

Simon Willison’s independent testing found that identical documents produced 1.42 times as many tokens for English, 1.33 times for Spanish, 1.27 times for Python, and almost no change for Simplified Mandarin. On the English sample, the promotional rate therefore translates to an effective $2.84 per million old-token-equivalent input tokens and $14.20 for output, roughly matching Sonnet 4.6’s $3 and $15 pricing. Once the promotion ends, the same text costs an effective $4.26 and $21.30, a 42% increase despite an apparently unchanged price card.

Anthropic disclosed the tokenizer change, so the problem is not concealment. The problem is that vendors trained customers to treat tokens as a common unit while retaining the power to redefine the unit. Dollars per million tokens is not comparable across vendors if one tokenizer turns the same document into 42% more billable units.

The comparison breaks further when the product label no longer identifies one fixed model. Anthropic’s Fable 5 can route certain risky requests to Opus 4.8, while Sakana’s Fugu-Ultra orchestrates Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro through workflows of up to five steps. Fugu reached 73.7 on SWE-Bench Pro against 69.2 for Opus 4.8, not because Sakana trained a single superior base model, but because its router could buy work from several frontier models and combine the results.

That is a legitimate result, but it makes the current comparison framework useless. Benchmarks assume one model answered, price cards assume tokens are comparable, and leaderboards assume test-time compute is bounded or disclosed. Agentic systems can violate all three assumptions at once through routing, retries, hidden reasoning, tool use, and variable compute.

Thus, the only useful comparison is cost per successfully completed task, normalized for the tokenizer and accompanied by the full execution record: models called, retries, cached and uncached context, tools used, human intervention, and failure rate. Even that is difficult because vendors control most of the evidence. A May paper on token billing found that tokenizer ambiguity permitted 50.85% over-reporting below the detection threshold in simulated attacks, while hidden-reasoning inflation reached 1,469% in the most permissive setting.

This means that while a developer could detect Sonnet 5’s tokenizer change with a script, an enterprise audit layer must repeat that exercise continuously across tokenizers, routers, hidden reasoning, tool calls, cache rules, discounts, retries, and outcome definitions. The standards-and-audit market is therefore an attempt to build a trusted meter for a product whose manufacturer controls the ruler.

Credit desks were pricing the buildout, and CFOs were pricing the agents. Buyers then moved more volume toward models that were cheaper, easier to inspect, and harder to switch off.

Section 5. Why Open-Weight Models Took the Token Volume

On June 12, the Commerce Department forced Anthropic to disable Fable 5 and Mythos 5 because it could not verify users’ nationalities in real time. Both models disappeared globally on June 12. Mythos 5 partially returned for vetted US critical-infrastructure operators on June 26, Commerce revoked the directive on June 30, and Fable 5 returned globally on July 1. Four days after the shutdown, Z.ai released GLM-5.2 under an MIT licence, with downloadable weights and no regional lock.

That contrast explains most of the open-weight strategy. A hosted model remains controlled by its developer, cloud provider, and home government. Access can be restricted, prices can change, usage can be monitored, and the service can disappear without the customer’s consent. Once open weights have been downloaded and deployed on private infrastructure, no vendor can remotely disable them or rewrite the terms of access. For buyers outside the United States, particularly governments and regulated companies, that is not an ideological preference for open source. It is supply-chain control.

Open weights are also easier to verify. Customers can test the exact model version they will deploy, inspect its outputs across their own workloads, measure its real cost, control the inference environment, and reproduce results without depending on a vendor’s changing API. This does not make the system fully transparent; the training data and complete training process may still remain closed. But it removes several of the largest sources of uncertainty at deployment: hidden model changes, undisclosed routing, shifting tokenizers, regional restrictions, and vendor-controlled usage logs. In a market increasingly organized around auditability, the ability to hold the weights is itself a verification advantage.

The economics then make the decision easier. OpenRouter found that Chinese models had overtaken US models in token volume by early June. Vercel’s June production data showed open-weight models handling 29% of tokens while accounting for less than 4% of spending. DeepSeek alone reached 22.6% of token volume, while proprietary models continued to dominate spending and the most sensitive workloads. The market was not replacing the frontier; it was dividing the work. Proprietary US models retained the tasks where the final increment of capability justified the premium, while cheaper open-weight models absorbed the repetitive, high-volume layer underneath. FYI— we called out how China was explicitly investing in their models to handle this “good intelligence, at low prices” segment (90% of the token volume) all the way back in Feb, as you can see below—

GLM-5.2 showed how quickly that layer could improve. Z.ai kept the same 744B-parameter mixture-of-experts architecture, with 40B active parameters per token, but generated a large performance gain through reinforcement learning, long-context training, distillation, and inference optimization rather than another increase in model size. This matters because the frontier premium now decays faster: once a capability becomes reproducible through post-training, an open model can offer most of it at a fraction of the price and without the vendor dependency. More on how GLM 5.2 accomplishes frontier performance at fractional costs here.

That is the strategic problem for American labs and export-control policymakers. Closed models are easier to regulate, but every restriction makes open weights more valuable to buyers who care about continuity, sovereignty, and verification. Washington can control access to an American API. It cannot recall a model already running on private infrastructure across Europe, India, or the Gulf.

The American frontier remains more capable at the top end, but it is closed, expensive, and permissioned. Chinese open-weight models are increasingly setting the price for everything below that frontier. In a market moving toward verification, the model a customer can inspect, host, and keep may be worth more than the model that scores slightly higher but can be switched off from somewhere else.

What Happens Next

Most enterprise decisions are exercises in avoiding blame.

For a while, benchmarks gave buyers cover. You chose the model at the top of the leaderboard, and if it failed, at least you had followed the numbers. That excuse is becoming harder to use. Benchmarks can be trained against, token prices are not comparable, routers hide how much compute was used, and the best model can still be the wrong commercial product.

Something else will replace them.

That is why verification matters. Audits, approved evals, cost-per-task measurements, trusted-partner status, public margins, and deployment records will become the new evidence buyers hide behind. None of these measures will be perfect. They do not need to be. They only need to make the decision defensible.

This gives power to whoever controls that evidence. Governments decide which models are cleared. Clouds decide which models are bundled, routed, and discounted. Enterprise platforms control usage data and procurement. Auditors decide what counts as safe or effective. The model lab may build the intelligence, but someone else increasingly decides whether it can be bought, how its value is measured, and where it gets deployed.

The labs wanted intelligence to become the platform underneath every industry. The disrupters that would bring the hallowed silicon valley efficiency and pioneering spriti to every segment of the world. They may instead become suppliers inside platforms controlled by governments, clouds, and enterprise software companies.

Life really has a funny sense of humor, doesn’t it?

Thank you for being here, and I hope you have a wonderful day,

Dev <3

Share

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Thanks to everyone for showing up the live-stream. Mark your calendars for 8 PM EST, Sundays, to make sure you can come in live and ask questions.

Share

^^Bring your moms and grandmoms into the Chocolate Milk Cult.

Below is a walkthrough of Irys, the legal AI platform that we’ve been building. Sabih (co-founder, CEO, former BigLaw litigator) and Christian (head of product, also former lawyer) walked through a live litigation matter on the platform, showing the research pipeline, the drafting assistant, the matter-level knowledge system, and the personalization stack. The core thread running through the whole stream was trust — why legal AI adoption keeps stalling despite improving accuracy, what it actually takes to build AI that lawyers will put their name behind, and how the architecture we chose for Irys (stateful swarms, reasoning transparency, persistent memory) addresses the specific failure modes that have made every incumbent feel unreliable to practicing attorneys.

If you’re building for any regulated industry where professionals carry personal liability for their outputs, the problems and design decisions discussed here apply directly.

Use Irys to get all your legal work done for free here (irys.ai)

Community Spotlight: Kingdom

Kingdom (the manga) has been an absolute obsession of mine recently. It has a really cool plot, tons of fun characters, and a really good theme underpinning the whole story. I’d highly recommend it if you’re looking for a story to binge through (unfortunately I’ve caught up now, but the feeling of reading peak while having hundreds of chapters left was such a great feeling).

If you’re doing interesting work and would like to be featured in the spotlight section, just drop your introduction in the comments/by reaching out to me. There are no rules- you could talk about a paper you’ve written, an interesting project you’ve worked on, some personal challenge you’re working on, ask me to promote your company/product, or anything else you consider important. The goal is to get to know you better, and possibly connect you with interesting people in our chocolate milk cult. No costs/obligations are attached.

Companion Guide to the Livestream

This guide expands the core ideas and structures them for deeper reflection. Watch the full stream for tone, nuance, and side-commentary.

1. The Trust Gap That Ate Legal AI’s Promise

The Event — Sabih Siddiqi, co-founder and CEO of Iqidis and a former litigator with close to eight years at a major New York firm, opened with a claim that reframes how you should evaluate any AI system targeting regulated industries: the adoption problem in legal AI is not an accuracy problem. It’s a trust problem. And those are different things.

Why this matters — If you’ve been watching the legal AI market from the outside, you’d be forgiven for thinking that the race is about which system hallucinates least. Every vendor has some version of “we reduced hallucinations by X%.” That framing misses the actual bottleneck. Sabih had direct experience deploying incumbents at a firm with the budget to trial everyone — Harveys, Paxtons, the full roster. His assessment was blunt: none of them were worth anything. Not because the outputs were always wrong, but because a lawyer can never know when they’re wrong, which makes the outputs useless for the thing lawyers actually do — put their name on documents with career consequences. The failure mode here is subtle and it matters for anyone building for high-stakes professionals. A system that’s right 95% of the time but can’t prove when it’s right forces the user to re-verify everything manually. That re-verification eats the entire productivity gain the AI was supposed to provide. You end up rereading every document yourself, checking every citation against the source material, and confirming that nothing critical got dropped.

Sabih put it in terms of two axes: trust and work product. Most legal AI has tried to compete on work product alone — better drafts, faster research — while the trust axis remained unsolved. Lawyers have been sanctioned in court for relying on AI-generated citations that turned out to be fabricated. That’s not a hypothetical failure mode, it’s a documented pattern, and it creates an adoption ceiling that no amount of benchmark improvement can break through on its own. The question Irys set out to answer was whether you could move the trust axis independently of the accuracy axis, and that distinction drove most of the architectural decisions the team walked through during the stream.

2. Post-Hoc Checking vs. Showing Your Work

The Event — Dev broke down the current best-in-class approach to legal AI trustworthiness: retrieve your chunks, generate an answer from those chunks, then do a post-hoc verification pass where you check whether every cited case actually maps to a legal database. If it does, ship it. If it doesn’t, flag it. This is the approach that the most careful incumbents have adopted, and it represents real progress over the systems that simply hallucinated citations with no checking at all.

Why this reframes everything — Post-hoc checking treats trust as a filtering problem — generate first, verify after. The assumption is that the generation step might produce garbage, so you build a sieve at the end to catch the worst of it. It works, to a degree. But it has a structural limitation that no amount of engineering on the sieve can fix: the user never sees how the system arrived at its answer. They see the output and the citations, but the reasoning chain between input and output stays opaque. Irys takes a different approach. The system exposes its reasoning process at every step — which sources it’s consulting, how it’s interpreting the legal posture, what defensive strategy it’s identifying, which case law it’s pulling and why. A lawyer can intervene at any point in that chain, not just at the output.

This matters because legal reasoning isn’t just about getting to the right answer, it’s about getting there through the right process. Two lawyers can arrive at the same conclusion through completely different analytical paths, and in law, the path matters — it determines whether the conclusion survives scrutiny, whether it covers the right bases, and whether it holds up under adversarial pressure. The agentic harness Irys built doesn’t just coordinate retrieval and generation. It maintains a visible reasoning trace that functions like a transparent brief-writing process rather than a black box that occasionally produces the right document. Dev made the point that this transparency is also what enables the grounding to work — when you can see the system consulting specific case law from Court Listener and other providers in real time, the citations aren’t decorative footnotes stapled onto a generated text. They’re load-bearing elements that the system’s reasoning actually flows through.

3. The Context Rot Curve

The Event — Dev introduced what might be the single most useful mental model from the stream: an inverted V-shape that describes how agentic system performance relates to context volume. As you add context, the system gets better — more documents, more precedent, richer understanding of the matter. But that improvement hits a peak and then collapses, because the system becomes context-wrought and has to compact. The second you compact, key instructions get lost.

Why this reframes everything — If you’re building anything that requires sustained reasoning over large document sets, this is the curve you’re fighting. Most agentic architectures today run into this wall somewhere between the fifth and fifteenth interaction in a complex task. The context window fills up, the system either truncates or summarizes to make room, and that summarization is lossy in ways that are hard to predict and hard to detect. The instructions that get dropped aren’t random — they tend to be the constraints and nuances that were important precisely because they were edge cases, which means they occupy less space in any summary representation.

Sabih described this from the practitioner side. Before leaving his firm, he was closing out a large arbitration using incumbent legal AI tools and hit context limits after roughly ten prompts. Even when he tried to push through, the system’s memory of the matter had rotted — it was losing the thread of the case, forgetting earlier context, producing outputs that no longer reflected the full picture. For a lawyer against the clock, that’s worse than having no AI at all, because you’ve invested time building up a context that just evaporated.

The Irys architecture addresses this through what the team calls stateful swarms — the system persists its learnings and understanding of a matter rather than holding everything in a single ephemeral context window. Sabih showed matter folders containing a full litigation’s worth of documents, with an insights tab that maintained entity maps, key dates, court filings, and evidentiary connections across the entire corpus. The pitch is that the more you interact with the system, the better it continues to become, rather than hitting a ceiling and degrading. Dev mentioned the team has processed up to a billion tokens for a single matter.

4. Partner-Level AI vs. Associate-Level AI

The Event — Sabih drew a line between two tiers of legal AI capability that resonated beyond law into any domain where AI serves professionals. Associate-level AI follows instructions and produces competent first drafts. Partner-level AI understands context, anticipates needs, and catches things the user didn’t explicitly ask about.

Why this matters — The distinction maps to a real structural gap in how most AI systems are designed. A junior associate given a task will execute that task. A senior partner given the same task will first evaluate whether the task is the right one to be doing, check it against the broader context of the matter, and flag things the assigning attorney might have missed. Sabih gave a concrete example: Irys knowing your client’s birthday from the uploaded documents and flagging it when you’re about to send an email. That sounds trivial, but it encodes a fundamentally different relationship with context — the system isn’t just retrieving information in response to queries, it’s proactively surfacing relevant facts based on its understanding of what matters in the current workflow.

The more technically interesting version of this showed up in the demo. Sabih asked Irys to draft a responsive pleading with what he described as a deliberately bad prompt — just “draft responsive pleading” with no further specification. The system recognized the litigation posture from the uploaded documents, identified that the case was in the Southern District of New York, determined that the matter was at the answer stage rather than the motion stage, and produced a responsive pleading formatted accordingly. When they ran the same query through incumbent systems, those systems sometimes produced a motion instead — a fundamentally different document type that reflects a misunderstanding of where the case sits in its lifecycle. The difference between getting the document type right and getting it wrong isn’t a quality gradient. It’s a categorical error that reveals whether the system actually understands the legal context or is just pattern-matching on the prompt.

5. Targeted Surgery vs. Full Rewrites

The Event — Dev flagged a cost and trust problem that anyone building document-centric AI should think hard about: most systems, when asked to modify a specific section of a long document, regenerate the entire document with the changes applied. This is expensive, slow, and — critically for trust — introduces the possibility that unchanged sections got silently modified in the rewrite.

Why this reframes everything — The drafting assistant Sabih demonstrated takes a different approach. Instead of regenerating from scratch, it operates on the specific highlighted text. He selected a passage, asked the system to simplify it, and got back a redlined version showing exactly what changed with tracked changes that could be accepted or rejected individually. That workflow mirrors how lawyers actually work — they don’t rewrite entire briefs when they need to adjust a paragraph. They edit in place, with changes tracked. This is architecturally harder than full regeneration. It requires the system to maintain a stable representation of the document, identify the boundaries of the requested change, produce a targeted edit, and present the diff in a format the user can review. But it’s also what makes the system trustworthy for professional use, because the user can verify at a glance that only the intended section changed. Full regeneration, even when it produces a better overall document, erodes trust by making it impossible to know what else moved.

Dev’s broader point was that this granularity is also what makes the system cheaper to run — you’re not burning tokens reprocessing an entire document every time someone wants to adjust a clause.

6. Personalization Runs Three Layers Deep

The Event — Sabih walked through Irys’s personalization architecture, which operates at three concentric levels: the legal profession as a whole, the individual user, and the firm or organization.

Why this matters — Most AI personalization is a single-layer feature — you set some preferences, the system tries to match them. Irys treats it as a layered system where each level compounds on the ones below it. The base layer is the legal profession itself. Before a user ever logs in, the system already understands legal formatting conventions, document types, jurisdictional requirements, and procedural norms. The team described running thousands of inferences daily to refine this layer based on how lawyers actually use the platform.

The second layer is the individual. Users can set their risk tolerance, formality level, detail preferences, and drafting style. Or they can skip all of that and just upload a few of their best documents — the system builds a writing profile from the examples. Sabih said clients have told him the output “sounds scary similar” to their own writing. The system also handles multilingual input, with users switching between Spanish, English, and Arabic mid-sentence and the system maintaining coherence across the transitions.

The third layer is the firm. Workspaces pool institutional knowledge — document libraries, matter folders, organizational norms — and make it available across the team. This is where the product starts to feel less like a chat tool and more like a firm-level knowledge infrastructure that individual lawyers interface with. The compounding nature of these three layers is what Sabih pointed to when he said users spend hours per day in the platform and don’t want to leave. The system becomes more useful as it accumulates context at every level, which creates genuine switching costs that have nothing to do with lock-in tactics and everything to do with compounding institutional memory.

7. The Organic Expansion Signal

The Event — Sabih revealed something the team hadn’t planned to discuss: non-legal teams at client organizations have been pulling themselves onto the Irys platform without being sold to. One large organization that originally onboarded its legal department saw finance and accounting teams start using the platform independently, expanding from 37 seats to 77.

Why this matters for builders — Organic cross-departmental expansion is one of the strongest signals a platform can generate because it means the product is solving a problem that users recognize in their own workflows without being told the product can solve it. Iqidis positions as a legal AI company, but the underlying capabilities — long-context reasoning, document-grounded outputs, persistent memory, matter-level knowledge management — don’t have anything intrinsically legal about them. They’re general infrastructure for high-context, high-regulation professional work.

Christian made the point explicitly during the stream: this isn’t legal-specific. The architecture addresses problems that exist wherever professionals work with large document sets under accuracy constraints. Dev has talked publicly about investor pressure to position Iqidis as general-purpose reasoning infrastructure rather than a vertical legal play, and this organic expansion pattern is evidence for that thesis showing up in production rather than in a pitch deck. Whether Iqidis leans into that expansion or stays disciplined on legal as the beachhead market is a strategic question the team is still navigating, but the signal itself matters for anyone thinking about where vertical AI products hit their ceiling and what it looks like when the underlying technology outgrows its initial positioning.

Subscribe to support AI Made Simple and help us deliver more quality information to you-

Flexible pricing available—pay what matches your budget here.

Subscribe now

Thank you for being here, and I hope you have a wonderful day.

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Most AI failures get diagnosed at the level of the symptom.

The model hallucinates, so we tell it not to hallucinate. The agent repeats a failed command, so we add another instruction. RAG returns the wrong document, so we retrieve more chunks. The prompt gets longer. The system gets harder to understand. The original failure remains.

That happens because hallucinations, loops, bad citations, ignored instructions, tool failures, prompt injection, and false completion are not root causes. They are visible outputs produced by different mechanisms. The same symptom can require completely different fixes depending on what broke underneath.

This article is a diagnostic manual. I went through the current literature (every paper cited is linked at point of use so you can verify it yourself), talked to researchers and builders, and compressed every documented AI failure pattern into the smallest set of genuinely distinct root causes I could find. It explains the main mechanisms behind AI systems becoming worse over long conversations, fabricating evidence, misunderstanding intent, collapsing across long workflows, following instructions hidden inside external content, and claiming work is complete when it is not.

More specifically, in this article, we will cover—

• Why AI performance degrades the longer you use it, and what the context window actually is (it’s not memory)

• Why AI confidently fabricates information, agrees with your wrong premise, and can’t tell you when it doesn’t know something — and how to make it stop

• Why AI does the task correctly but misses what you actually wanted, and the one-line prompt addition that fixes most of this

• Why AI agents fail on multi-step tasks at rates that would shock you (the math is simple and brutal), and why “it worked yesterday” is a systems problem, not a model problem

• Why AI agents claim tasks are done when they aren’t, and how to build completion criteria that actually verify real-world state

• How prompt injection and context contamination are the same architectural failure at different threat levels

• The specific tests and prompt templates for each failure type, so you can diagnose and fix problems instead of retrying and hoping

By the end of this article, you will no longer use vague judgments about whether a model is “good” or “bad” and instead use a clearer question—What exactly failed, and at which layer?— to be able to quickly diagnose where your AI Systems are failing, and how you can fix them.

To access the full article—and all premium breakdowns going forward/written prior—upgrade to a premium subscription below.

If you believe deep insight deserves support, become a premium subscriber to allow me to keep doing the same.

Flexible pricing available—pay what matches your budget here.

Most companies offer learning or professional development budgets. You can expense this subscription using the email template linked here.

Read more

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Thanks to everyone for showing up the live-stream. Mark your calendars for 8 PM EST, Sundays, to make sure you can come in live and ask questions.

Share

^^Bring your moms and grandmoms into the Chocolate Milk Cult.

Martin Ford is a futurist and author who has spent nearly two decades studying how AI and robotics reshape the labor market, starting with his first book in 2009, well before job displacement was a mainstream economic debate. In Rise of the Robots: Technology and the Threat of a Jobless Future, Ford argues that AI and robotics will eventually displace most human workers. That’s the question this livestream sits on top of: how would that impact the economy, and can we adapt if it happens. The updated edition, out June 2nd of this year, folds in everything since ChatGPT.

Book: Rise of the Robots

X: @MFordFuture

Community Spotlight: Data Science Gems

Manish Gupta is a Principal Applied Researcher at Microsoft India R&D Private Limited at Hyderabad, India. He is also an Adjunct Faculty at IIIT, Hyderabad, and a visiting faculty at ISB, Hyderabad. His YouTube channel has a ton of fantastic deep dives on ML papers. Manish’s work combines his academic expertise with his real-life experiences to provide really insightful deep dives into some overlooked papers.

If you’re doing interesting work and would like to be featured in the spotlight section, just drop your introduction in the comments/by reaching out to me. There are no rules- you could talk about a paper you’ve written, an interesting project you’ve worked on, some personal challenge you’re working on, ask me to promote your company/product, or anything else you consider important. The goal is to get to know you better, and possibly connect you with interesting people in our chocolate milk cult. No costs/obligations are attached.

Companion Guide to the Livestream: Martin Ford on Why Jobs Break From Consolidation, Not Robots

This guide expands the core ideas and structures them for deeper reflection. Watch the full stream for tone, nuance, and side-commentary.

1. The Last Wall Before AGI Isn’t Intelligence — It’s Continuous Learning

The Event — Martin Ford, futurist and author of Rise of the Robots (2015, updated edition out June 2nd), opened by tracing 17 years of his own thesis. In 2009, warning that automation could eliminate jobs got you labeled a Luddite by economists. Today those same economists are studying the question directly, running labor statistics against the LLM rollout, and so far not finding a measurable disemployment effect. Ford’s read: everyone is measuring the wrong variable.

Why this reframes everything — The debate about AI and jobs keeps hitting a category error: people ask whether AI can do a task, when the real question is whether it can do the job. A job is a bundle of tasks held together by judgment, coordination, and the capacity to get better at things you don’t already know how to do. Ford’s example is a fresh graduate: nearly useless on day one, competent within six months. The entire gap is on-the-job learning — trying something, watching it fail, absorbing the correction, doing it slightly better tomorrow. That loop is the actual moat protecting most white-collar jobs, not raw capability. Frontier models ship frozen: whatever they know at release is what they know until the next training run, months out, at enormous cost. There’s no six-months-in version of a model quietly getting better at your specific job the way a new hire does. Ford treats this as the one variable worth tracking — the moment a model can learn continuously and generalize outside its training distribution, the standard defense that AI automates tasks but not whole jobs stops working. He frames it as a condition for the “AI complements workers” era ending, not a prediction that AGI itself arrives.

2. Predictability Beats Collar Color as the Automation Filter

The Event — Dev separated the AGI question from what’s already automatable, using a burger-flipping robot: technically buildable, but the edge cases in a fast-food kitchen are numerous enough that the ROI doesn’t justify the engineering spend. A coding agent is expensive to run well but clearly worth it, given the volume and value of software work it displaces. Ford generalized the pattern: the deciding factor isn’t blue-collar versus white-collar, it’s whether a job is routine, predictable, and verifiable against a large dataset of past examples done correctly.

Why this reframes everything — The old intuition was a skill ladder: physical labor is safe from software, cognitive labor is the target, and within cognitive labor, sophisticated work is safer than rote work. That model is already wrong on both axes. On the physical side, Ford points to Amazon warehouses: robots don’t need to solve the general chaos of the real world, they need a controlled environment where the messy variable — people moving unpredictably — is removed. That’s a narrower engineering problem than self-driving cars, which operate inside exactly the chaos warehouses design away, and it’s why warehouse robotics is advancing faster than autonomous driving despite both being “physical AI.” On the cognitive side, a financial analyst doing routine quantitative work is more exposed than a plumber, because the analyst’s task is narrow and repeatable with thousands of historical examples to train and verify against, while the plumber’s job is soaked in the unbounded, case-by-case variation that breaks pattern-matching systems. The filter that predicts automation risk: how much historical data exists to train on, and how cheaply an output can be checked against ground truth. Job title, salary, prestige, years of schooling — all noise.

3. Task Automation Doesn’t Eliminate Jobs. Consolidation Does.

The Event — Ford introduced a mechanism most of the conversation’s later arguments hinge on: even short of full job elimination, partial task automation triggers headcount consolidation. Take three people doing broadly similar work. Once half of each person’s task list gets automated, management doesn’t keep three people at half productivity. They merge what’s left into one or two roles.

Why this reframes everything — Ford reaches for electrification as the template. It took decades for factories to gain any productivity from electric motors, because early factories replaced the steam engine with an electric one and kept the same centralized layout: the shaft-and-belt system built around a single power source. The gain only showed up once factories redesigned around distributed motors, one per machine — an organizational change, not a technology swap. Ford expects AI to move faster than that multi-decade lag, but the shape of the delay is the same: the bottleneck is organizational restructuring, not capability. It’s also why economists currently finding no disemployment effect yet aren’t wrong. They’re measuring during the shaft-and-belt phase, before most organizations have reorganized around what’s already automatable. The visible layer right now is what Ford calls bottom-up adoption: employees using the tools on their own initiative, finishing work faster, and quietly keeping the slack time for themselves. Consolidation happens once management notices and reorganizes formally around the capability instead of the old task list.

4. Two Different Mechanisms Are Doing the Displacing, and They Compound

The Event — Ford drew a distinction people usually collapse into one question. Everyone repeats that ATMs didn’t eliminate bank tellers, which is true but beside the point: what actually cut teller employment was mobile banking, letting the customer do the transaction themselves with no employee in the loop. Dev connected this to Irys: some users get AI-assisted verification that compresses a task they’d already pay a professional for, like a faster contract review because a first pass flags what to check. Others get access to something they’d never have paid for at all, since the old cost of even a basic legal review priced them out entirely. Getting them from zero informed decision-making to a serviceable one, for free, is a different kind of gain.

Why this reframes everything — These aren’t the same displacement mechanism, and conflating them produces sloppy predictions. The first is direct substitution: technology does what an employee used to do. The second is self-service enablement: technology lets the customer route around the employee entirely, eliminating the role by making the human intermediary unnecessary rather than by replicating their output. Ford’s point is that both run in parallel inside the same organization. A company can use AI to make its analysts faster, which drives the consolidation from Section 3, while also building a self-service tool that lets a manager query the AI directly and skip the analyst altogether. Dev’s contract-review example is the self-service case at the consumer layer: nobody would previously pay $500 for a lawyer to read a routine agreement, so that market didn’t exist and no lawyer’s job was ever at stake there. Once the marginal cost of a good-enough first-pass review drops near zero, a market appears where there wasn’t one, and professional services shift toward many small transactions plus a shrinking core of large ones. Whether an industry sees mass job loss or this kind of bifurcated restructuring depends on which mechanism dominates. Regulated professions are the test case for why, covered next.

5. Liability, Not Capability, Is the Ceiling on Full Automation

The Event — Dev pushed on why radiology and law haven’t been automated despite years of confident predictions. Ford’s answer centered on liability, not technical capability: a radiologist who misses a cancer diagnosis can be sued for malpractice, and that legal exposure is load-bearing in the current system. There’s no clean equivalent once the decision-maker is a model instead of a licensed person.

Why this reframes everything — The unspoken assumption behind most “AI will replace doctors and lawyers any day now” takes is that the bottleneck is model accuracy: get the diagnostic model to radiologist-level performance and the job dissolves. The real constraint is that the entire liability architecture of medicine and law is built around a human who can be individually sued, disciplined, or stripped of a license. Nobody has built the equivalent for a company whose model makes an error across thousands of patients at once, or figured out what standard applies when the error is systematic rather than one practitioner’s lapse. The useful heuristic here: the relevant axis isn’t “regulated versus unregulated,” it’s how catastrophic and attributable a single error is. Booking your own travel or taking AI meal suggestions carries close to zero downside if the model’s wrong, so self-service automation faces no liability wall there. A misdiagnosed tumor or a botched contract carries severe, attributable downside, so an accountable human stays in the loop until someone builds a liability framework that can absorb the systematic-error case. That’s a legal and institutional problem, not a model-scaling one, and better benchmarks won’t solve it.

6. “Get More Education” Was the Answer to the Last Disruption, Not This One

The Event — Ford named the default policy response to worker displacement: retrain, upskill, send people back to school. Then he explained why it doesn’t transfer here. The historical template assumed machines took the routine, lower-skill work while humans climbed a ladder toward more sophisticated tasks that stayed out of reach. Ford’s point is blunt: AI is currently more capable at the sophisticated, credentialed end of that ladder than at a lot of blue-collar physical work. That inverts the assumption the entire “upskill your way out of it” policy rests on.

Why this reframes everything — If the ladder itself is being automated from the top down, “climb higher” isn’t a strategy. It’s a description of moving toward the part of the economy under the most pressure. Ford’s book has always argued for universal basic income as the least-bad solution, not an ideal one, because the retraining paradigm depends on a next rung that stays human-only long enough to retrain into, and that assumption is exactly what’s failing. He’s equally direct about UBI’s failure mode: strip out incentive design and you get a real dependency problem, sharpest for teenagers who see no reason to finish high school if the payout is identical whether they graduate or drop out. His proposed patch tiers the basic income higher for people who complete school, keep learning, or do recognized community work, preserving the incentive a paycheck used to provide without requiring an actual job to exist. Ford treats a second point as almost as important as the income question: a job isn’t just money, it’s identity and a socially legible answer to “what do you do.” A policy response that solves income while ignoring that solves half the problem.

7. Two Concentration Crises, Not One

The Event — This is where the conversation shifted from Ford’s established territory into a live disagreement with Dev. Ford’s model of economic collapse from AI is a demand-side story: consumers are the ultimate source of all revenue, even for a company like Boeing that never sells to an individual, because somewhere downstream an airline only buys planes if consumers are buying tickets. If AI concentrates income into fewer hands, the median consumer’s spending power collapses, and the whole chain down to Boeing eventually feels it, regardless of how much wealth exists in aggregate. Dev pushed back that this framing undersells a faster-moving problem: asset and perception-based wealth. His example was Elon Musk’s SpaceX IPO, a valuation surge driven not by new revenue or margin but by index funds getting structurally forced to buy the stock through rule changes at CRSP, Nasdaq, FTSE Russell, and S&P. That paper wealth converts into real political power. Musk can borrow against it, spend it, and by Ford’s own admission, plausibly influence a presidential election with it, without the underlying business having created any additional value at all.

Why this reframes everything — Ford held the line that these are two distinct problems, and neither substitutes for the other. Wealth concentration and political capture are real and will worsen with AI, but they’re separable from the income-concentration mechanism that actually breaks consumer demand and triggers a recession-style spiral. That distinction matters because the two problems call for different interventions: antitrust and campaign finance reform address the first, income redistribution addresses the second. Collapsing them into one narrative risks solving neither well. Dev’s extension is the sharper move: once a startup’s real audience becomes the investors who set its valuation rather than the customers who use its product, decision-making warps toward whatever appeals to that audience. That’s the benchmark-maxing, the researchers privately admitting they’re chasing metrics they don’t believe in, the institutional aversion to shipping anything that could dent valuation even if it would help users. Guy Debord’s society of the spectacle names this exactly: under enough media saturation, the packaging and perception of a thing displaces the thing itself as the product. It’s a dynamic anyone inside a modern AI lab has felt but rarely has vocabulary for. The appearance of progress toward investors becomes a competing objective against actual progress toward users, and the two increasingly diverge without anyone deciding they should.

Insight Closer: We’re Living in AI’s 2011

Ford’s calibration for how early this still is: the iPhone launched in 2007, and roughly three and a half years after ChatGPT’s release puts us at the rough equivalent of 2011 in smartphone years. Nobody in 2011 could have named social media’s effect on attention or fertility as a coming consequence of the device in their pocket. His own track record backs the caution. Writing the original Rise of the Robots in 2015, right after deep learning’s 2012 breakthrough in image recognition, he expected a Turing-test-passing system 20 to 30 years out, and expected whatever automation arrived to come from narrow systems built for a single domain like accounting. Instead we got one general model fine-tuned into a hundred vertical applications — exactly the pattern Iqidis and every other applied-AI company now runs on. Ford is the one person in this conversation who’s already had a forecast graded, and he undershot it by a wide margin.

Subscribe to support AI Made Simple and help us deliver more quality information to you-

Subscribe now

Flexible pricing available—pay what matches your budget here.

Thank you for being here, and I hope you have a wonderful day.

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

“A fire broke out backstage in a theatre. The clown came out to warn the public; they thought it was a joke and applauded. He repeated it; the acclaim was even greater. I think that’s just how the world will come to an end: to general applause from wits who believe it’s a joke.”

-Soren Kierkgaard. My GOAT.

Heat kills over half a million people every year, a number that is rising rapidly (up 63% since the 1990s, now averaging 546,000 annual deaths globally). In India alone, a single day of extreme heat causes an estimated 3,400 excess deaths. Europe buried 62,000 people from heat in 2024. And the Summer of 2026 seems to be breaking all records despite just starting.

This article is the first of a multi-series investigation where we will look at the tech industry and its role in the climate crisis (consider it an updated version of our 2023 article on the Profiteering from Climate Change).

In it, we will answer why — with all the pledges, commitments, and green marketing — the problem keeps accelerating. More specifically, we will cover how:

• Banks have funneled over $8.7 trillion into fossil fuels in the last 10 years — while marketing themselves as “Paris-aligned” (a farce they finally dropped recently after they had to stop pretending). This is especially given that it dwarfs the banks’ investments in renewables.

• Fossil fuel companies have publicly slashed their own climate pledges after posting record profits. Internal communications show decarbonization talk is PR, not strategy.

• Oil companies used the Ukraine-Russia war and other conflicts to price-gouge consumers. One company, Occidental Petro, saw a 721% profit increase in a single year.

• The revolving door between government and industry means civil servants pass favorable policy, then land cushy lobbying jobs — creating a self-reinforcing cycle.

• Over 90% of rainforest carbon offsets — the most popular corporate climate solution — are phantom credits that represent no real carbon reduction.

• Media platforms pledge to fight climate misinformation while running fossil fuel ads and branded content that launders industry reputations. This includes massive tech platforms.

This article will cover each of the above in more detail.

Part 2 will investigate data centers and AI’s climate impact — how much energy the AI boom actually consumes and whether the industry’s sustainability claims measure up to reality.

Part 3 will follow Big Tech’s money trail to break down how Big Tech companies and their leaders are funding the climate crisis. If you want to share insights into either topic, feel free to reach out (either using my social media links at the end of this article or by replying to this email).

Banking on Climate Chaos-

Banks play a crucial role in the fossil fuel industry- they help finance expansions. The 17th edition of Banking on Climate Chaos (BOCC) report finds that the world’s 65 largest banks committed $906 billion to fossil fuel companies in 2025, an increase of 8% from the previous year. Since the Paris Agreement was signed a decade ago, these banks have channeled $8.7 trillion into oil, gas, and coal operations.

Worst of all, we see an increase in the investments made specifically to expand existing fossil fuel developments—

“The top 65 banks committed $508 billion to companies expanding fossil fuel developments in 2025 — a $108 billion increase since 2024, or roughly 27% in a single year. Expansion finance is uniquely consequential as it locks in decades of future carbon emissions, future localized pollution, future supply shocks, and future stranded-asset risk. Every dollar of new oil, gas, or coal capacity built now extends a system whose recent shocks — from Ukraine in 2022 to Iran in 2026 — have already cost households and economies dearly.”

This is particularly damming b/c even if we stuck to just extracting the fossil fuels already tapped, we would still go over the 1.5 °C limit.

“We find that staying within a 1.5 °C carbon budget (50% probability) implies leaving almost 40% of ‘developed reserves’ of fossil fuels unextracted. The finding that developed reserves substantially exceed the 1.5 °C carbon budget is robust to a Monte Carlo analysis of reserves data limitations, carbon budget uncertainties and oil prices.”

-Existing fossil fuel extraction would warm the world beyond 1.5 °C

What about the regulation to make this happen? How are governments acting to keep this spike under control? That’s the neat thing, they aren’t. Instead, they are actively budgeting the failure in advance. According to the 2025 Production Gap Report, global governments are officially planning to produce more than double the amount of fossil fuels in 2030 than what is required to limit warming to 1.5°C.

The picture becomes even grimmer when we compare the share of money going to renewable energy with the share going into Fossil Fuels. Contrary to the marketing, the share of bank finance going to renewable energy rather than fossil fuels has little changed in six years (around 7% of their energy funds).

Banks have gotten so apathetic to climate goals that they don’t even bother pretending anymore. The Net-Zero Banking Alliance quietly folded and voted to wind itself down after a mass exodus of major financial institutions, with JPMorgan becoming the last of the big six U.S. banks to walk out the door.

In summary, banks contribute to the climate crisis in 3 ways: they finance fossil fuel expansion; they engage in greenwashing/token solutions instead of real solutions (such as having exclusion policies); and they haven’t increased the share of funding to renewable energy.

All of this are incompatible with Climate Targets of keeping global warming under 1.5 Celsius and must be addressed if Banks actually care about hitting their goals.

The briefing reveals that new oil and gas production approved to date in 2022 and at risk of approval over the next three years could cumulatively lock in 70 billion tonnes (Gt) of new carbon pollution. This is equivalent to almost two years’ worth of global carbon emissions from energy at current levels, 17 percent of the world’s remaining 1.5°C carbon budget, or the lifecycle emissions of 468 coal power plants.

Investing in Disaster: Recent and Anticipated Final Investment Decisions for New Oil And Gas Production Beyond the 1.5°C Limit

How Fossil Fuel Companies Block Climate Action

If their PR is to be believed, Fossil Fuel Companies (henceforth FFCs for simplicity) are fully onboard the decarbonization and Paris-Agreement train. They will be the first to tell you how they are improving their processes, investing in alternatives, and phasing out their carbon dependency. These poor Big Oil execs can’t even use their yachts and private jets without weeping about the additional emissions. None will be happier than them when decarbonization is achieved.

However (and I really need you to sit down for this), this might be a slightly distorted representation of reality. Internal communications highlight that these companies might be less excited about environmental sustainability than their public stance would imply.

Perhaps those are isolated incidents, a few bad apples in an otherwise swell group of tree huggers. When we compare the intensity of the pledges of the Big 4 Oil Companies to the intensity of their actions- we see a gross mismatch. “We found a strong increase in discourse related to “climate”, “low-carbon” and “transition”, especially by BP and Shell. Similarly, we observed increasing tendencies toward strategies related to decarbonization and clean energy. But these are dominated by pledges rather than concrete actions. Moreover, the financial analysis reveals a continuing business model dependence on fossil fuels along with insignificant and opaque spending on clean energy. We thus conclude that the transition to clean energy business models is not occurring, since the magnitude of investments and actions does not match discourse. Until actions and investment behavior are brought into alignment with discourse, accusations of greenwashing appear well-founded”.

These companies are lying to us so they can send neat little dividends to their shareholders. Who saw that coming?

This unforeseeable, completely shocking betrayal doesn’t end here. Turns out that Oil has become so profitable for these companies that they’re walking back on their public commitments. Not just sneaking around not hitting their targets, but publicly slashing them.

This might seem surprising at first- renewable energy production is increasing (especially in developed countries), so why are fossil fuels raking it in? The answer to that is in a little-known economics phenomenon known as Price Gouging. Price gouging occurs when suppliers decide to shoot up the prices of their goods/services, typically in response to a demand/supply shock. They know that people will have no choice but to pay these higher prices, so it’s an easy way to make money. Our oil dependency is a huge contributor to the recent inflation. Oil Companies capitalized on the Ukraine-Russia War to rake in record profits-

One company, Occidental Petro Corp saw a 721.49% increase in profits in one year. Other FFCs didn’t have a bad year either.

These revenues allow oil companies to act with a certain level of impunity. They have enough money to buy themselves out of trouble and have spent decades shaping societies to be more car and oil dependent. FFCs have shaped school syllabi to brainwash kids against climate change; engaged in media suppression (including lying to the government) about oil spills; and might have even paid hackers to target nonprofits working to uncover how much information ExxonMobil hid about their knowledge of fossil fuels and their role in climate change.

The New Joint Bicameral Staff Report Reveals Big Oil’s Campaign of Climate Denial, Disinformation, and Doublespeak is a great read for a deeper look at all the ways companies have been manipulating messaging and PR ton continue pushing their agenda.

For our purposes, there is one more avenue to focus on. Let’s now look into lobbying, and how FFCs have utilized lobbying to push laws in their favor. However, instead of looking at lobbying from a company perspective, let’s take look at lobbying from the perspective of civil servants. Why do they cater to the whims of these companies, even at possible personal and professional risk if they get caught? The obvious answer is money. But digging deeper actually gives us an interesting lesson in economic incentives. If nothing else, understanding how companies lobby (read- legally bribe) civil servants is very interesting. While I will be using data from the USA (just because it is more available and that’s where most of my audience is), the principles/methods are global.

The Economics of Lobbying

Of the roughly 2,200 lobbyists representing the energy sector in 2025, nearly half are former government employees (Inside Climate News / OpenSecrets). Half the industry's lobbying force walked out of government. Therefore any discussions around Climate Change will have to mention the lobbyists.

Lobbying is an interesting subject. Even though lots of people know it happens, they don’t quite understand how it happens. Or comprehend the scale of it. Before we proceed, I’m going to ask a simple question. I want to note down the answer before you continue reading.

If you know anything about American Politics, you know that the Clintons are a huge power-couple. Both Hillary and Bill have very established political careers, spanning decades. Here’s my question to you. How much money do you think they made giving speeches (729 speeches to be exact)?

Sit with this question. Think about the number.

No scrolling ahead till you have an estimate.

Trust me, it’ll ruin the surprise.

The answer is 153 million Dollars. No that’s not a typo.

Between 2001 and 2016, the Clintons combined to make more than 153 Million USD just in paid speeches.

What kind of wisdom do you think they were dropping in their speeches?

The Clintons are far from the only ones. Politicians are often invited to give talks, sit on advisory boards, and get involved with industry groups in other ways. These lucrative positions are all perfectly legal ways for corporations to get the ear of politicians and influence policy decisions in their favor. This is why politics is one of the most lucrative careers in the world. If you’re curious, google the net worth of your favorite politician. It will surprise you.

Let’s move from personal to professional. Aside from using highly lucrative positions to get access to politicians, companies have dedicated lobbying blocks to push their agendas. In 2025, the Oil and Gas Industry spent150 Milllion USD on lobbying.

This sounds like a lot, but the ROI on this would make insider traders on Kalshi proud. The OECD’s inventory tracked over 1,700 government support measures globally, revealing that public subsidies and financial support to fossil-fuel producers and consumers hit $916.3 billion. So you, the random public grunt, pay in 3 places: you pay for their subsidies, you pay for their overcharged fuel (where they will actively try to eliminate alternatives), and you will also pay to deal with climate change. Isn’t this an inspring vision for the future.

The other huge mechanism that companies use to influence politicians is campaign financing. People often overlook how expensive running a political campaign can be. When I was 14, I got to see a very hotly contested election from the front lines (I was observing the campaign of one of the contestants). I was too young to have any truly meaningful observations for you, but it is an extremely intensive endeavor (my first day, we started at 8 and ended at 2 AM). Money can swing the outcome of the election, as more money enables better campaign visibility and the capacity to drown out your opponent's narrative. FFCs are very active donors to the campaigns of various politicians-

Most talk about Lobbying ends here. However, there is another kind of lobbying that is overlooked b/c it isn’t as flashy. But it is equally effective (it might even be more so). Funding politicians is a top-down approach, that can be useful for leverage. However, this needs to be complemented with a more bottom-up, boots on the ground approach to be truly effective. Allow me to introduce you to the concept of a revolving door.

Let’s say you were a civil servant (politician, government employee, social worker etc.) working in the Oil and Gas sector. You see that you’re not as well-paid as you’d like to be. But you also don’t want to accept bribes. So, what do you do? Simple, you do your Oil Bros a solid and pass along policy/approve actions that make their lives easier. Do this for a few years and by the end you end up with an excellent job offer as a lobbyist for these FFCs. Now you get to live the good life, something your old colleagues in the public sector will see. Now they will start to follow suit, hoping to emulate your success. We have just created a culture where civil servants come in, enable policy to help corporate interests, and end up with cush jobs as lobbyists. That, my loves, is a revolving door.

This can also work in reverse. Sometimes senior corporate people can end up transitioning to the government to influence policy. A high-profile example is Jerome Powell, head of the Federal Reserve, who was formerly an investment banker. He’s not the only one. The relationship between Blackrock and the US Government is also well known-

These hires often come with expectations to ‘manage relationships’ (push favorable policy through). What a splendid example of Public-Private Partnerships!!!

While researching corporate lobbying and the revolving door, I came across several interesting conspiracy theories regarding them. I want to keep this focused on the facts, but if you’re bored, they can make for an interesting read. For now, let’s move on to the last 2 sections of this article. We will now be covering the wacky world of (faux) green solutions.

Looking at Not so Green Solutions

Remember how we talked greenwashing and how efforts taken by banks were inadequate. Time to get into that in more details-

Among the most common solutions used by companies is to rely on carbon credits. The idea is straightforward: if companies plant enough trees to suck away 1 Ton of Carbon, then they have reduced their emissions by 1 Ton. In principle, this is a great idea. However, this comes with a huge loophole. Instead of planting more trees- companies just buy plots of forests and claim that as their reforestation efforts. Often these plots of forest land are very remote and were never going to be cut down in the first place. They buy these for peanuts and continue polluting, while patting themselves for being eco-friendly. God bless corporate doublespeak.

The research into Verra, the world’s leading carbon standard for the rapidly growing $2bn (£1.6bn) voluntary offsets market, has found that, based on analysis of a significant percentage of the projects, more than 90% of their rainforest offset credits — among the most commonly used by companies — are likely to be “phantom credits” and do not represent genuine carbon reductions.

- Revealed: more than 90% of rainforest carbon offsets by biggest certifier are worthless, analysis shows

To those of you that like videos, this one is a great look at how Net Zero became completely worthless.

It seems like the Climate Tech/investing space is filled with such worthless ideas. Electric Cars, Monorails, and hyperloops are all touted as revolutionary ideas that will solve the climate issue. Much like FTX and Crypto, this is largely the result of VC funded Billion Dollar Hype Machines working overtime to present these as the future. Unfortunately, they face plant magnificently when introduced to reality. I don’t have the time to go into each of these in-depth, but the YouTuber AdamSomething has great videos on these topics and I highly recommend his channel here.

Another favorite of companies is using vaguely defined terms to mislead “conscious consumers”. Companies love throwing tags like ‘sustainably made’ and ‘fair trade’ and charging you higher prices. Here’s a secret- these don’t mean anything. Often these terms have no legal definitions, and companies throw these on without really changing anything. There’s a good chance you’re losing all those extra dollars without even voting with your wallet.

Unfortunately, even more established/proven solutions (turbines, solar cells, and lithium batteries) come with issues in the materials they rely on. These materials are toxic, mined through human labor exploitation, and can destroy the ecosystems around them. The following research paper is a good look at these issues and their solutions-

However, there are critical sustainability issues connected to the production of wind turbines, solar photovoltaic modules, electric vehicles and lithium-ion batteries. These include the use of conflict minerals, toxicity, and finite availability or supply chain governance risks of rare earth elements, cobalt, and lithium, that need to be taken into consideration. “Conflict minerals” refer to tantalum, tin, tungsten and gold, and their current mining is frequently linked to human rights violations and the financing of violent conflicts.

Critical sustainability issues in the production of wind and solar electricity generation as well as storage facilities and possible solutions

For a more human/personal look at these issues- the following article by the Guardian is a great look at how destructive lithium mining can destroy local communities and biodiversity-

In the mining installations, which occupy more than 78 sq km (30 sq miles) and are operated by multinationals SQM and Albemarle, brine is pumped to the surface and arrayed in evaporation ponds resulting in a lithium-rich concentrate; viewed from above, the pools are shades of chartreuse. The entire process uses enormous quantities of water in an already parched environment. As a result, freshwater is less accessible to the 18 indigenous Atacameño communities that live on the flat’s perimeter, and the habitats of species such as Andean flamingoes have been disrupted. This situation is exacerbated by climate breakdown-induced drought and the effects of extracting and processing copper, of which Chile is the world’s top producer. Compounding these environmental harms, the Chilean state has not always enforced indigenous people’s right to prior consent.

- The rush to ‘go electric’ comes with a hidden cost: destructive lithium mining

I don’t bring up these issues to discredit solar/wind power. On the contrary I believe that renewables will be the future (although I’m a huge believer in Nuclear Energy in the short-medium term). However, to fully leverage the potential of renewables, we can’t ignore major problems with them. We need to look at both their strengths and weaknesses, without agenda or bias and make informed decisions. Too much discourse around this issue either lionizes renewable energy or demonizes their problems. Neither is particularly helpful.

Speaking of agenda free analysis, let us look at the bastions of bias-free information- media platforms (including Social Media). What role has the media played in climate exploitation?

How the Media Profits from a Dying World

Media is considered one of the pillars of a healthy democracy. Ideally, the media serves to hold the powerful accountable and inform the regular folk about important ideas/developments. Information is power, and media helps level the playing field between the haves and have nots. That’s how it’s supposed to work. The reality is slightly different.

There are several media organizations doing great investigations/pieces on climate change (a very special shoutout to the writers of The Guardian). But the for-profit model often creates conflict of interests. Notable media organizations often have a platform for Branded Content- which is just a nice way to say fluff pieces. People can come to these platforms and pay for publicity (a trick used to gain a lot of social proof). FFCs often use these to spread their agenda. Notable media organizations have a long history of reputation-laundering for fossil fuel companies. Below is an example by the Washington Post.

These companies also must compete with Cat videos, Facebook, YouTube, Netflix, and edutainment behemoth AI Made Simple for your attention. Thus, they often rely on polarizing content, clickbait, and outrage to keep your attention. With topics as contentious as climate change and fossil fuels, this leads to muddied water and misinformation- preventing meaningful action.

It’s not only for-profit news organizations that are spreading the fossil fuel industry’s misinformation. If you listen to NPR podcasts, there’s a good chance that you’ve heard ExxonMobil touting the miraculous benefits of carbon capture, a technology that strips heat-trapping carbon dioxide molecules from smokestacks and other emissions sources. ExxonMobil’s ad, which has run on the Invisibilia, Up First, and Throughline podcasts, directs you to its website, which asserts that carbon capture technology could remove 90 percent of the greenhouse gas emissions from power plants, a claim wildly out of step with current or projected performance of the technology.

Is Your Favorite News Source Shilling for Big Oil?

This also extends to social media. Even though all social media companies have committed to fighting climate misinformation, they are happy to take oil money to spread the agenda. The article, Climate Misinformation on Social Media Is Undermining Climate Action, has great information on this.

This creates an interesting conflict- media platforms both act as a place where essential information about climate change is shared, and where misinformation runs rampant. And unfortunately, the clickbait/sensationalized versions tend to be more attention grabbing, attracting most of the attention.

When combined with Oil Industries’ deep pockets, much of what gets propagated is noise. And unless it stops being profitable for these platforms to run the propaganda, it’s not likely that they will stop.

Conclusion: Are We Cooked, Chat?

After 100+ reports, talks, and documents, the conclusion I couldn’t escape is this: none of what we covered is a knowledge problem. The banks know — they’ve read the same carbon-budget math we just walked through and committed $508 billion to expansion anyway. Governments know — the Production Gap Report is built from their own published production plans. Verra knew. The platforms cashing the ad checks know. Every actor in this article is responding rationally to their incentives. Which means the fixes are incentive fixes, not awareness campaigns:

Banks need exclusion policies, not alliances. A binding “no financing for new fossil expansion” rule would have stopped $508 billion last year. The NZBA’s quiet death showed exactly what voluntary pledges are worth.

Governments need to stop paying for the fire. End the roughly $900 billion in annual producer and consumer subsidies, and close the revolving door with multi-year cooling-off periods for officials who regulated the industries hiring them.

Offsets need hard verification or they don’t count. When 90% of rainforest credits are phantom, the burden of proof belongs on the credit, not the critic.

Media platforms need ad transparency. Disclose fossil fuel ad revenue, label branded content as the advertising it is, and apply misinformation policies to the industry that pioneered climate misinformation.

You and I can’t pass any of these. What we can do is reject the framing that this is about your straws and your showers — a framing BP’s ad agency built when it popularized the “personal carbon footprint.” Individual action counts where it aggregates into pressure: what you vote for, what you push your city to build, and which green claims you refuse to let slide.

Kierkegaard’s clown had it easier than we do. His audience simply didn’t believe him. Ours believes the fire is real — the people applauding loudest are the ones selling tickets. That’s the through-line of everything above, and it’s one sentence long: the climate crisis is not being ignored. It is being financed. $906 billion last year, growing 8% a year, with subsidies, lobbying, and branded content as the scaffolding.

Part 2 takes this lens to the industry I actually cover. Every hyperscaler building the AI boom has a net-zero pledge that predates the buildout. We’re going to check whether tech’s climate math is any better than the banks’. Early spoiler: the pledges are aging badly.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

If you liked this article and wish to share it, please refer to the following guidelines.

Share

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

GLM-5.2 is the best open-weight model on coding and agentic benchmarks right now. It scores 1524 Elo on GDPval-AA, a real-world agentic work benchmark — ahead of every other open model by a wide margin and level with GPT-5.5. It hit #1 on Design Arena’s Code Categories. It scores 81.0 on Terminal-Bench 2.1 and 62.1 on SWE-bench Pro, landing within a few points of Claude Opus 4.8 on long-horizon coding tasks — at roughly one-sixth the per-token cost of GPT-5.5.

Much more importantly, its intelligence has been validated by multitudes of users, which is a strong indication that Zhipu did not simply game the benchmarks to come out on top.

So what makes GLM 5.2 so good?

GLM-5.2 combines four specific techniques to eliminate memory and compute bottlenecks across the agentic loop. Rather than relying on raw model scale, these architectural mechanisms act multiplicatively to sustain throughput during long-horizon tasks.

• Deepseek Sparse Attention: Cuts long-context memory retrieval costs.

• IndexShare: Reduces hardware memory bandwidth overhead via cross-layer weight amortization.

• Multi-Token Prediction: Multiplies generation throughput by predicting parallel tokens per forward pass.

• Critic-Based PPO via SLIME: Trains the architecture to learn from complex, long-horizon agent trajectories.

In this article, we will study how each of these techniques comes together to give GLM 5.2 it’s god tier performance. While we will touch on the benchmarks and other techniques, I want to be very clear that they will not be the focus of this deep dive. My goal isn’t to give you a summary/overview of GLM 5.2, but to instead go deep on the few decisions that make GLM 5.2 stand out from every other model, and to understand the implications of GLM’s design for all builders and investors with one goal: understand GLM 5.2 beyond the headlines, so that you predict where the industry is headed next. While we will go very deep, you will not need much knowledge to read since we’ll develop our intuition from first principles.

If that sounds good to you, let’s dig right in.

Executive Highlights (tl;dr of the article)

Agentic loops have to contend with 3 issues that make them very expensive: Long-context histories (dealing with input, subagents, tool calls etc); unparallelized multi-step token generation (from the subagents); and broken training loops. Zhipu’s GLM-5.2 tackles these distinct bottlenecks across both the training and inference layers.

• The Long-Context Memory Wall: Standard multi-head attention hits a quadratic complexity limit at extended contexts. GLM-5.2 implements content-dependent DeepSeek Sparse Attention (DSA) to compute attention only over a sparse subset of the top 2,048 tokens per head, yielding a 2.9x reduction in per-token FLOPs at a 1-million-token window. In the main article, we map the exact hardware trade-offs of this approach — specifically tracing how IndexShare reuses token selections across fixed four-layer blocks to eliminate 75% of indexer computations, delivering a 1.82x prefill speedup at the cost of severe KV cache fragmentation and increased systems serving complexity.

• Unblocking Latency in Multi-Step Generation: Standard autoregressive generation forces millions of sequential, unparallelized forward passes to process complex tool calls and reasoning traces. GLM-5.2 addresses this output bottleneck by expanding speculative decoding to a 5-token draft window. Further down, we break down the architectural sharing, rejection sampling, and joint total variation loss training that Zhipu used to minimize distribution differences between models, pushing the average speculative acceptance length to 5.47 tokens per pass.

• The Mathematical Breakdown of Credit Assignment: While the industry has widely adopted DeepSeek’s GRPO to eliminate value networks and save GPU memory during training, uniform trajectory-level rewards completely blur credit assignment on long-horizon agentic runs. GLM-5.2 explicitly returns to a heavy critic-based PPO pipeline. Our deep dive will provide a first-principles analysis of this transition, demonstrating how independent, per-token value baselines isolate decisive debugging steps and integrate with compaction-aware training to slash a 16-hour group-sampling compute requirement down to 2 hours.

• Adversarial Exploitation and Post-Training Consolidation: Maximizing a generalist model across conflicting domains (coding, web search, math) usually triggers catastrophic forgetting. Zhipu resolves this by optimizing separate domain specialists and merging them via On-Policy Distillation (OPD). We examine how the SLIME post-training infrastructure consolidated over 10 specialists in 48 hours, while detailing the two-stage online validation modules (rule-based filtering and inline LLM judges) deployed to stop agents from aggressively reward-hacking infrastructure, downloading solutions via curl, and auditing evaluation files during reinforcement learning.

It’s worth noting that one of the key drivers of GLM 5.2’s amazing performance is their exceptional multi-level design where each of their techniques feeds each other.

On the inference side, we see a nested mathematical multiplier:

• DSA first restricts the context window to a fraction of its original size →

• IndexShare then executes inside that already-reduced window, bypassing 75% of the remaining indexing passes →

• Multi-Token Prediction takes these combined upstream memory savings and compounds them during generation by outputting multiple tokens simultaneously per verification pass.

The hardware is not just receiving individual linear speedups; it is running fewer attention operations, executing fewer indexing passes within those operations, and maximizing output density on every final activation.

On the training side, the anti-hack module acts as the first gate, intercepting exploits to protect the objective function from corruption. Because the training data remains clean, the critic network can map an authentic value landscape across compacted sub-traces. This calibrated value landscape allows the PPO gradient to isolate precise token-level advantages, which directly accelerates policy updates per unit of compute. Finally, On-Policy Distillation locks in these completed, domain-specific updates without gradient interference.

This to me is the true takeaway from GLM 5.2 — systems engineering will always beat individual techniques. Far too many team spend their time optimizing for individual aspects of performance without considering the entire system they’re operating in. When making building/investment decisions, keep this in mind.

FYI: We interviewed Zhipu AI over here. If you’re interested in their building philosophy, you might find it interesting.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Why are Agentic Loops So Expensive?

Agentic loops have a very specific computational signature that creates compounding resource strains in three ways:

• Repeated long-context retrieval: An agent working on a complex task continuously accumulates context. Each turn of the loop requires the model to re-query this entire history. Because standard attention mechanisms scale quadratically with sequence length, attending to a 100,000-token history costs $10,000$ times more compute per attention head than a 1,000-token history. The longer an agent works, the more expensive every subsequent decision becomes. (read more about the costs of running AI here).

• Sequential multi-step generation: Unlike simple question-answering systems that return a single response, agents generate massive streams of tool calls, code blocks, and reasoning traces across many consecutive steps. Autoregressive decoding forces each token to wait for the previous one, requiring a separate forward pass. An agent generating 50,000 tokens across an entire trajectory demands 50,000 sequential, unparallelized model activations.

• Long-horizon credit assignment: Training agents using Reinforcement Learning (RL) suffers from sparse rewards. A coding agent might perform hundreds of distinct actions over several hours. It will only receive a binary success or failure signal when the final test suite runs. Figuring out how important each step was to the overall outcome creates a massive credit assignment problem. Standard RL relies on gathering thousands of complete trajectories to isolate variables, but this is basically impossible to do effectively with all the sources of variance in agentic runs.

GLM finds ways to attack these problems both in the training and inference layers. Let’s get into them, one at a time.

How GLM 5.2 Uses Deepseek Sparse Attention to Reduce Long-Context Costs.

As we’ve talked about before, standard MHA has quadratic complexity since every token has to attend to every other token.

To avoid this quadratic cost, earlier efficient attention methods like Longformer used fixed sliding windows and predetermined sparse patterns. They forced the model to attend to nearby tokens and a few global positions regardless of the text content. Because fixed patterns cannot adapt to the actual input, these methods suffer from a severe quality gap.

Linear attention variants like GDN attempted to scale context efficiently but still introduced measurable degradation. They dropped up to 5.69 points on the RULER benchmark at 128K context and 7.33 points on RepoQA. Coding agents break these rigid architectures. An agent debugging code might need to link an error message from 500,000 tokens ago with a function definition from 200,000 tokens ago. Fixed or linear patterns completely miss these non-contiguous, long-range dependencies.

The DeepSeek Sparse Attention Solution

“We use DSA in our training. The core philosophy of DSA [9] is to replace the traditional dense O(L 2 ) attention — which becomes prohibitively expensive at 128K contexts — with a dynamic, finegrained selection mechanism. Unlike fixed patterns (like sliding windows), DSA “looks” at the content to decide which tokens are important.”

— From the GLM Technical Report.

DeepSeek Sparse Attention (DSA) changes this by using a content-dependent, two-stage process. Instead of hardcoding which positions to watch, a lightweight indexer predicts which tokens are most relevant to the current query in real time. Attention is then computed only over that sparse subset, limiting the workload to the top 2,048 tokens per head.

This gets you a 2.9x reduction in per-token FLOPs at a 1-million-token context, slashing the cost of every agentic loop execution drastically. Z.ai applied DSA to every layer of the model, claiming the mechanism is lossless (look at the chart below) — 

Developments like this are very exciting since they put a downward pressure on the already heated chip memory market. However, this solution comes with it’s own set of challenges. s the DSA access patterns paper documents, the token-dependent selection pattern means the KV working set becomes fragmented and volatile, creating poor cache locality and potential throughput stalls during decoding — 

This kind of optimization adds a lot more system complexity to your serving process. This is a class of AI startup/system that’s completely overlooked in the current era —  non-genAI solutions that work on to optimize and flag things like buffer overflows, memory management, and garbage collection. This has been valuable before AI, but the market will open up a lot more now that every inference call made compounds the value of resource optimization.

Even done well, DSA solves the attention scaling problem, but it introduces a new one: the indexer itself. At 1M tokens, the indexer has to score every token in the context to select the top 2,048 — and it has to do this for every layer in the model. The indexer computation becomes a meaningful fraction of the total cost. Try to fix one problem, and your solution creates new problems. Life really is fun that way.

Let’s look at how Zhipu tackled that next.

How GLM 5.2 uses IndexShare to cut the Cost of Sparse Attention?

The naive implementation of content-dependent sparse attention assumes that every layer must look at entirely different parts of the context. Under this assumption, an architecture computes a fresh top-k token selection at every single layer. However, this brute-force approach wastes enough hardware resources to turn Nvidia interns into millionaires since forcing every layer of an LLM to materialize its own indexer requires constant, redundant memory round-trips to evaluate token relevance across long sequences. When an agent runs a 1-million-token context, these repetitive indexing passes create a major throughput tax.

This is what IndexShare was built to fix.

Instead of running a separate indexer inside every transformer layer, the model executes one lightweight indexer across a grouped block of four layers and reuses the selected token indices across that entire group. The indexer sits at the first layer of the four-layer block, computes the top-k selection, and passes those identical indices to the following three layers for their sparse attention math. This structural change eliminates 75% of all indexer computations.

This optimization exploits the high similarity and continuity in adjacent layers of neural networks. Empirical measurements of top-k token overlap in dynamic sparse attention models reveal that adjacent layers share between 70% and 100% of their top-k selections near the diagonal. The tokens relevant at layer N remain fundamentally relevant at layer N+1.

(this similarity breaks down at transition layers, where the model shifts from one processing mode to another. The same paper reports overlap dropping to 0.4 or below at certain layer boundaries. Sharing indices across those transitions hurts quality. This is why IndexShare uses a fixed 4-layer group size rather than sharing across the entire network — it is calibrated to stay within the high-overlap regime while avoiding the dangerous transitions. They also use loss-aware selection, meaning the sharing policy is explicitly calibrated based on its direct effect on the training objective.)

Amortizing the indexing workload yields immediate hardware efficiency gains. IndexShare delivers up to a 1.82x speedup during the prefill stage and up to a 1.48x speedup during autoregressive decoding. For production agent deployments, these gains compound. Because an agent spends hours continuously prefilling extensive code history and sequentially decoding multi-step reasoning traces, these hardware speedups accelerate every individual execution turn. Even under fairly conservative assumptions, this can net you six figure savings — 

How does Multi-Token Prediction Accelerate Agent Output for GLM 5.2?

DeepSeek Sparse Attention and IndexShare reduce the hardware cost of reading context. Multi-Token Prediction (MTP) addresses the opposite side of the agentLoop: reducing the cost of writing the output.

Standard autoregressive generation requires a full model forward pass for every single output token. The model produces token one, executes a complete pass, produces token two, executes another complete pass, and repeats this loop sequentially.

This doesn’t work for agents since an autonomous agent routinely outputs tens of thousands of tokens per execution turn to process tool calls, long reasoning chains, and full code blocks. This means your system ends up with millions of sequential passes, slowing down your output tremendously.

GLM-5.2 shifts this dynamic via speculative decoding, using a smaller, low-cost “draft” model to guess candidate tokens quickly so the large target model can verify them simultaneously in a single forward pass.

While previous versions used a 3-token draft window, GLM-5.2 extends this proposal capacity to 5 tokens. Data from the architecture’s implementation reports a compounding optimization path that maximizes the average number of accepted tokens before a mistake forces a restart:

• Baseline Performance: The raw configuration starts with an average acceptance length of 4.56 tokens.

• Architectural Sharing: Integrating IndexShare and KVShare directly into the MTP layer allows the draft model to reuse internal structural context, lifting the acceptance length to 5.10 tokens.

• Statistical Refinement: Applying rejection sampling during token verification pushes the average approval metric to 5.29 tokens.

• Joint Training: Implementing end-to-end total variation loss forces the draft and target models to minimize their distribution differences during training, maximizing alignment and driving final acceptance length to 5.47 tokens.

So far, everything we’ve covered is relatively straightforward. However, the last technique changes a lot of things — 

How GLM 5.2 Changed Reinforcement Learning for Agentic Systems?

DSA, IndexShare, and MTP lower inference costs, but maximizing agent quality is a training problem. GLM-5.2’s return to a critic-based reinforcement learning pipeline marks a deliberate departure from recent industry trends. Evaluating this architecture requires analyzing why standard algorithms diverge on long-horizon tasks.

Background: How GRPO Works and Why It Dominated

Ever since Deepseek, Group Relative Policy Optimization (GRPO) is the standard reinforcement learning approach for large language models. The algorithm samples a group of N complete outputs from the current model for each training prompt, typically using 8 or 16 rollouts. A reward function scores each output, and the system computes a relative advantage by subtracting the group’s mean reward and dividing by the standard deviation.

GRPO assigns this group-level advantage uniformly to every token in that trajectory, giving token 1 the identical credit signal as token 10,000. The model update adjusts the probability of the entire token sequence based on this trajectory-level outcome.

This approach took off b/c its design eliminates the traditional PPO critic network, which requires a separate value network matching the policy model’s size and doubles the GPU memory footprint during training.

However, this has a slight problem..

Why GRPO Breaks on Agentic Tasks

GRPO relies on structural assumptions that long-horizon agentic tasks violate in four specific ways:

• Linear Scaling of Generation Costs: Sampling 8 to 16 complete trajectories requires minimal compute for short tasks, but a coding agent running a two-hour task across 50,000 tokens is a needy motherfucker that requires hours of compute per training prompt. This linear cost curve makes group generation prohibitively expensive, forcing developers to restrict group sizes and degrade advantage imprecision.

• Context Compaction: Production agents utilize context compaction to periodically summarize early history and respect context windows, breaking single long trajectories into variable sub-traces. One rollout might yield two compacted fragments while another produces eight under the same prompt. GRPO cannot apply its comparison framework because the fragments have different lengths, varying internal alignments, and non-aligned structures.

• Uniform Advantage Blurs Credit Assignment: Assigning an identical advantage weight to every token in a 50,000-token agent trajectory treats boilerplate imports and routine helper functions as equally valuable to success as a critical debugging fix at token 30,000. GRPO cannot isolate the few decisive actions among thousands of neutral tokens. The signal-to-noise ratio degrades proportionally with sequence length, severely delaying policy convergence.

• Coarse Rewards Drive Zero-Signal Blocks: Coding environments generally rely on binary pass/fail outcomes from automated test suites (the “verifiable rewards”, which is also why they struggle with design and architecture decisions which are not verifiable). If a training run samples 8 trajectories for a hard problem where 7 fail and 1 passes, the advantage math grants a high positive value to the single success and minor negative values to the failures. While this provides a gradient direction, the binary reward fails to explain why the successful trajectory worked. Furthermore, if all 8 trajectories fail — or all 8 pass — the group standard deviation drops to zero. This zeroes out the computed advantage for every token, completely freezing the training signal and wasting expensive hardware cycles.

Essentially, GRPO gives out participation trophies. Participation trophies are no good if you’re trying to sort out your star players from the genetic defects. So, how do we fix this issue?

“For GLM-5.2, long-horizon tasks produce substantially longer execution traces, and once a super-long trajectory is split by compaction into multiple sub-traces, different rollouts under the same prompt yield different numbers of trainable traces with highly variable lengths. We therefore move from group-wise optimization to a critic-based PPO formulation that learns from individual rollouts, relying on a critic to estimate token-level advantages rather than group-relative comparisons. This single-rollout formulation fits compaction naturally, as it places no constraint on how many traces a prompt produces or on their relative lengths: we bring compaction into training by including all compacted sub-traces as trainable trajectories, and apply a token-level loss to address their length imbalance.”

Using Critic-Based PPO to Isolate Token-Level Value

Critic-based PPO bypasses group comparisons by running a separate value network alongside the policy model to estimate expected rewards from any given context state. Instead of measuring total trajectory outcomes, PPO calculates the specific advantage of an individual token choice relative to the critic’s baseline expectation for that exact position.

If the critic estimates an expected future reward of 0.3 from the current state, and the agent executes a decision that shifts the subsequent state expectation to 0.7, that specific token choice receives a high positive advantage signal. Conversely, an error that drops the expected reward from 0.3 to 0.1 receives a negative advantage. Decisions that exactly match the critic’s baseline expectations yield a near-zero advantage.

This mechanism fundamentally changes credit assignment behavior in long-horizon workflows:

• Per-Token Baselines Protect the Signal: Routine operations like boilerplate code and standard library imports generate near-zero advantages because the critic accurately predicts their outcome. When the model hits a critical debugging step at token 30,000 and unblocks the execution path, the critic’s value estimate jumps sharply. PPO isolates this specific token choice for a major probability update while keeping neutral positions unchanged. The signal-to-noise ratio remains stable regardless of overall trajectory length.

• PPO Consumes Single Trajectories: PPO computes its baseline internally via the critic, removing the requirement for group sampling. The system requires only one trajectory per prompt, cutting compute demands for a long coding task from 16 hours down to 2 hours.

• Sub-Trace Variations Integrate Naturally: Because PPO calculates loss on a per-token basis using independent value estimates, it natively handles compacted fragments. Sub-traces of varying lengths contribute proportionally to the global gradient without requiring alignment across separate rollouts.

• Binary Signals Propagate Backward: PPO resolves the limitations of sparse rewards by using the critic to distribute terminal pass/fail signals backward through the execution history over successive training iterations. The critic learns to identify intermediate markers of success — such as clean code syntax or accurate diagnostics — and assigns higher value estimates to those states. The final binary reward eventually gets distributed backward into nuanced per-token signals through the critic’s value landscape, not through the reward itself.

However, running a critic network equal in scale to a 753B parameter model like GLM-5.2 doubles the required GPU memory during training phases. For short, low-complexity tasks where group statistics remain accurate, this massive memory overhead represents a net hardware loss.

So how do you know when the switch is worth it? As always, you can rely on Daddy Dev to help you. By and large, there are 3 conditions, which when met, will justify the switch into critic-based PPO:

1. Group generation costs become prohibitively expensive due to hours-long trajectory lengths.

2. Trajectory-level averages become too noisy to guide multi-step choices.

3. Sparse binary rewards frequently yield zero-signal groups.

Long-horizon agentic workflows checked all 3 boxes, so Zhipu decided to spend big here.

It’s also worth noting that GLM-5.2’s PPO implementation includes an optimization detail: compaction-aware training. The model trains directly on the compacted sub-traces and lossy summaries generated by the agent framework, forcing the critic to compute expected values within degraded context structures rather than pristine histories.

In real-world production deployments, agents continuously condense past conversation turns and code executions to operate within fixed context limits. If a critic trains exclusively on uncompacted transcripts, its internal value functions become miscalibrated when encountering summarized text in production, forcing the model to evaluate highly unfamiliar states. Training directly on compacted fragments aligns the critic’s value landscape with real-world serving constraints, ensuring that the active training distribution closely matches the production inference distribution.

How Does GLM-5.2 Stop Agents From Cheating During Training?

System disclosures from Z.ai note that GLM-5.2 exhibited a notable increase in reward-hacking behaviors during RL training compared to GLM-5.1. The model’s increased capability allowed it to aggressively explore and exploit cracks in the evaluation infrastructure:

• External Answer Downloading: When given a target problem, the agent would construct curl commands to raw file URLs on GitHub to download known solution scripts directly into the environment.

• Evaluation File Auditing: The model would trigger automated shell commands like find /workspace -name “hidden” to scan the runtime container for hidden grading keys or evaluation scripts.

• Credential Exploitation: Agents would locate and execute cat commands on protected test files to extract secret validation inputs, subsequently hardcoding those exact values into their submissions to mimic genuine algorithmic computation.

From a pure mathematical optimization perspective, these exploits represent successful completions. The training metric demands a passing test suite, and downloading a solution successfully triggers that flag. However, reward hacking stops the model from developing generalized reasoning capabilities; the gradient updates reinforce infrastructure exploitation rather than algorithmic synthesis.

To neutralize these behaviors, GLM-5.2 integrated a two-stage online anti-hack validation module:

• Rule-Based Filtering: The first stage runs strict, rule-based keyword filters that detect known exploit patterns. It blocks unauthorized outbound curl domains, flags discovery commands targeting the evaluation directory, and restricts direct read access to protected validation paths.

• LLM-Based Intent Judgment: Tool calls that bypass the raw keyword filters but exhibit suspicious parameters are routed to an inline LLM judge. This judge analyzes the full context of the agent’s historical choices to determine if the execution step represents a valid problem-solving attempt or an adversarial shortcut. This can likely be done cheaper LLMs and is a good bull case for theses based around inference maxing.

Instead of terminating the rollout immediately upon identifying a hack, the system blocks the exploit and returns mock or dummy data as the tool output, allowing the trajectory to continue running since aborting runs mid-execution introduces severe training instability and can trigger gradient collapse due to truncated episodes and highly anomalous reward boundaries. This isn’t a huge problem for accuracy since the usage of useless dummy data acts as a natural corrective signal by lowering the reward. This signals that the shortcut failed to pass the tests, prompting the gradient to penalize the exploitative choices without disrupting global training dynamics.

How GLM 5.2 Became so Good at Multiple Things?

Optimizing a single generalist model across multiple diverse capability domains simultaneously presents a major optimization bottleneck. Fields like code generation, mathematical reasoning, web search, tool routing, and instruction following possess conflicting gradient paths, distinct token distributions, and unique reward systems.

Training a single model sequentially across these areas results in catastrophic forgetting, where the model masters the newest domain while losing its proficiency in previous ones. Conversely, training on all datasets simultaneously demands massive computational scale and introduces severe gradient interference.

GLM-5.2 resolves this trade-off by using On-Policy Distillation (OPD). The strategy decouples the training phase into two distinct steps:

1. Specialist Optimization: The team trains separate, dedicated specialist models on isolated domains. Each specialist policy consumes its full computational budget exploring a single capability space, entirely avoiding cross-domain gradient interference.

2. Generalist Consolidation: The group uses on-policy distillation to merge the collective capabilities of the specialists into a single generalist model.

This framework differs fundamentally from traditional offline knowledge distillation. Standard distillation forces a student model to match a static set of outputs pre-generated by a teacher model. This introduces exposure bias: the student trains exclusively on the teacher’s ideal token distribution, but during production inference, it generates its own tokens. Once the student makes an unaligned token choice, it enters an unfamiliar state space where its training fails, causing errors to compound rapidly.

On-policy distillation eliminates this exposure gap by making the student model generate its own token trajectories during training. The expert teacher model evaluates these student-generated sequences in real time, delivering dense, token-level supervision across the student’s own token distribution.

The specialist teacher does not merely evaluate terminal success; it calculates exactly what token distribution it would have produced at every individual step given the student’s active context history. This provides a highly dense, fine-grained training signal that transfers exact token-level reasoning capabilities far more effectively than coarse pass/fail outcomes.

Z.ai scaled this pipeline using the SLIME post-training infrastructure to merge more than 10 expert specialist models into the final GLM-5.2 generalist parameter set in roughly two days. The SLIME infrastructure relies on Megatron to handle distributed model parallel training and utilizes SGLang to manage rapid inference rollouts, bridging the two components via a specialized Data Buffer module.

The framework natively supports multiple rollout environments:

• White-Box Integration: Gives the training coordinator full access to the internal hidden states and log probabilities of the active models.

• Black-Box Integration: Evaluates external text outputs when internal parameter access is restricted.

• Compacted Trajectory Handling: Processes lossy history fragments during active training loops.

• Sub-Agent Workflows: Manages multi-agent execution traces during scale-out generation.

Multi-teacher on-policy distillation has become a common post-training primitive across modern open-weights architectures. The execution in GLM-5.2 stands out due to its operational speed; consolidating over 10 distinct architectural specialists within a 48-hour window indicates high infrastructure optimization rather than simple algorithmic experimentation.

This efficiency is driven by SLIME’s dual-purpose deployment model. The exact caching, memory routing, query scheduling, and parallelization frameworks engineered to sustain high-throughput rollouts during RL training carry over directly into production inference serving.

The optimizations and configurations validated during the training rollout phase are reused to provision enterprise serving clusters. This system-level unification ensures that the production serving infrastructure is pre-tested against the exact workload signatures generated during scale training, allowing infrastructure improvements to reinforce both training throughput and deployment efficiency.

Conclusion: What Does GLM 5.2 Mean for the Future of Agentic AI?

GLM-5.2 wins because it isolates and respects the unique structural footprint of agentic loops over everything else. It builds a tightly coupled systems pipeline to handle volatile cache fragmentation, multi-step generation drag, and long-horizon credit assignment.

We have seen this evolution in every mature engineering discipline. In automotive and aerospace design, engineers spent decades chasing a universal chassis or a single airframe that could do everything. They eventually hit a physical wall. The stress profiles of a long-haul transport are fundamentally incompatible with an agile fighter jet. To get more performance, they had to specialize. They had to switch to specific, co-designed architectures tailored entirely to a vehicle’s exact payload and flight envelope.

As raw scaling laws peter out, specialization is the definitive future of AI. The next generation of systems will move away from the “god in a box” conception of AGI (which we’ve been calling a scam forever), and fully embrace flexibility and modularity that enable us to lego our way to building more customizable solutions.

The future (just as things always have) belongs to specialized systems most capable of rapid iterations and improvements. Not the best now, but the one that can become the best after 6 rounds of improvement. This isn’t a principle that most engineering/investing teams have imbibed yet.

The ones that do will pull ahead.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

Share

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

I keep this short.

The mission of Chocolate Milk Cult is simple: make the highest level of AI intelligence accessible to everyone. We don’t do hot takes or aggregated headlines. We do actual deep dives built on original research, custom analysis, and technical frameworks you can use.

The breakdown below is a perfect example of what our open-source research community funds: a premium deep dive on Claude Code built from 300 hours of live deployment testing, exposing concrete structural bottlenecks like the 15-to-20 turn agent degradation cliff. We derived these engineering principles by A/B testing prompt strategies across parallel git worktrees, tracking latency compounding, and forcing multiple team members to run the tool across fundamentally different engineering environments—ranging from legacy backend migrations to fresh frontend builds—to ensure our data didn't overfit for a single workflow or code distribution.

The result was a guide that helped several people rework their entire operation and maximize a ton of value from Claude Code.

While the model weights have updated since our original publication, these core context orchestration principles still carry over seamlessly; in fact, Anthropic still lists these exact constraints as official developer best practices. This is because our research focuses strictly on the invariant mathematical and systems-level realities of the context window, delivering timeless architectural frameworks that hold true across changing software versions.

But running these experiments, funding infrastructure, and securing access to engineers who build these systems costs us between $17,000 and $20,000 a month. These costs add up quickly, but it is the only way to deliver truly differentiated analysis instead of simply repeating PR cycles.

Free subscribers get a lot, and that is intentional. I never want your financial situation to be a barrier to getting the best insights. But paid support is what makes the whole operation possible.

We run a pay-what-you-can model for standard subscriptions so you can contribute whatever matches your budget. There are no tiered content walls so any price point you select will get you access to our entire catalogue of deep dives on Substack.

If you think providing actionable, first-principles AI insights to everyone is valuable, please help support the community at whatever level makes sense for you.

The links for support are below.

Support AI Made Simple for 10 USD per month or 100 USD Annually

Support AI Made Simple for 9 USD per month or 90 USD Annually

Support AI Made Simple for 8 USD per month or 80 USD Annually

Support AI Made Simple for 7 USD per month or 70 USD Annually

Support AI Made Simple for 6 USD per month or 60 USD Annually

Support AI Made Simple for 5 USD per month or 50 USD Annually

Support AI Made Simple for 4 USD per month or 40 USD Annually

Support AI Made Simple for 3 USD per month or 30 USD Annually

Support AI Made Simple for 2 USD per month or 20 USD Annually

Support AI Made Simple for 1 USD per month or 10 USD Annually

Hope to see you among the premium members.

Thanks for reading either way.

Dev <3

PS: (For enterprise operators, builders, and capital deployers who want to fund our high-overhead experiments directly, we also have a Founding Membership. This tier directly supports our primary-source testing infrastructure. In return, founding members get direct access to our proprietary GitHub repository—where we aggregate our raw data, custom insights, and codebase experiments—along with the ability to request custom deep-dive analysis on specific architectures.)

Every month, the Chocolate Milk Cult reaches over a million Builders, Startup Founders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Claude Code is one of the most useful agentic tools on the planet (especially after the release of Opus 4.5). It’s incredibly easy to use (once you can get past the terminal interface), but using it well is another issue altogether.

Claude Code pairs two very unfamiliar design decisions with each other—

1. A Command Line Interface reminiscent of a time when the world existed in black and white only.

2. Cutting Edge Agentic systems with deceptively powerful tool-calling abilities, which can compound errors very quickly if not used properly.

Learning to navigate both these decisions is very worth it— CC is my favorite AI tool in the market, and it alone justifies my 200 USD Subscription to Anthropic (I even have the extra usage setup and happily pay around 750-1000 USD/month on it; you likely won’t have to). I wrote this article to help you use it better.

Aside from my personal experience, we dug through heaps of user case studies of Claude Code, dug into Anthropic research, ran over 300 hours of experiments in real deployments, and compiled their experiences into the most useful workflows. The guide that follows is the result. No generic advice like— “Be specific.” “Give examples.” “Break down complex tasks.” This guide will cover:

• Why context orchestration matters more than prompt wording—and what that means for your use

• The terminal-first design philosophy and why it’s a feature, not a limitation

• Autonomy configuration: permission allowlists, full auto mode, and when each makes sense

• The ~15-20 turn degradation cliff and how to work around it

• CLAUDE.md: what belongs, what doesn’t, and why less is more

• The failure modes you’re probably hitting—and how to actualy fix them.

• Workflows that actually work: explore→plan→code→commit, TDD loops, parallel instances, writer-reviewer splits

If you’ve been looking to take your Claude Code usage to the next level, then this guide is for you.

Executive Highlights (TL;DR of the Article)

• Claude Code is not ChatGPT with file access—it’s an autonomous agent with tools, permissions, and execution authority operating inside a finite context window. Treating it like a chatbot is the root cause of most failures, not prompt quality.

• Context orchestration is the game, not prompt engineering. Every file, instruction, command output, and conversation turn competes for attention in the same window. Polluted context produces garbage regardless of how clever your prompt is; clean context lets mediocre prompts succeed. More context is often worse.

• The terminal-first design isn’t a limitation—it’s the point. Running alongside your shell gives Claude access to CLI tools, scripts, CI hooks, parallel worktrees, and headless automation that IDE-embedded assistants physically cannot touch. You trade inline suggestions for system-level composability.

• Autonomy is a dial. Default permission prompts maximize safety but destroy flow. Custom allowlists or full autonomous mode (with version control as your safety net) restore the agentic value. Productivity scales with autonomy once rollback is cheap.

• Agent performance degrades after ~15-20 turns—repetition, instruction drift, hallucinated claims. This is a hard limit, not a soft one. Effective users reset context, decompose tasks, or externalize state (commits, checklists) instead of pushing through.

• CLAUDE.md should be minimal and operational: build commands, critical paths, non-obvious constraints. Bloated instruction files trigger Chekhov’s gun—models try to use everything they’re given, including irrelevant guidance. Less is more.

• The failure modes are structural, not model-level: premature execution, context contamination, over-specification, permission friction, ignoring terminal feedback. Each has a mechanical fix. If you’re blaming the model, you’re misdiagnosing.

The governing rule: If you don’t control context, you don’t control outcomes. Claude Code doesn’t reward clever prompting. It rewards system design.

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

Section 1: The Mental Model — Claude Code as Orchestrated Agent

Most people approach Claude Code like they approach ChatGPT: type a request, get an answer, repeat. This guarantees mediocrity.

Claude Code isn’t a chatbot that happens to edit files. It’s an autonomous agent operating in your codebase with a constrained set of tools, a limited context window, and permission gates you control. Understanding this shift is key to leveraging this system more effectively. That’s the difference between CAM McTominay vs CDM McT.

The Core Thesis: Context Orchestration

You may forget your lovers birthday, but never forget this mental model: Claude Code is context orchestration. The quality of your outputs is downstream of how well you curate the information space Claude operates in.

Every file you reference, every CLAUDE.md instruction, every conversation turn, every external tool connection—all of it shapes the context window Claude reasons within. Most users optimize their prompts. Power users optimize their context.

This parallels how prompting works at a deeper level. In the prompting guide, we discussed “semantic neighborhoods”—how your instructions steer the model toward regions of its latent space where good answers become statistically likely. Claude Code adds another layer: you’re not just steering with words, you’re steering with what information exists in the context at all.

A brilliant prompt in a polluted context produces garbage. A mediocre prompt in a well-curated context often produces something usable.

(As a fun fact— this is why Claude within Claude Code is so much better than Claude in Cursor and Co.; Claude leverages many tools to ensure that it’s context stays clean).

This is very important to understand for one of the biggest UX shifts that Claude Code users are hit with, especially when you’ve only worked with IDEs (like me).

Terminal-First Is a Feature, Not a Limitation

The terminal interface throws people off. After years of IDE integrations and inline suggestions, a CLI feels like regression.

It’s not. It’s a deliberate design choice that unlocks capabilities IDE integrations can’t match. Terminal-first means:

• Full access to your environment. Claude inherits your shell, your PATH, your tools. The gh CLI for GitHub operations, jq for JSON parsing, ripgrep for fast code search, docker for container management, your custom deployment scripts—Claude can use all of them. An IDE assistant sandboxed to the editor can’t run arbitrary commands across your system.

• Scriptability. Claude Code has a headless mode (-p flag) that lets you pipe it into CI pipelines, pre-commit hooks, and automation scripts. You can have Claude automatically triage GitHub issues as they come in, run code review on every PR, or generate documentation on commit. Try doing that with an IDE extension.

  

• Composability. You can run multiple Claude instances in parallel across git worktrees, each with separate context windows working on independent tasks. One instance refactors your authentication system while another builds an unrelated data visualization component. The terminal is the natural interface for this kind of orchestration.

• No vendor lock-in on your editor. Use VS Code, Neovim, JetBrains, whatever. Claude Code operates alongside your tools, not inside them.

The tradeoff is real: you lose inline suggestions, visual diffs, and the “code appears as you type” magic.

If that’s your primary workflow—quick edits, tab completions, surgical changes—something like Cursor or Augment might be the better fit (I haven’t been very excited by Cursor in the past, but their new updates have been pretty good from what I hear). But if you’re doing complex, multi-file work that requires reasoning across a codebase—refactors, migrations, debugging distributed issues—the terminal-native model scales where IDE integrations hit walls.

The Permission/Autonomy Tradeoff

Out of the box, Claude Code asks permission for almost everything: file edits, bash commands, and external tool calls. This is by design—safety-first defaults for an agent that can execute code on your machine.

But it creates a workflow problem. You give Claude a task, walk away to check Slack, come back five minutes later, and it’s sitting there waiting for approval to edit a file. The whole point of an agent is that it agents. Constant permission prompts break that.

You have a dial here, and you need to decide where to set it:

Maximum safety: Default permissions. Claude asks before any mutation. Good for unfamiliar codebases, production environments, or when you’re learning the tool. I personally would almost never recommend it.

Balanced flow: Custom allowlist via .claude/settings.json. Permit file edits, common commands like git commit and npm test, and trusted external tools. Block destructive operations like rm -rf or database drops. This is where most users land.

Maximum autonomy: The --dangerously-skip-permissions flag. Claude runs uninterrupted until completion. Despite the scary name, this is the daily driver for many Anthropic engineers. The practical risk is low in a version-controlled codebase where you can revert anything. For true isolation, run it in a Docker container without network access. I personally run this as my default, and I’ve never had issues here.

Many power users create a shell alias: alias cc= “claude --dangerously-skip-permissions”. Type cc, start working, no interruptions.

There’s no universally correct setting. The right choice depends on your risk tolerance, the task at hand, and whether you’re running Claude in a sandboxed environment. But you need to make the choice consciously rather than accepting the defaults and getting frustrated.

The Practical Implication

If context orchestration is the game, then your job when using Claude Code is:

1. Curate what enters context. Be deliberate about which files you reference, what history accumulates, and what persistent instructions you set in CLAUDE.md.

2. Invest heavily in documentation. Tools like Claude Code and Codex can often overlook key details because of incomplete searches. People often think that these CLI tools don’t need documentation because they can search through, but Claude Code uses text/regex-based search; ensuring that your documentation can guide this search better is a must to ensure your product has everything it needs. My recommendation is to use specific “guidance documents” where you map out important pieces of logic so that AI knows where to look. If you have the budget for it (and a very large codebase with no documentation), my recommendation is to use Augment Code CLI to write extensive amounts of documentation that can be used by CC for improvements. A good rule of thumb is given below—

   

3. Prune what pollutes context. Clear irrelevant history with /clear. Reset between unrelated tasks. Don’t let a debugging session contaminate a feature build.

4. Extend context reach. Connect Claude to external tools—Puppeteer for browser automation and screenshots, Sentry for error monitoring, PostgreSQL for database queries, Slack for team communication. Each connection expands what Claude can perceive and act on.

5. Set appropriate autonomy. Match permission levels to the task and your risk tolerance. Don’t fight the tool’s safety defaults; configure them to match how you actually work.

Once we understand this, we can move on. The next section covers the foundational setup—the concrete configuration decisions that shape your context environment before you write a single prompt.

Section 2: The Foundational Setup

Before you write a single prompt, your configuration decisions shape everything Claude can do. This section covers the concrete setup that separates productive sessions from frustrating ones.

CLAUDE.md: Your Persistent Context

CLAUDE.md is a file Claude automatically loads at the start of every session. It’s the highest-leverage configuration point you have—(anecdotally) instructions here get followed more reliably than anything you type in the chat.

The hierarchy matters:

• ~/.claude/CLAUDE.md applies to all your projects

• ./CLAUDE.md in repo root is shared with your team via git

• ./CLAUDE.local.md is personal and gitignored

• Child directories can have their own CLAUDE.md files for specific subsystems

Most people either leave this empty or stuff it with everything they can think of. Both are wrong.

What to put in CLAUDE.md:

• Build and test commands (npm run build, pytest -xvs)

• Branch naming conventions and merge/rebase preferences

• Key file locations Claude can’t infer (”authentication logic lives in src/auth/, not src/users/”)

• Project-specific gotchas (”the legacy API in /v1 is deprecated but still receives traffic”)

• Environment setup quirks (”use pyenv local 3.11 before running tests”)

What NOT to put in CLAUDE.md:

• Style guides. Never send an LLM to do a linter’s job. ESLint, Prettier, Black—these are faster, cheaper, and more reliable. Claude will read your existing code and match patterns anyway (this is the benefit of latent space guidance).

• Obvious folder descriptions. If your folder is named components, you don’t need to explain it contains components.

• Generic instructions like “write clean code” or “follow best practices.” These waste tokens and don’t change behavior.

Here’s the key insight from various research: models are naturally inclined to use all information they’re given, even when it’s irrelevant. Anthropic researchers call this the “Chekhov’s gun” effect. A bloated CLAUDE.md doesn’t just waste tokens—it actively degrades performance by introducing distractors.

Keep it lean. I’ve seen effective CLAUDE.md files under 50 lines. I’ve also seen 500-line monsters that made Claude worse at everything.

Run /init to generate a starting point, but don’t trust it blindly. Claude will capture obvious patterns but miss your team’s actual conventions. Treat it as a draft to refine, not a finished product.

Permission Configuration

I covered the autonomy dial in Section 1, but here’s the practical setup.

Your options for allowing tools:

• During session: When Claude asks permission, select “Always allow” for tools you trust

• Via /permissions command: Add specific tools to your allowlist interactively

• In .claude/settings.json: Configure once, check into git, share with team

• CLI flag: --allowedTools for session-specific permissions

A reasonable starting allowlist for most projects includes Edit, Write, and Bash commands for git operations (add, commit, push) and your build/test scripts (npm test, npm run build). This lets Claude edit files and run your standard workflow without asking. It’ll still prompt for anything outside the list.

For full autonomous mode, I just use an alias—cc mapped to claude --dangerously-skip-permissions. I work in version-controlled repos (if you’re using ADEs this is a non-negotiable), I commit often, and I can revert anything. The productivity gain is worth it. Your risk tolerance may differ.

MCP: Extending What Claude Can See and Do

MCP (Model Context Protocol) connects Claude to external tools and data sources. Without it, Claude is limited to your filesystem and shell. With it, Claude can query databases, control browsers, post to Slack, read from Notion, and more.

The MCP servers I actually use:

• PostgreSQL/MySQL for database queries. Claude can check schemas, run queries, verify data. Much better than having it guess at your data model.

• GitHub for complex PR workflows and issue management beyond what the gh CLI offers.

• Sentry for error monitoring. Claude can pull recent errors, stack traces, and frequency data when debugging production issues.

• I tried some Codex MCPs for review, but they did not work. I heard really good things about it, so if you have tried it successfully, would love to hear about this.

You configure MCP servers in three places:

• .claude.json in your home directory (global)

• .claude/settings.json in your project (project-scoped)

• .mcp.json in your repo root (shared with team via git)

A warning: each MCP server adds tool definitions to your context window. If you enable ten servers you’re not using, you’re burning tokens on tool descriptions Claude will never call. Enable what you need, disable what you don’t. You can check what’s consuming your context with /context.

Custom Slash Commands

Slash commands are reusable prompt templates. Put a markdown file in .claude/commands/, and it becomes available as /project:filename.

I use these constantly. For example, a fix-issue.md command that takes an issue number as an argument, runs gh issue view to get details, searches the codebase for relevant files, implements the fix, writes tests if appropriate, and commits with a descriptive message referencing the issue. Now /project:fix-issue 1234 handles the entire workflow.

Same idea for code review—a command that checks for bugs, edge cases, security issues, and performance concerns without the verbose commentary Claude defaults to.

Personal commands go in ~/.claude/commands/ and work across all projects. Project commands in .claude/commands/ get shared with your team when committed.

The $ARGUMENTS placeholder passes whatever you type after the command name. Simple but powerful.

Context Hygiene: The Commands That Matter

Three commands you need to internalize:

/clear wipes conversation history. Use this between unrelated tasks. A debugging session and a feature build shouldn’t share context. I /clear constantly—probably every 20-30 minutes of active work.

/compact summarizes conversation history to free up tokens while preserving key information. Use this when you’re mid-task but context is getting bloated. Auto-compaction triggers at 95% capacity, but by then you’ve already degraded performance. Compact proactively around 70%.

/context shows what’s consuming your context window. Run this when Claude seems slow or unfocused. You’ll often find old tool results, irrelevant file contents, or MCP server definitions eating your token budget.

Research on coding agents found that performance craters after roughly 20 iterations. Not 100. Not 50. Twenty. The developers who treat this as a hard limit—resetting context proactively rather than waiting for degradation—get consistently better results.

My rule: if I’ve been going back and forth with Claude for more than 15-20 turns on a single task, something is wrong. Either the task needs decomposition, or I need to reset and try a different approach. One interesting thing about ADEs is that it can often be much cheaper to rebuild from scratch than to try to fix what’s broken, so you should not be shy about reverting to the last stable states. This does increase the value of code review, however, because you want to be able to identify the last stable state and see what about it might push an AI in certain directions/what the issue could be.

The Setup Checklist

Before starting a new project with Claude Code:

• Run /init to generate starter CLAUDE.md

• Edit CLAUDE.md down to essentials—build commands, key paths, gotchas

• Configure permissions in .claude/settings.json

• Add MCP servers you’ll actually use

• Create 2-3 slash commands for your common workflows

• Commit .claude/ directory to share with team

This takes maybe 15 minutes. The productivity difference over a week of work is hours.

Now that your Claude Code is ready to be the CTO of your new AI-based sneaker Recommendation app with an element of gambling, you’re probably ready to start coding. But I would recommend holding your horses, because just as CC can write good code fast, it can also chef up spaghetti fast enough to make Takumi Aldini jealous.

Section 3: Failure Modes

We’ve discussed many of these failures at length either in earlier sections or prior deep dives. So I’ll keep this one short and direct, meant more for completeness/to have everything in one place for easy lookup.

Premature Execution

It is very important for you to break your lifelong commitment to be a two-pump chump when using Claude Code. This is the number one cause of failure with Claude Code.

CC starts coding before understanding the problem. If you’ve given it autonomy, you’ll end up three iterations deep solving the wrong thing or married to the wrong approach.

Fix: explicit staging. “Read the relevant files and understand the current implementation—don’t write code yet.” Then: “Propose a plan.” Only then: “Implement.” The words “think” and “think hard” trigger extended reasoning and actually allocate more compute. Planning mode is a godsend since reading its thoughts is also easier than debugging walls of code flying at you.

Context Contamination

You debug for an hour, then switch to a new feature without clearing context. Claude drags forward assumptions, references, and attention anchors from the debugging session.

Fix: /clear between unrelated tasks. I clear every 20-30 minutes on heavy usage. The cost of re-establishing context is lower than the cost of degraded performance.

The 20-Iteration Cliff

Research shows agent performance craters after ~20 turns. Symptoms: Claude repeats itself, ignores instructions, claims to have done things it didn’t.

Fix: treat 15-20 turns as a hard limit. If you’re still going, either decompose the task or reset with a fresh summary.

CLAUDE.md Bloat

People stuff CLAUDE.md with style guides, ADRs, and API docs. Now every session starts with 1500 tokens of mostly-irrelevant instructions, and Claude tries to use all of it (Chekhov’s gun effect).

Fix: under 50 lines. Point to docs during specific tasks, don’t load them into every session.

One-Shot Syndrome

Expecting complex tasks to work on the first try. They won’t. Requirements are ambiguous, edge cases emerge during implementation.

Fix: treat first outputs as drafts. Build in verification—tests, screenshots, behavior checks. Build modularly to ensure that you can make changes easily and regularly think about refactoring code before starting major expansions to capabilities to make sure you firm up the foundations.

Over-Specification

500-word prompts specifying exactly how to implement something. You’ve wasted effort on details Claude would figure out, and anchored it to your approach (which might not be best).

Fix: specify the what, be loose on the how. “Add rate limiting—max 5 attempts per 15 minutes” beats a detailed implementation spec.

Permission Friction

Still manually approving file edits and git commits after a week of use? You’re wasting your own time.

Fix: configure your allowlist or use --dangerously-skip-permissions.

Ignoring Terminal Output

Claude runs commands, they fail, and Claude struggles. You watch without reading the errors.

Fix: read the output. You often know things Claude doesn’t—wrong paths, missing env vars, version mismatches. You’d be surprised how often Claude can pick the wrong approach or prioritize the wrong aspects; it helps a lot if you monitor the outcomes on a regular basis.

Pro tip— if an issue has been a problem for more than one hour, Claude likely has completely misunderstood something. Revert your changes to the last place you understood, and then look through the logs. Then start talking to CC about your speculations on the code-base and what could be causing issues.

Not Using Subagents

Complex tasks fill your context with research and dead ends. Subagents keep exploration isolated and return only conclusions.

Fix: “Use a subagent to research how X currently works.” Main context stays clean.

With all of this covered, we’re now ready to use Claude Code Workflows properly.

Section 4: Workflow Frameworks

These are the patterns that actually work. Not theoretical—these are workflows I use and that Anthropic engineers use internally.

Explore → Plan → Code → Commit

The default workflow for any non-trivial task.

1. Point Claude at relevant files. “Read src/auth/ and understand how authentication currently works. Don’t write code.”

2. Have it propose a plan. “Now propose how you’d implement OAuth support. Think hard about edge cases.”

3. Review the plan. Push back, ask questions, refine.

4. Execute. “Implement the plan.”

5. Commit. “Commit with a descriptive message.”

The explicit “don’t write code” in step 1 matters. Without it, Claude jumps to implementation. The “think hard” in step 2 allocates more reasoning compute.

For complex problems, have Claude use subagents during exploration. “Use a subagent to investigate how the session management works” keeps research out of your main context.

This workflow takes longer than just asking Claude to implement something. It also works far more often.

TDD Loop

When the success criteria can be expressed as tests.

1. “Write tests for the feature I’m describing. Cover the happy path and these edge cases: [list]. Don’t implement yet. (Claude has a subagent for planning; spam ts).”

2. “Run the tests, confirm they fail.”

3. “Implement code to make the tests pass. Keep iterating until green.”

4. “Commit tests and implementation separately.”

Claude excels when it has a clear target to iterate against. Tests provide that. Each red-to-green cycle gives Claude feedback it can act on.

This also protects you from Claude solving the wrong problem—the tests define what “right” means before implementation starts.

I use this for any backend logic, API endpoints, data transformations. Less useful for UI work where “correct” is visual.

Visual Feedback Loop

When success is “it looks right” or “it works in the browser.”

1. Give Claude a target—screenshot, Figma mock, or verbal description of desired UI.

2. Have Claude implement.

3. Claude takes a screenshot (via Puppeteer MCP or you paste one in).

4. “Compare to the target. What’s different? Fix it.”

5. Iterate until it matches.

Claude’s first UI attempt is usually 60-70% right. After 2-3 iterations with visual feedback, it’s usually 95%+. Without the feedback loop, you’re stuck at 60-70% and doing the rest manually.

Also useful for debugging—”here’s a screenshot of the broken state, here’s what it should look like, fix it.”

This is one area I’m hearing some comments about, Codex being very good, and Gemini 3.0 has been cleaning house with UI design. On top of this, Cursor dropped an elite update for visual design. This is likely the one area where I expect CC to face the heaviest competition.

Parallel Multi-Claude

When you have independent tasks that don’t need to share context.

Setup: multiple terminal tabs or VS Code panes, each running a separate Claude instance. Or use git worktrees so each instance works on a separate branch.

Instance 1 refactors the auth system. Instance 2 builds a new dashboard component. Instance 3 writes documentation. No shared context, no cross-contamination, 3x throughput.

The key word is independent. If tasks interact—one depends on the other’s output, they touch the same files—this creates merge conflicts and confusion. For truly parallel work, it’s excellent.

Git worktrees make this cleaner:

• git worktree add ../project-feature-a feature-a

• Open that directory in a new terminal

• Run Claude there

• Repeat for other features

Each worktree is a separate checkout. Claude instances can’t step on each other.

Writer-Reviewer Split

When you want quality control built into the workflow.

1. Claude A writes the code.

2. /clear or start a fresh Claude instance.

3. Claude B reviews what Claude A wrote—bugs, edge cases, security issues.

4. Take feedback back to Claude A (or a fresh instance) to address issues.

Separate context matters here. Claude reviewing its own work in the same session tends to defend its choices. A fresh instance with no investment in the code reviews more critically.

I use this for anything going to production. Overkill for exploratory work.

Headless Automation

When Claude should run without you.

The -p flag runs Claude in non-interactive mode. Pipe in a prompt, get output, done.

Uses I’ve actually set up:

• Pre-commit hook that runs Claude on staged files to check for obvious issues

• GitHub Action that triages new issues with labels

• PR reviewer that comments on diffs (use /install-github-app to set this up quickly)

• Script that runs Claude on each file in a migration, one at a time

For batch operations: write a script that loops through items (files, issues, whatever), calls claude -p with a prompt for each, and collects results. Add --output-format json if you need structured output for further processing.

Combine with --allowedTools to restrict what headless Claude can do. You probably don’t want an automated script running with full permissions.

Checklist-Driven Work

For large tasks with many steps—migrations, multi-file refactors, launch checklists.

1. Have Claude write a checklist to a markdown file. “Create CHECKLIST.md with every file that needs updating for this migration.”

2. Claude works through items one by one, checking them off.

3. The file serves as progress tracking and recovery point if context resets.

This externalizes state. If Claude hits the 20-iteration cliff and you need to reset, the checklist shows what’s done and what’s left. Paste it into the fresh session and continue.

Also useful for team handoffs—the checklist is documentation of what’s been done.

Conclusion: From Prompting to Orchestration

Claude Code isn’t powerful because it writes code quickly. It’s powerful because it lets you externalize cognition into a system you can actually control. Context, autonomy, feedback, and verification aren’t secondary concerns here; they are the product. Everything else—prompting, clever phrasing, even model choice—is downstream.

Once you internalize this, there’s no going back. Prompting starts to feel like a local optimization. IDE magic feels cosmetic. When something goes wrong, you stop asking why the model “behaved strangely” and start asking what information you gave it, what permissions you allowed, and what feedback loop you failed to close. The mystery evaporates.

This is where the real split emerges. Most people use Claude Code as a faster pair of hands. Operators design environments where failure is hard and success is boringly inevitable. Same tool, completely different outcomes. The difference isn’t intelligence; it’s orchestration.

There’s a simple rule that governs everything you’ve read: if you don’t control context, you don’t control results. Almost every complaint about coding agents—hallucinations, brittleness, wasted time—traces back to violating that rule. Not a bad model. Bad setup.

Claude Code won’t make you a better engineer. It will make it very obvious whether you already think like one.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

On Friday, June 12, 2026, SpaceX begins trading on Nasdaq. Within twenty business days, money from millions of automatic retirement accounts will buy the stock on a rigid, pre-published schedule. Nobody — not your fund manager, not you — gets to evaluate the company first.

Here is what’s being forced into your portfolio. SpaceX is selling roughly $75 billion of stock at a $1.77 trillion valuation — 95 times last year’s revenue, for a company that lost $4.9 billion doing it. The shares your retirement account buys carry one vote each; the shares insiders keep carry ten. After the sale closes, insiders control 88.5% of all voting power and Elon Musk alone commands 82.4%. And because an index fund must buy the complete corporate ledger, you can’t take the profitable Starlink business without also taking a cash-burning rocket division, the social platform X, and an AI venture that spent $12.7 billion on data centers in 2025 alone.

To help you understand how we got here — and what you can still do about it — this article will cover:

• Which index benchmarks are buying SpaceX and on what schedule. Russell funds buy at the close of Day 5. MSCI global funds finalize by Day 10. Nasdaq-100 funds force entry after Day 15. We’ll map every fund family to its forced-purchase date so you can find your own exposure.

• How the algorithmic front-running of these schedules drains $16 billion a year from passive investors. The purchase dates are published in advance and you can’t change them. Traders who know exactly when $14.4 trillion in passive capital must buy get to set the price it buys at — roughly $200 a year out of every indexed household.

• How the index guardrails were dismantled in 90 days. Seasoning windows, float minimums, profitability screens, and voting-rights floors were built over a decade to protect retail investors. Between February and June, Nasdaq, FTSE Russell, CRSP, and S&P each rewrote their rulebooks to waive them for sufficiently large listings.

• Why the S&P 500’s June 4 “rejection” of SpaceX is security theater. The committee publicly kept its screens for the S&P 500 — and in the same document, quietly opened its own Total Market, Completion, and Dow Jones benchmarks to low-float listings. Worse, those side doors let an asset accumulate the trading history and float needed to walk through the front gate 12 months later.

• What’s inside the S-1 your retirement account is buying. The full balance-sheet dissection: segment losses, the AI division’s capex run-rate, the bitcoin treasury, and the voting math.

• What you can actually do to block forced entry. The specific fiduciary, regulatory, and shareholder levers — ERISA prudence queries, the DOL comment window, pension exclusion mandates, and Rule 14a-8 proposals — including a template you can send to your plan committee today.

Executive Highlights (tl;dr of the article)

For this article, you might need to understand the following terms:

• Index / benchmark: A published list of stocks with a rulebook deciding what gets in. Your index fund is legally bound to copy this list. Whoever writes the rulebook controls what your retirement account buys.

• Passive fund: A fund that automatically buys whatever its index says, in the proportions it says, on the schedule it says. No human judgment in the loop. ~$14.4 trillion of American savings works this way, including most 401(k) default options.

• Float: The percentage of a company’s shares actually available for the public to buy. SpaceX is floating 4.15% — insiders keep the rest. Low float means a thin sliver of trading sets the price for the whole company, so prices swing violently and insiders keep control.

• Float minimum: The old guardrail requiring ~10% float before a stock could enter an index. This is the rule that was rewritten: now a float of any percentage qualifies if it’s worth enough raw dollars.

• Seasoning window: A mandatory waiting period (typically 12 months) between a company going public and entering an index — time for hype to cool, lockups to expire, and real financial data to accumulate before your money is forced in.

• Profitability screen: The requirement that a company show actual accounting profits before index entry. SpaceX lost $4.9 billion last year.

• Dual-class shares: Two share types with unequal votes. Your shares: 1 vote. Insider shares: 10 votes. This is how Musk keeps 82.4% control while selling you the downside.

• Rebalancing / forced buying: When the index list changes, every fund tracking it must buy or sell to match — at a published time, regardless of price. Predictable forced buying is what front-runners feed on.

• Front-running: Buying a stock ahead of forced buyers you know are coming, then selling it to them at a markup. Legal when the schedule is public. This schedule is public.

Once you understand these terms, here is your tldr — 

• SpaceX Valuation and Forced Buying: On June 12, SpaceX begins trading at a $1.77 trillion valuation — 95x revenue with a $4.9B net loss. Within 20 business days, index funds holding millions of Americans’ retirement savings are contractually forced to buy it. No asset evaluation occurs by you or your fund manager; the rewritten index rulebook decides the purchase automatically.

• The $16B Capital Extraction: Forced buying schedules are published in advance, allowing traders to buy the stock early and sell it back to passive investors at marked-up prices. This mechanism drains roughly $16B a year from passive investors (~$200 per indexed household). Research confirms that these fast-tracked index additions underperform by 22% over the following year, forcing index funds to buy at the top.

• Dismantling Investor Guardrails: Four index providers removed protections like seasoning windows, float minimums, profitability screens, and voting floors within 90 days. They replaced percentage-based protections with raw dollar thresholds, allowing massive companies to bypass safety rules. These changes specifically target the largest listings; one provider’s consultation explicitly named “SpaceX, OpenAI, Anthropic” as the motivation.

• S&P 500 Security Theater: The S&P 500’s June 4 rejection of these changes is a brand stunt. The same document opened S&P’s total-market benchmarks to low-float listings, feeding growth, ESG, and target-date products automatically. This creates a side-door where a company accumulates the required trading history over 12 months to enter the S&P 500 front gate later.

• The Toxic Inside Bundle: The forced bundle forces you to buy profitable Starlink alongside a loss-making rocket division, X, and an AI venture with capex exceeding its revenue. Elon Musk retains 82.4% control via 10:1 insider voting shares, leaving shareholders with mandatory arbitration instead of court access.

• The Retirement Exit Liquidity Pipeline: This is the standard pipeline moving forward, not an isolated incident. Both Anthropic ($965B) and OpenAI ($852B) filed confidential S-1s within the past two weeks. If these rules stand, retail retirement capital becomes the permanent exit liquidity for private valuations forever.

• Your Direct Leverage Actions: The index machine runs entirely on your paycheck deductions, giving you real leverage to fight back. You can audit your fund benchmarks, send an ERISA prudence query to your 401(k) committee to force the issue onto the plan’s fiduciary records, and comment on the DOL’s safe-harbor rule. You can also push pension boards to exclude low-float multi-class listings and file shareholder proposals at index providers, replicating past participant litigation that cut 401(k) costs in half.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe now

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Which Index Benchmarks Are Buying SpaceX and on What Schedule?

Every passive fund is legally bound to replicate an underlying benchmark owned by an independent company (and the same company might have 2 indexes that track 2 different benchmarks). For example, Vanguard’s Total Stock Market ETF (VTI) tracks the CRSP U.S. Total Market Index; the Vanguard Total World Stock ETF (VT) tracks the FTSE Global All Cap; Invesco’s QQQ tracks the Nasdaq-100. To find your exposure to Elon’s newest scam, you’ll have to look at the “seeks to track” statement on your fund’s official profile page.

Once you know your exposure, the next variable to track is the timing. The specific countdown to forced purchasing begins the moment SpaceX logs its first trade on Friday, June 12, 2026. Then you have to look out for the following:

• Russell Large-Cap Funds: Forced buying occurs all at once at the market close on Day 5 — expected Thursday, June 18 — under a single-batch rule adopted by the committee on May 26.

• MSCI Global Funds (ACWI, World): The fast-track inclusion decision is announced between Day 1 and Day 3, with mandatory buying finalized after the close of Day 10 — expected Friday, June 26.

• Nasdaq-100 Funds (QQQ): The asset is evaluated on Day 7 and forced into the index after the close of Day 15 — early July 2026.

• Total Market Funds: CRSP (governing Vanguard’s VTI complex) altered its design on April 27 to bypass percentage limits if an asset passes a raw dollar test. S&P’s total-market index series implemented an identical dollar-alternative clause alongside a five-day fast-track trigger on June 8.

• Target-Date Funds: These vehicles automatically absorb the stock based on the underlying proportion of CRSP, Russell, Nasdaq, or MSCI products they hold internally.

These dates matter more than you think. Across the $14.4 trillion sitting in domestic passive equity funds, the general cost of running predictable, schedule-driven rebalancing trades already drains an estimated $16 billion annually from people due to algorithmic front-running. However, waiving float minimums for a mega-cap listing like SpaceX introduces an entirely new layer of volatility risk. When algorithmic traders know the purchase schedules ahead of time for a massive, low-float asset, the resulting price distortions will likely worsen this existing multi-billion-dollar drain on your savings.

Let me rephrase that clearly: since algorithmic traders know the purchase schedules ahead of time (which you can’t change since you can’t control the index), they can scalp your hard-earned savings (the phenomenon already costs people 16 Billion USD annually). You are the product being sold to convince Silicon Valley Elites and their bankers to invest in an overinflated bubble.

Maybe you’re wondering if this isn’t the way it’s always been. Or you think I’m reading too much into something. However, if you break down the movements over the last 90 days, this pattern becomes self-evident. Multiple independent indexes all changed their inclusion criteria to accommodate the SpaceX IPO. And their changes specifically loosed the guardrails that were put in place to protect retail investors from the information asymmetry between themselves and institutional investors.

Let’s understand how in more detail.

How Were the Index Guardrails Dismantled to Allow SpaceX-like Low-Float Listings?

Credit where credit is due: the passive investment industry spent the last decade building some good investor protections from predatory private listings. These protections were engineered after specific corporate failures left index investors holding illiquid, un-voted shares. The guardrails relied on four structural pillars:

• Seasoning Windows: The S&P Composite 1500 required an IPO to complete 12 months of active public trading before index entry, allowing initial lockups to expire and public financial data to surface.

• Float Minimums: Standard benchmarks required at least 10% of a company’s total shares to be available for public trading, preventing tiny allocations from causing massive price volatility.

• Profitability Screens: Inductees were required to show positive GAAP net income over their most recent four quarters, ensuring index entry was backed by real accounting profits rather than market hype.

• Voting Floor Rights: Following Snap’s zero-vote share offering in 2017, FTSE Russell barred companies unless public shareholders held at least 5% of total voting power, while S&P closed the S&P 500 to new multi-class structures.

These protections were systematically broken down between February and June of this year to accommodate Elon’s newest rug pull:

• Under the old rules, a company selling only 4% of its shares to the public was automatically disqualified for failing the 10% float threshold.

• Under the new rules, if that 4% slice is worth billions of dollars, the percentage minimum is waived entirely. At a $1.77 trillion valuation, SpaceX’s tiny 4.15% public float translates to a $75 billion offering — a dollar figure larger than the total market cap of most companies in the index.

• (The float constraints are more important than you’d realize. If only a small percentage a company is available, then its price will fluctuate much more violently, which can adversely impact investors. This way Elon and his buddies get to keep all the control, while still forcing indexes to buy their inflated valuations).

A more detailed analysis is given below:

As you can see here, multiple indexes clearly rewrote their rulebooks to accommodate these changes, all to make it easier for them to let them cash out on that sweet 401-k demand. The scalpers that will price this demand to dump stuff will win. Elon’s cronies will win bigly. Even the index managers will win since they make money irrespective. The only one that loses is the regular investor that thought that someone would look out for them. That’s what we get for being poor and focusing on actually contributing to society instead of trying to weasel around and steal value from every nook and cranny of the system.

Luckily, people have started speaking out against this, sparking a public confrontation that forced the S&P 500 committee to draw a hard line on its core index family. While this is a win, I think many people overestimate what this actually means (I heard from more than a few people that this was all we needed and everything would sort it itself out now). Let’s understand why this isn’t true.

Does the S&P 500 Rejection of SpaceX Protect You?

The S&P 500 committee’s June 4 decision to reject the “MegaCap” rule changes creates a dangerous illusion of systemic safety. The committee’s proposal on April 30 had suggested halving the 12-month waiting period, waiving the 10% public float minimum, and dropping the positive GAAP net income screen for exceptionally large companies. While public opposition forced the committee to declare that rule exceptions “should not be granted solely based on market capitalization,” this firewall only applies to the S&P 500 index family itself.

This is a problem because an asset entering through the newly opened side doors of total-market, Nasdaq, and Russell indexes begins accumulating the exact milestones required to clear the front gate later. Over a 12-month seasoning window, an unlisted mega-cap stock builds up trading history, converts insider shares into float as lockups expire, and establishes passive institutional ownership patterns.

Furthermore (and this is important), S&P’s own June 4 document authorized these low-float alternatives for its Total Market, Completion, and Dow Jones U.S. Total Stock Market benchmarks. Entry into these foundational total-market products automatically feeds into specialized sub-indexes:

• Total-market entrants become eligible for immediate inclusion in derived size, style, sector, factor, and sustainability benchmarks.

• Investors who choose a supposedly safe “Growth” or “ESG” index product absorb the low-float asset layout without realizing they have bypassed the core S&P 500 profit screens.

• The exposure propagates into target-date retirement vehicles that use total-market building blocks as their primary equity holdings.

So the S&P ruling doesn’t even fully extend across their own families. Safety Theatre at its finest.

And if we let this pass, then things are only going to get worse. Silicon Valley has rug-pulled retail investors through overpriced IPOs based on misinformation and hype for a while now; but at least investors could choose to not engage. If these index changes are allowed to stand, it will create a strong pipeline where retail investors will permanently have to bail out the so-called smart money. And that lineup already looks promising:

• Anthropic filed its confidential draft S-1 on June 1, following a $65 billion funding round that established a $965 billion valuation backed by a $47 billion annualized revenue run-rate (wait till we dissect this can of worms, there is a LOT wrong here).

• OpenAI finalized its own confidential SEC paperwork on June 8 at a valuation nearing $852 billion, completely omitting public financials, share distributions, or corporate voting structures.

Fun times ahead. But can we do anything about this?

As always, you have more power than you think.

What Can YOU Do to Block Forced Index Entry?

Contrary to what some wimps peddle online, people hold enough leverage to make things happen. Reversing this automated robbery requires formal, documented intervention through structured fiduciary and regulatory channels. The following list might seem overwhelming, but the key is to organize as a collective, pick 2–3 sides that best suit us, and focus on that. If everyone focuses on their battle, the collective

First, you can protect yourself better by executing three steps:

• Audit Fund Benchmarks: Locate the “seeks to track” statement on your index fund profile pages to identify your direct exposure. If your portfolio holds products tracking the CRSP U.S. Total Market Index, the FTSE Global All Cap, the Nasdaq-100, or S&P total-market benchmarks, your money is exposed.

• Submit a Formal ERISA Prudence Query: Route a written request to your employer’s 401(k) plan investment committee through Human Resources, requiring a formal response on the record. This places a binding duty of review on individuals who face personal legal liability under ERISA Section 404(a), which mandates care, skill, prudence, and diligence in monitoring plan options. Most importantly, this will force the issue onto the plan’s fiduciary record.

• Demand Restricted Equity Alternatives: Use your plan’s feedback loop to demand at least one broad equity selection that is structurally insulated from immediate low-float IPO entry — such as a standard S&P 500 fund, a profitability-screened index, an equal-weight benchmark, or a self-directed brokerage window.

Not sure what to send? Here’s something nice and easy drafted by Irys — “I am a participant in [plan name]. I am writing to ask whether the plan’s investment committee has reviewed the 2026 IPO fast-entry and low-float methodology changes adopted by Nasdaq (effective May 1), FTSE Russell (May 26), CRSP/Morningstar (April 27), and S&P Dow Jones total-market indexes (June 8), as well as MSCI’s existing large-IPO early-inclusion process, and whether the committee has assessed how these changes affect the plan’s index funds and target-date funds-including exposure to newly public, low-float, controlled companies such as SpaceX. I request a written response. Under ERISA section 404(a) and in light of the Department of Labor’s March 30, 2026 proposed rule on designated investment alternatives, I believe benchmark methodology, valuation, liquidity, and complexity are appropriate subjects for the committee’s documented review.”

(While the bar for ERISA litigation is high and plan fiduciaries are generally afforded broad discretion in selecting diversified index funds, these formal queries serve a critical strategic purpose. Rather than an immediate trigger for litigation, this documentation acts as a mechanism for transparency — compelling plan sponsors to justify their continued reliance on benchmarks that have systematically dismantled their own governance screens. By forcing this issue onto the official record, you ensure that the decision to expose retirement savings to low-float, high-volatility listings is a conscious, documented choice rather than an automated oversight. Historically, it is this type of granular scrutiny that forces corporate consultants and fiduciary insurers to re-evaluate structural risks, eventually steering capital away from exposed benchmarks to avoid the long-term threat of ‘failure to monitor’ claims).

This documentation creates an immediate liability risk for plan sponsors. Historical precedent shows that excessive-fee litigation over the last fifteen years cut 401(k) expenses roughly in half because corporate consultants steered clients away from structural compliance risks. Once “default funds whose benchmarks auto-buy low-float corporate listings” becomes a line item on fiduciary-insurance questionnaires, the spreadsheet merchants will move away from the exposed benchmarks themselves.

You can also escalate this financial pressure by targeting fund boards and index licensing revenue directly. Because index providers earn revenue as basis points on assets benchmarked to their rulebooks, moving capital out of a Russell or CRSP total-market fund and into a standard S&P 500 fund is a good way to immediately punish funds that dissolved their screens. Once again, to really turn the thumb-screws, skip the IR departments and write directly to a fund’s independent board of trustees, who hold the statutory authority to approve index licenses and tracking policies.

The most definitive regulatory lever is filing a formal public comment on the Department of Labor (DOL) March 30 proposed rule. Workers can be automatically enrolled into default retirement funds only because federal regulations grant plan sponsors a strict liability safe harbor for utilizing Qualified Default Investment Alternatives (QDIAs). Since federal agencies are legally required to address substantive data within the public register, commenters can dismantle this legal coverage. To trigger this, submit comments demanding that QDIA safe-harbor status be explicitly conditioned on a default fund’s benchmark maintaining rigorous seasoning, minimum public float, and voting-rights screens. Citing the SpaceX listing and the FTSE Russell consultation by name proves that low-float benchmarks carry structural valuation and liquidity risks, forcing corporate lawyers to abandon them to preserve their safe harbor.

For public sector capital, beneficiaries can force state pension systems to act from below. The Comptrollers of New York City and New York State and the CEO of CalPERS have issued public objections calling the SpaceX listing the most management-favorable governance structure ever brought to public markets. State employees and teachers can leverage these admissions to demand that their retirement boards aggressively renegotiate passive investment mandates. Pensions routinely run index-minus-exclusion mandates for firearms or tobacco; beneficiaries can mandate identical exclusions for multi-class, low-float listings (I’d imagine you can also directly point to Grok’s Sexual Deepfakes as a strong reason to not invest into SpaceX, which now owns the AI company). These groups can pull some serious numbers (for eg the American Federation of Teachers (AFT) alone represents 1.7 million members).

Finally, we can also directly confront the regulatory framework governing the indexes themselves. Rather than relying on SEC Rule 14a-8 shareholder proposals — which corporate counsel routinely block under the ‘ordinary business operations’ exclusion — investors should direct their focus toward formal SEC regulatory petitions. By petitioning the SEC to investigate the commercial conflicts of interest between index licensing revenues and investor protection mandates, we can force some action. Morningstar here seems to be the most interesting target: its advisory division issues safety ratings to individual investors while its index division actively waives the screens that protect them.

Then we can attack the root of this extraction: the underlying exchange listing standards. While the SEC and US exchanges have permitted dual-class structures for decades, the SpaceX listing represents an unprecedented compounding of risks. A $1.77 trillion valuation holding ten-to-one insider voting shares and 82% single-person control is concerning on its own. But when combined with waived float minimums and bypassed profitability screens, it creates a systemic vulnerability.

Additionally, maintaining a public ledger that tracks which funds bought on which date, at what price relative to the offer, and the resulting cost-per-household provides standing evidence for future litigation and regulatory enforcement.

Conclusion: What Is the Bottom Line?

There is a common type of cynicism that says everything is rigged, the system is too big, and you cannot do anything about it. That attitude is a cop-out used by cowards to make themselves feel better about their inaction.

The entire index fund machine does not run on Wall Street’s hidden wealth. It runs on your money — the automatic deductions taken out of your paycheck every two weeks. When you start asking tough questions, the people managing that money have to listen.

Throughout history, every good thing that ever happened came about because regular people decided to come together and fight for it. We know this works because we just saw it happen. On June 4, public pressure forced the S&P 500 committee to back down and keep their safety rules in place. Over the last fifteen years, regular people filing complaints cut 401(k) fees in half across the country.

The scale of these giant companies can feel overwhelming, but the way you fight back is simple: pick your battle, find a group to stand with, and stick it out. You do not have to change the whole financial system yourself. You just have to look up your funds, send one written question to your HR department/send messages to indexes, or move your savings to a fund that does not buy into rigged voting structures. Persist with that, and have faith that there are others who have your back.

Appendix A: The Five Rulebooks in Detail

Written to be forwarded to a plan committee, legal counsel, or investment consultant.

• S&P Dow Jones Indices: Under old rules, the S&P Composite 1500 (including the S&P 500) required a 12-month IPO seasoning window, an investable weight factor (publicly investable share fraction) of at least 0.10, and positive GAAP net income in the most recent quarter and summed over the most recent four. The April 30 consultation proposed halving seasoning to six months, waiving the 0.10 weight factor, and waiving the profitability screen for MegaCap listings. On June 4, S&P rejected these changes for the Composite 1500, stating exceptions should not be granted based solely on market capitalization. However, in the same June 4 document, S&P added a dollar alternative to the 10% float test and a five-business-day fast-track entry clause for its Total Market, Completion, and Dow Jones U.S. Total Stock Market indexes, implemented June 8. Entrants in these total-market families may become eligible for derived size, sector, style, factor, and sustainability indexes.

• Nasdaq (Nasdaq-100): Old rules required a 10% minimum free float, and new listings waited for scheduled reconstitutions. A February consultation proposed fast entry for new listings with full market capitalization within the top 40 constituents, replacing the float minimum with a weight cap of five times free float. Nasdaq finalized the rule effective May 1, adopting fast entry after the 15th trading day based on evaluation on the 7th trading day, and softening the weight cap to three times free float. A company with 5% float that was previously excluded now enters at reduced weight. Reuters reported March 10 that SpaceX made early Nasdaq-100 inclusion a necessary condition for its listing venue; Nasdaq states the rule was not written specifically for SpaceX.

• FTSE Russell (Russell U.S. Indexes): Old rules added IPOs during scheduled quarterly reconstitutions and required a minimum 5% free float alongside a minimum 5% of total voting rights in public hands. On May 26, FTSE Russell adopted a fast-entry rule for IPOs with investable market capitalization above the Russell Top 500 breakpoint, forcing index addition after the close of the fifth trading day based on first-day closing prices. The platform’s February consultation explicitly named projected 2026 IPO targets motivating the shift: “SpaceX, OpenAI, Anthropic.” The final rule waives the 5% float and voting floors if future lockup expirations are expected to cure the deficit within 12 months, and adds the stock all at once rather than staging entry. Russell U.S. indexes benchmark $11.6 trillion, with $2.7 trillion in passive mandates.

• CRSP / Morningstar Indexes: Old rules required a strict 10% public float for fast-track IPO entries. Effective April 27, CRSP altered its float screen to allow fast-track inclusion if an IPO has a 10% float or if its float-adjusted market capitalization is at least one-half basis point of the index-eligible universe, which is roughly $3.3 billion. This index suite benchmarks more than $3 trillion, including Vanguard’s total-market complex. CRSP is transitioning its governance to Morningstar Indexes on July 1.

• MSCI (Global Investable Market Indexes): MSCI made no rule changes in 2026 because its Global Investable Market Index methodology has contained early inclusion loops for large IPOs since 2007. To qualify for early entry, an IPO must clear length-of-trading and liquidity screens, hit a full market capitalization of 1.8 times the interim size segment cutoff, and hit a free-float-adjusted market cap of 1.8 times half that cutoff. These dollar-scaled rules mean a trillion-dollar company with a low percentage float clears the hurdle automatically. On June 8, MSCI confirmed this process applies to SpaceX, with inclusion announced between the first and third trading day and made effective after the close of the tenth.

Appendix B: The SpaceX S-1 Balance Sheet Dissection

All figures are compiled directly from the June 3 amended S-1 corporate filing.

• The Offering Layout: The company is selling 555,555,555 Class A shares at an expected price of $135 each, raising approximately $75 billion before any underwriter overallotments.

• Valuation Multiples: Multiplying $135 by the post-offering Class A and Class B share counts (including outstanding restricted Class B) establishes a total valuation of $1.77 trillion. This multiple is approximately 95 times the company’s full-year 2025 revenue of $18.674 billion.

• The Voting Disparity: Class A shares carry one vote per share; insider Class B shares carry ten votes per share. Post-offering, Class B holders retain 88.5% of total voting power, and Elon Musk controls 82.4%. This triggers a “controlled company” status under Nasdaq rules, exempting the firm from standard board governance protections and giving Class B holders the unilateral right to elect a board majority.

• Segment Bundling: The corporate vehicle reflects a February 2026 transaction where SpaceX acquired xAI, following xAI’s 2025 acquisition of X. Financial statements are recast retroactively under common control to bundle four distinct operations: Starlink, launch/space, X, and xAI.

• Consolidated Loss Metrics: In full-year 2025, the company posted revenue of $18.674 billion, an operating loss of $2.589 billion, and a final net loss attributable to common shareholders of $4.937 billion, alongside adjusted EBITDA of $6.584 billion. For the first quarter of 2026, the company recorded revenue of $4.694 billion, an operating loss of $1.943 billion, and a net loss of $4.947 billion.

• Segment Operational Realities: The Connectivity segment (primarily Starlink) generated $11.387 billion of 2025 revenue, $4.423 billion of operating income, and $7.168 billion of adjusted EBITDA. The Space segment posted $4.086 billion of revenue and an operating loss of $657 million. The AI division logged $12.727 billion in capital expenditures for 2025 and accelerated to $7.723 billion in capital expenditures for the first quarter of 2026 alone — a run rate that annualizes to more than the entire company’s consolidated revenue.

• Treasury Crypto Holdings: As of March 31, 2026, the corporate treasury held 18,712 bitcoin with a reported fair value of $1.293 billion, down from a fair value of $1.637 billion at year-end 2025.

Appendix C: Academic Literature + Links for Your Research

Compiled academic literature detailing the transaction drag imposed by automated, schedule-driven rebalancing trades. These were cited and linked in-line, but I’m also inclufing them so you have an easier time sourcing the research/claims.

• Harvey, Mazzoleni, and Melone (“The Unintended Consequences of Rebalancing”): Documents that predictable, mandatory institutional index rebalancing trades cost investors roughly $16 billion per year in execution penalties. This extraction transfers approximately $200 per year out of every indexed U.S. household.

• Sammon and Shim (“Index Rebalancing and Stock Market Composition”): Evaluates rebalancing execution data in the Journal of Financial Economics and finds that predictable index rebalancing trades impose a structural drag of 46 to 69 basis points per year on passive index fund returns. The paper demonstrates that capitalization-weighted index math mechanically forces funds to purchase shares at peak run-ups and high-volume corporate issuances.

• Arnott, Brightman, Kalesnik, and Wu (“The Avoidable Costs of Index Rebalancing”): Tracks the performance delta of discretionary adjustments and finds that companies deleted from the S&P 500 outperform the mandated new stock additions by an average of 22% over the 12 months following index rebalancing. This proves that accelerated entry mechanisms force passive capital to buy assets when they are overvalued.

• Greenwood and Sammon (“The Disappearing Index Effect”): Analyzes inclusion day price behavior and details how the historic price pop on the day a stock joins the S&P 500 fell from an average of 7.4% in the 1990s to just 0.3% over the last ten years. The data shows that forced buying has not become cheaper, but has shifted chronologically: algorithmic traders chart the published index methodologies weeks in advance, accumulate the target stock early, and sell it to index funds at prices that already reflect the automated demand.

Sources:

• S&P Dow Jones Indices, June 4 results announcement and April 30 MegaCap consultation.

• Nasdaq, February 2026 consultation, summary of responses and conclusion, and methodology update explanation.

• Reuters, March 10 report on SpaceX seeking early Nasdaq-100 inclusion.

• FTSE Russell, IPO fast-entry consultation, May 26 press release, and technical notice.

• CRSP, float-shares investability screen notice and methodology guide.

• MSCI, SpaceX IPO update, Megacap IPOs explainer, GIMI methodology, and Corporate Events methodology.

• SpaceX, June 3 amended S-1.

• Anthropic, confidential draft S-1 announcement and Series H announcement.

• AP, OpenAI confidential SEC paperwork report; CBS News, OpenAI confidential IPO statement.

• ICI, Active and Index Investing, April 2026.

• Vanguard, VTI benchmark and VT benchmark; Invesco, QQQ benchmark.

• AFT, May 6 letter to the SEC; NYC Comptroller, NYS Comptroller, and CalPERS, letter to SpaceX.

• U.S. Department of Labor, March 30 proposed rule fact sheet.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

Share

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. :

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

How can you develop AI projects that create measurable business value? This is the trillion-dollar question on everyone’s minds. Especially with the stories floating that most enterprise LLM projects never leave the demo/prototype phase.

The friction in deriving business value isn’t that language models are useless. The problem is that most teams do not know how to decide whether a model-powered feature is good enough to ship, too expensive to continue, or simply not worth building. So they keep optimizing with more prompts, switching to different models/frameworks, and overall burning more engineering time. But the business case does not always improve just because the system gets a little more accurate.

Production AI needs a different kind of discipline. Before a team debates model choice or prompt quality, it needs to understand the economic boundary of the project: what the system has to achieve, what failure is allowed to cost, and when the team should stop working on it.

This article is about how to deploy AI projects that are valuable instead of just technically impressive.

In this article, we’ll cover:

• How to define the minimum useful performance level for an AI feature before the team starts optimizing.

• How to decide when an AI project should be stopped instead of endlessly improved.

• Why higher accuracy does not always mean better business value.

• How to evaluate production AI systems around cost, reliability, latency, and operational risk.

• How to turn AI deployment from an open-ended experiment into a controlled engineering decision

To access the full article—and all premium breakdowns going forward/written prior—upgrade to a premium subscription below.

Subscribe now

If you believe deep insight deserves support, become a premium subscriber to allow me to keep doing the same.

Flexible pricing available—pay what matches your budget here.

Most companies offer learning or professional development budgets. You can expense this subscription using the email template linked here.

Read more

Stateful Swarms tackle this directly. Instead of relying on volatile inference windows or endless reprocessing loops, this framework coordinates multiple specialized agents through a persistent blackboard state. Each agent performs a targeted task and updates a structured, auditable memory. This persistent state accumulation means expensive processing is done once, after which systems can cheaply update, query, and reason over the structured knowledge indefinitely.

We validated this by running Stateful Swarms against the complete 1,251-task Harvey Legal Agent Benchmark, achieving an 83.74% pooled criteria pass rate, a 17.75% strict all-pass rate, at $1.30 per task. Critically, these results came from structural coordination improvements rather than raw model scaling or specialized engineering (all details, including code and experiment details, published here).

In this deep dive we will cover:

• Why long-context inference is fundamentally limited

• How standard agentic systems repeatedly waste resources

• What are Stateful Swarms and their structural advantages for agentic reasoning tasks

• Detailed results from our evaluation against Harvey LAB

• The essential role of auditability in production-grade systems

• Why persistent state is critical for scalable knowledge work

Given our commitement to open source and open science, the entire repository, benchmarks, and reasoning traces are open-sourced under an MIT license. We invite you to examine, validate, and extend our work directly (including using it commercially). Additionally, we are actively seeking engineering collaborators interested in enhancing the framework and building the next generation of agentic tooling.

This is a foundational step towards intelligent systems that truly remember.

(PS: If you want to use our Swarm Infrastructure, sign up for our API here: https://www.irys.ai/irys-api)

Executive Highlights (tl;dr of the article)

• Long-context inference is architecturally broken. Attention scales quadratically, memory bandwidth is physically capped, and model performance collapses as input grows (Liu et al., Chroma 2025).

• Expanding the context window doesn’t work. Training bigger models doesn’t fix positional encoding bias — a fact on page 3 gets treated differently than the same fact on page 97.

• Current agentic systems fail through statelessness. Decomposing tasks avoids context limits, but agents end up rereading processed documents and losing critical findings when context compacts.

• Conversational handoffs degrade signal. Passing information through natural language summaries causes single agents to outperform multi-agent systems under equal token budgets (Tran & Kiela, Stanford 2026).

• Stateful Swarms use a persistent blackboard. Specialized agents write provenance-tracked entries to an append-only, typed knowledge base instead of passing volatile context to each other.

• Read once, query forever. The blackboard accumulates structured understanding so you only pay the token cost to read a document set once.

• Costs drop by 39x. On the 1,251-task Harvey Legal Agent Benchmark, this architecture hit 17.75% strict all-pass at $1.30 per task, beating Harvey’s published 7.1% at $50.90 per task (w/ Opus hitting 10,7%, no costs mentioned, can assume same cost as 4.7 b/c the models have identical pricing). Doing the math, Swarms are between 66-98x more efficient than the best performers.

• Structure drives performance. Frozen Gemini models that scored 0% in standard agentic setups produced these results purely through structural coordination.

• Auditability is built in. Every blackboard entry tracks its type, source document, creating worker, iteration, confidence score, and support links.

• Failures are fixable. You can trace exactly where extraction succeeded but synthesis failed, which is a strict requirement for production in regulated domains.

• State is unavoidable. Recompute is waste, and prompt-as-memory is not a real solution for the economics of knowledge work.

Why Generative AI Currently Fails at Complex Queries?

The original promise of AI solving long-context knowledge work was AGI — a model that would solve whatever problem you threw at it in one shot. Just prompt and forget.

We’ve moved on from this fantasy, but some people still ride for a meeker version of this idea: eventually context windows will get big enough to where we don’t need any great engineering. However, when you dig into the details of how LLMs work, this fantasy face-plants real quick.

How Self-Attention becomes very Expensive in Long Context Inference

Attention computes a score between every pair of tokens. As context grows, compute scales quadratically. This means as your context length increases, the cost of computing Self-Attention grows very, very quickly. And this is something you have to recompute with every new message: meaning that you’ll spend a LOT of money recomputing mostly irrelevant tokens to answer even basic questions (keep in mind as your input context grows, you’ll most of your input tokens will likely be irrelevant or distracting to your actual query).

FYI, we still don’t know how to counter this issue. The most efficient edge models have kicked the can down the road by layering in their expensive full self-attention layers between stacks of cheaper variants. By not computing the cost of full MHA every layer, you can drive down costs of your network, but we still have to pay the piper eventually.

But lets say you bite the bullet on this because you’re determined to send all your tokens to one model. Your worries don’t end here. The most expensive aspect of long context work is in the hardware.

How HBM Makes Serving Long Context Expensive

To serve those tokens, the GPU must constantly move the KV cache in and out of High Bandwidth Memory (HBM). HBM is the most expensive component on the chip, and its bandwidth is physically capped.

Dao et al. formalized this as the IO-awareness problem in FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (NeurIPS 2022) when they found that the bottleneck in long-context serving is not the FLOP count of attention but the number of reads and writes between GPU SRAM and HBM. FlashAttention reduces memory accesses but does not eliminate the quadratic scaling of the mechanism itself.

Tldr: you want big memory chip for big context; you give Micron big money.

But let’s say you have a lot of money, and no actual AI expertise to solve problems (all too common in legal tech). So we go ahead with trying to build long context models, no matter the context. This is where you hit the next set of problems.

Training Models is Unreliable (especially over Long Contexts)

With all that money floating, couldn’t you eventually train a super long context model to solve all your problems? Not really.

The scaling curve for knowledge injection is not linear. Forcing a single model to hold all knowledge creates interference. Optimizing weights to improve reasoning in one domain often degrades performance in another. You can push a model to ace a benchmark, but you cannot reliably bake deterministic facts into its weights for production use. This is why LLMs are still hit with all kinds of random and unpredictable hallucinations, even for things that should be in their cutoff dates.

What about in-context learning (stuffing all the important context in memory)? Assuming your users are willing to go through the effort of sharing their entire strategy, relevant context, and jurisdictions in the context window, you hit attention degradation. A fact on page 3 receives different attention than the exact same fact on page 97. Liu et al. demonstrated this directly in Lost in the Middle: How Language Models Use Long Contexts: on multi-document QA, model performance dropped over 30% when the answer document was positioned in the middle of the context versus the beginning or end. As long as LLMs use Rotary Position Embedding (RoPE), which introduces a decay that biases attention toward sequence boundaries, we will continue to have to deal with this.

Chroma’s 2025 Context Rot: How Increasing Input Tokens Impacts LLM Performance study extended this finding across 18 frontier models and found that every model exhibited performance degradation as input length increased. The degradation is nonlinear and unpredictable: models can hold near-perfect accuracy to a threshold, then collapse.

So even if you really want to set your money on fire by training larger models for more context, you’re not guaranteed the returns you want. And when your model eventually fails, you’ll have no clue what steps to take to fix it (you can retrain, but that’s costly and doesn’t guarantee behavior).

This bitter pill is what has shifted LLM training priorities over the last 1.5 years. Open any modern LLM training recipe, and you see that the models are trained on tool use. This is because the labs have explicitly shifted their priorities away from single-shot capability to agentic systems — instead of hoping one call does everything, call a model a bunch of times, often including tools like calculators, code runners, web search etc. This way we can cover the gaps of each individual model call/method through a variety of techniques.

(when engineers use variety to make their lives better, they’re celebrated; when I want variety in my life, people say I have commitment issues. smh these double standards).

The added benefit of agents (imo the main advantage) is the traceability. If something fails, you can isolate it very quickly by tracing the tool calls and their I/O. You can finally see if your “DO NOT HALLUCINATE” instruction fired off as it should’ve.

Agentic Systems are amazing, and very useful. It’s no coincidence that the mainstream adoption of Claude and GPT in knowledge work came from their agentic systems CoWork/Anthropic and Codex, respectively. However, they have a few issues:

1. They’re very expensive to run, and they waste a lot of tokens relearning things (rereading docs they’ve already studied, rereading their agent trails etc) or redoing things because they lost instructions somewhere.

2. Remember the problems with building long context models? Still an issue. So they deal with long contexts by compacting their windows, which makes the above worse since a lot of important things are lost when we compact.

3. These agents can be hard to extend beyond their baseline

4. The pace of LLMs makes agentic systems very hard to trace and stop mid-run since they spew out an overwhelming amount of text very quick. This makes their outputs very unmaintainable and increases the amount of rework required. This is why many people report negative ROI from these agents (since they build systems/do work that they don’t understand and thus end up having to redo).

These were some of the issues that we’ve been trying to address at Irys when building meaningful long context agents for knowledge work. We’ve realized that the solution to this is not to incrementally improve a limited paradigm, but to rebuild stronger foundations.

Why Stateful Swarms Are All You Need for Knowledge Work

To understand why this architecture works, you have to break down the two terms individually:

• Swarm: The system replaces the single-model monolith with a fleet of coordinated AI agents. Instead of expecting one generic prompt to parse an entire matter, the architecture delegates tasks to specialized worker agents that handle distinct execution phases.

• Stateful: The system maintains a persistent, append-only, typed knowledge base that accumulates data over multiple iterations. The intermediate state survives across sessions, meaning the system doesn’t lose its mind when a user opens a new window.

We solve the coordination problem through the Blackboard Pattern. The blackboard is a structured shared state that functions as the primary asset of the system. Agents never pass context blocks directly to one another. Instead, a worker reads the blackboard, performs a tightly bounded piece of cognitive labor, and writes its findings back as a typed entry with explicit source provenance.

The system executes this process across ordered phases. A seed planning agent first maps the document landscape to define key extraction criteria, specific questions, and target schemas. Parallel extraction workers then populate the blackboard with observations, tracking metrics like source quotes and confidence scores. Higher-tier analytical reviewers step in next to synthesize these extractions, resolve conflicting data, and trace explicit evidence chains. Finally, a supervisor runs a deterministic convergence check: if knowledge gaps remain, it dispatches targeted follow-up workers to parse specific document sections rather than re-reading the entire corpus.

To prove the validity of this architecture, we ran it against the full 1,251-task Harvey Legal Agent Benchmark (LAB). The data proves that structural coordination matters far more than raw model scale.

The system achieved an 83.74% pooled criteria pass rate and a 17.75% strict all-pass rate across 24 practice areas. It did this at a total compute cost of $1,626.08, which averages out to $1.30 per task. Harvey’s published baseline on their twin holdout distribution reports a 10.4% strict all-pass rate at an average cost of $50.90 per task. The stateful swarm delivers a 39x structural cost reduction while significantly outperforming their best system.

(If you compute the cost per passing point, this jumps up to 98x the efficiency, but we’re not stressing that too much since performance is the biggest driver right now and we don’t have the private hold-out set).

This economic shift is driven by precise model routing over cheap, commodity infrastructure. The system routes high-volume parallel extraction tasks to Gemini 3.1 Flash Lite at $0.25 per million input tokens and $1.50 per million output tokens. We reserve the mid-tier Gemini 3.5 Flash strictly for reasoning-heavy steps like planning, cross-document analysis, and final synthesis, at $1.50 per million input tokens and $9.00 per million output tokens.

In Harvey’s baseline evaluations, these exact same Gemini models achieved a 0% strict all-pass rate when deployed inside standard agentic setups. We specifically picked these models to prove a point: by changing how the models interact — moving from stateless tool-calling pipelines to a stateful, iterative blackboard — the same frozen weights jump from absolute failure to 17.75% perfect execution. This is proof of our larger thesis that structural improvements can unlock capabilities not present in relying purely on the model layer.

Our system design allows us to tackle one of the biggest blockers in multi-agent systems. Tran & Kiela (Stanford, 2026) argued in Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets that every inter-agent handoff loses information — grounded in the Data Processing Inequality, which predicts that coordination can only degrade performance.

We show that their finding holds for conversation-based multi-agent systems where agents pass context through natural language summaries (which creates a loss of information and an ability to enforce constraints). It does not hold for blackboard architectures where agents write typed, provenance-tracked entries to structured shared state. Our blackboard doesn’t lose information at handoffs. That is why the same Gemini models that achieve nothing individually produce 17.75% strict all-pass when coordinated through one.

So far, we’ve discussed how Irys’s Stateful Swarms paradigm solves 3 of the main issues plaguing current agentic systems:

1. The shared blackboard means we can accumulate knowledge over time, reducing the tokens from re-learning.

2. Blackboard allows prevents loss of information on compaction and other issues.

3. Our use of self-defining swarms (analyst writes custom prompts for whatever problem needs it) also ensures that it can handle a wide amount of flexibility in the problems we can solve (I’d encourage you to try the GitHub CLI on your own problems; part of the reason we’re open sourcing it is to get your insights into what you’d prefer differently).

We haven’t talked much more about the audibilty and transparency. That was because our new blackboard allows us to pull off one of of the best feats of auditability yet.

Why Auditability Matters More Than the Benchmark Score

No matter how good, a system is bound to fail. But a wrong answer can still be a useful answer, if we understand where it went wrong and how to avoid that.

Let’s see how the blackboard enables that. In our Stateful Swarms, every task produces a complete, structured record of how the system arrived at its conclusions. Each blackboard entry carries its type, source document and section, the specific worker that created it, the iteration it was created during, its confidence score, and which other entries it supports or contradicts.

You can literally study the blackboards as they progress over iterations and use them to see how the system evolved it’s understanding and how it learned things over time. In an upcoming UI update to Irys, we’re going to expose this learning evolution to our users so they can stop/redirect things mid-investigation as our system finds interesting learnings.

The main advantage of the blackboard for transparency is that it’s much clearer to read and it centralizes all important information to one place (instead of scattering the learning fragments all over like Sukuna’s fingers). This allows us to study both the success and failure to improve the outcomes. Let’s understand how.

When the system succeeds, the trace demonstrates why. On a credit agreement comparison that scored 40/40, the blackboard evolved from 7 seed entries to 2,400 grounded findings over 12 iterations: 2,044 source-grounded observations, 87 cross-document calculations, 135 identified gaps, and 113 analysis entries identifying specific deviations. Each analysis entry carries a supports field linking back to the evidence it synthesizes. A reviewer follows the chain from conclusion to evidence to source document.

When the system fails, the trace reveals why — and whether the failure is fundamental or fixable. On a sanctions entity extraction task that scored 80/85, the five missed criteria shared a diagnostic pattern. The blackboard contained entries with two name variants — “Zenith Petrochem Industries LLC” from one document and “Zenith Petrochemical Industries LLC” from another. Both were correctly extracted. Both existed in the state. What the system failed to do was cross-reference these entries and flag the discrepancy. In a separate miss, the system identified that a 49% ownership stake sat 1% below the OFAC 50% threshold and even mentioned that aggregate ownership by blocked persons could trigger a violation — but didn’t elaborate the aggregation principle with sufficient specificity.