Kimi K3: How an Open Weights Model Made Claude Fable Level Intelligence Cheap

A Deep Dive into What Makes Moonshot’s Newest Model Special

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction. We run on a “pay what you can” model—so if you believe in the mission, there’s likely a plan that fits (over here).

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

---

Kimi K3 attracted a lot of attention for it’s insane performance, cheap cost, and for being an open weights model that performs at frontier model levels. With it come several numbers that have been catching everyones attention: the massive 2.8 trillion total parameters (104 B active), 1-million-token context window, and an insane 57 at launch on the Artificial Analysis Intelligence Index (they’ve now upgraded it to 60, putting it within touching distance of the top and above other open weight models).

Intelligence aside, this system really stands out with increasingly important intelligence per dollar, completing high value value tasks at a fraction of the cost of the othe frontier models.

While all of that is very cool, I think people have been overlooking a much more interesting story regarding how Moonshot (the makers of Kimi) attacked transformer scaling limits across three different (but related) bottlenecks that dilute important signals in transformers:

• Across Tokens: Instead of storing every past token’s key-value pair across every layer, it maintains a dynamic recurrent state that updates with new inputs. This keeps KDA’s recurrent-state cache fixed in size (overall more efficient), although K3’s 24 MLA layers still use a KV cache that grows with sequence length.

• Across Layers: Standard transformers pass a blunt running sum of hidden states down the stack, which dilutes earlier signals. Attention Residuals lets deeper layers selectively retrieve earlier block-level representations, preserving fine-grained context across deeper architectures.

• Across Capabilities: Traditional Mixture-of-Experts pushes high-dimensional token representations through every selected expert, creating a massive inter-device communication bottleneck. Stable LatentMoE routes compressed latent states to hundreds of fine-grained experts instead, scaling model capacity without saturating network bandwidth.

Combining dynamic state tracking with cheap routing and direct layer retrieval gives you a system that maintains high arithmetic throughput and coherent memory over long autonomous task runs. To really amp up that last ability, Moonshot trained this architecture inside execution environments across thousands of sequential tool calls, persistent application state tracking, and millions of context tokens.

More generally, Moonshot consistently pushes the boundaries on interesting model architecture changes (they’re the ones that pioneered the Muon -Clip optimizer) so I think this aspect is worth studying in depth to understand the future of how models will be designed going forward. Let’s get deeper into it.

A visual comparison of why MuonClip let AI converge faster.

Executive Highlights (tl;dr of the article)

Kimi K3 achieves its exceptional performance per dollar by attacking four places where Transformers either lose information or become expensive: across long sequences, deep networks, large expert pools, and long-running tasks.

Across tokens, the problem is memory. Standard attention stores information about every previous token, so memory usage grows with the conversation. Moonshot created Kimi Delta Attention to replace most of this growing history with a fixed-size recurrent state that continuously updates what it remembers. Since compression can lose details, K3 places one global-attention layer after every three KDA layers, giving the model periodic access to the complete token history. Because KDA’s state changes in sequence, it also carries information about token order, allowing K3 to extend its context to 1 million tokens without modifying positional encodings such as RoPE.

(recurrent mechanisms are making an insane comeback)

Across network depth, the problem is information dilution. A normal Transformer repeatedly adds each layer’s output into one running hidden state. As the network becomes deeper, useful early representations can become buried under everything added later. Attention Residuals gives later layers access to summaries of earlier stages, letting them selectively recover the representations they need instead of depending entirely on the latest accumulated state. K3 groups its 93 layers into blocks to make this retrieval practical.

(FYI, this is another one of Kimi’s structural innovations. One of the reasons that I’m such a Moonshot glazer is that while a lot of attention is put on how good their models are, people don’t appreciate just how much good research they do to push the field in innovative directions)

Across experts, the problem is communication. Mixture-of-Experts models increase capacity by routing each token through a small selection of specialist weights. However, moving large token representations between experts spread across different GPUs can become more expensive than the expert computation itself. Stable LatentMoE compresses K3’s representation from 7,168 to 3,584 dimensions before routing it to 16 of 896 experts. Quantile Balancing, normalization, and bounded activations prevent this unusually large expert pool from collapsing during training.

Across long tasks, the problem is execution. A model can answer difficult questions while still failing at work that requires hundreds of actions, tool calls, corrections, and changing application states. Moonshot post-trained K3 inside reasoning, coding, and agent environments where it had to act, inspect the result, recover from errors, and continue until an external system verified the outcome. This produces unusually persistent behavior, although it can also make K3 expensive, excessively proactive, and sensitive to whether its earlier reasoning history is preserved.

K3 points toward a future where models do not process every token, layer, or parameter uniformly. They will increasingly decide what to remember, what to retrieve, and which capabilities to activate. If that direction holds, the next research frontier shifts toward better selectors, reliable judges, and persistent state — while hardware, memory systems, and post-training data reorganize around those decisions.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

How Kimi K3 Uses KDA to Get Compressed Memory Without Losing Full Attention?

I will now attempt to summarize the ~roughly 30K words we wrote self attention into two important bullet points — 

• Standard full attention lets every new token compare itself against every previous token in a sequence. That gives the model exact recall (within the window, not accounting for the model not overlooking/blurring the token, also subject to randomness behaving), but this causes the KV cache to expand linearly with sequence length. This means that memory footprint and bandwidth demands will break standard serving infrastructure when trying to hande the 1-million-token context that your AI girlfriend needs to remember how you like your little love letters to be written (something we broke down in depth when discussing how much a token actually costs) —

• Linear attention avoids that growing cache by routing inputs through a fixed-size recurrent state. Each token updates and reads from this compressed memory, keeping memory costs flat regardless of context length. However, this undrmines the precise recall of full self attention, leading to a more lossy compression of your corpus (see more here — How Long Context Inference Is Rewriting the Future of Transformers).

Moonshot built Kimi Delta Attention (KDA) to address this memory degradation. At its crux lies a simple idea — prioritize writing a delta-rule update that corrects an existing memory association before writing new data (generally a wise principle to consider w/ agentic systems these days), to keep KDA from overwriting representations blindly. It also assigns independent retention rates to individual channels, allowing separate parts of the state to preserve or discard features independently rather than forcing all channels through a single global forget gate. The mechanics of this change are quite interesting, so worth studying if you’re on the model layer and want to look at the different ways attention is evolving. For our purposes, this high level explaination will have to suffice.

Because compressed memory can still drop fine-grained details, K3 interleaves compressed state tracking with exact retrieval. Across its 93-layer backbone, the model runs a 3:1 hybrid layout: three KDA layers for cheap local token mixing, followed by one Gated Multi-Head Latent Attention layer that restores unrestricted global attention across the raw token history. This setup is topped off by a final global-attention layer to squeek out that extra performance.

This is essentially the same setup, scaled up, from their Kimi Linear research, where they introduced this architecture with a 48B param model and it outperformed the standard architectures (emphasis mine). “We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6× decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths.”

“(a) Performance vs. acceleration. With strict fair comparisons with 1.4T training tokens, on MMLU-Pro (4k context length, red stars), Kimi Linear leads performance (51.0) at similar speed. On RULER (128k context length, blue circles), it is Pareto-optimal, achieving top performance (84.3) and 3.98× acceleration. (b) Time per output token (TPOT) vs. decoding length. Kimi Linear (blue line) maintains a low TPOT, matching GDN-H and outperforming MLA at long sequences. This enables larger batches, yielding a 6.3× faster TPOT (1.84ms vs. 11.48ms) than MLA at 1M tokens.”

(although one might argue that a vanilla architecture is a bad baseline nowadays given how much is changing, but this was still directionally useful).

One interesting outcome of these changes is that your model now handles position across long contexts w/o needing standard positional encodings like RoPE. From an AI perspective, this is a change that deserves a deep-dive, which is what we will do next.

Why Does K3 Remove RoPE from Global Attention?

Standard self-attention has no concept of sequence order. The core score between two tokens comes down to a dot product divided by the square root of the head dimension:

Because dot products are permutation-invariant, swapping the order of the inputs produces the exact same arithmetic. Models use a causal mask to stop a token from looking forward, but it can’t tell whether a past key was generated 1 token ago or 100,000 tokens ago.

Rotary Position Embedding (RoPE) injects distance by rotating queries and keys in the 2D plane before computing their dot product. For a query at position m and a key at position n, the score becomes:

The absolute position terms m and n cancel out, leaving only the relative distance (n — m). The rotation angle theta_i for dimension pair i is set by a geometric base frequency:

Fast frequencies (low i) spin rapidly over short distances, letting the model distinguish adjacent tokens. Slow frequencies (high i) spin slowly, tracking long-range separation across the context window.

However, this approach struggles to scale context beyond your training limits. If you train at 8,000 tokens, the slowest frequency dimensions complete only a fraction of a single rotation. When you push the sequence to 1,000,000 tokens, those slow dimensions rotate into unseen coordinate regions, while the fast dimensions complete hundreds of full periodic cycles (remember that these are based on trigonometric functions, which lap around each other every 360 degrees/2 pi radians) . At lengths far beyond training, the model encounters relative distances and combinations of phase relationships that it was never trained on causing its positional coordinate grid is producing out-of-distribution values (not merely “sequence length” as is often misunderstood).

Standard fixes try to rescale theta. Position interpolation scales the sequence index down so that m / scale fits within the original 8K window. YaRN scales the base constant directly (e.g., jumping from 10,000 to 500,000 or higher) to stretch the wavelength of slow dimensions while interpolating fast ones. This works short-term but every extension requires manually selecting a new scaling factor and retuning hyperparameters for a fixed maximum length. AI promised to automate manual work, and now you end doing a lot of manual labor to make AI work. Isn’t this world simply wonderful?

To avoid the rising labor costs in China, Moonshot decided to skip the manual labor by forgoing by removing RoPE from its 24 global MLA layers. Instead of having global queries and keys carry position rotations, K3 tracks sequence order by feeding inputs into MLA pass through Kimi Delta Attention which tracks sequence evolution through an explicit recurrent state matrix S_t:

Because S_t depends strictly on S_(t-1), the state at step 100 cannot exist without executing the preceding 99 updates in chronological order. An earlier token’s signal decays over time through repeated multiplications of alpha, and is updated whenever newer tokens write to overlapping memory coordinates.

This creates a pretty interesting geometric diference:

• RoPE represents distance as a rigid geometric coordinate (n — m).

• KDA represents history as a dynamical system, encoding whatever information survives the intermediate state transitions.

By alternating three KDA layers with one NoPE MLA layer across its 93-layer backbone, K3 delegates sequence ordering to KDA’s state transitions, allowing the global MLA layers to perform pure content-based retrieval across already-ordered representations.

This simplifies the context extension a lot, and let Moonshot scale pre-training from 8K to 64K tokens, and then to 1M tokens in cooldown, w/o modifying frequency bases or interpolating position grids. Code base/Architectural simplicity is an often an overlooked aspect of performance, even though I think it’s one of the most important.

“Within this research setting, we found that differences in architectural complexity could account for 50% drops in productivity, three-fold increases in defect density, and order-of-magnitude increases in staff turnover.”

— How bad is Architectural Complexity. I would imagine this is worse now that AI Coding Assistants tend to be overtly verbose and overcomplicate code bases with extra tests, making maintaince harder.

The combination of KDA and NoPE allow Moonshot to not lose important signal across many tokens. However, this is not only one part of the battle when it comes to information dilution in transformers. Next, we will talk about how Moonshot built Kimi K3 to keep information across the layers (ensuring important information wasn’t diluted as the transformer reasons).

“Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer’s contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights…Scaling law experiments confirm that the improvement is consistent across model sizes, and ablations validate the benefit of content-dependent depth-wise selection. We further integrate AttnRes into the Kimi Linear architecture (48B total / 3B activated parameters) and pre-train on 1.4T tokens, where AttnRes mitigates PreNorm dilution, yielding more uniform output magnitudes and gradient distribution across depth, and improves downstream performance across all evaluated tasks.”

— [2603.15031] Attention Residuals

How Do Attention Residuals Let Kimi K3 Retrieve Features Across Model Depth?

Transformers solved horizontal context scaling by letting tokens query tokens across a sequence. Their solution for handling vertical depth has been a bit more…. primitive.

In a standard PreNorm architecture, each layer adds its computed output directly into the residual stream:

x_(l+1) = x_l + Layer_l(x_l)

By the time you reach layer 90, the hidden state is an unweighted sum of every operation that came before it. While this keeps gradients stable during backpropagation, it does unfortunately mean that your early, fine-grained representations get drowned out inside a massive accumulated vector.

(it’s perhaps a bit poetic that our AI, much like ourselves, overlooks its history and suffers from a strong recency bias/myopia)

Attention Residuals (AttnRes) fix this by running attention vertically through the depth of the network — 

Instead of inheriting an unweighted sum of the entire stack, a layer computes attention scores over previous layer outputs to pull the specific representations it needs. This lets your system get very customized with the level of feature extraction it needs, enabling a better quality of reasoning.

More proactive ones here might be thinking about how running full depth-wise attention across 93 individual layers would have your fast GPU memory moving like Al-Teta’s Haramball (if it doesn’t cause memory crashes). You would be correct. K3 avoids this by grouping its 93 layers into eight 12-layer blocks, with a partial final block at the top. The model attends across these block-level checkpoints rather than individual layers, capturing the cross-depth retrieval gains without running out of memory bandwidth.

In Moonshot’s 48-billion-parameter Kimi Linear run across 1.4 trillion tokens, Block AttnRes matched the validation loss of a standard residual model trained with 1.25x more compute. It also kept hidden-state vector norms bounded across deep layers and distributed gradient updates far more evenly across the stack.

The two sets of solutions ensure that the “input” to AI reasoning is of a high quality. But no amount of “preprocessing” (again I use these analogously instead of literally) can save you if your information is being processed by a crayon snorting idiot. If we think of the last 2 stages as cleaning the input and priming the problem space, then the next natural layer of attack becomes how do we level up the raw intelligence + knowledge stored in the network weights.

The answer, as has been increasingly for everyone in the space, has been adopting Mixture of Experts. If you’re reasing this newsletter, you’re likely familiar with MoE so I won’t waste time describing for the sake of driving up word count. But that also begs the question — given how table stakes MoE is, why bother talking about it?

Typically more experts allow you to add more specialization. So the folk responsible for K3 decided to expertmaxx to the moon with a massive 896 experts.

If that number means nothing to you, let me put this into context. Normal Frontier models are b/w the 128–256 expert range. For additional context, Kimi K3’s 896 is roughly 3.5× the viral DeepSeek-V3 and 7× Qwen3.

So…why doesn’t everyone do this? There are several engineering constraints in mind. Kimi overcoming these is a massive deal. Let’s cover how Kimi did that since I predict that that others will copy this approach.

How Kimi K3 Uses StableLatent MoE to Make 896 Experts Usable

Let’s do some math to understand why Kimi’s approach is such a big deal:

• K3 runs 896 routed experts per MoE layer and picks 16 for every token. Activating only 1.8% of the pool looks cheap on paper, but life is unfortunately rarely that simple.

• This still means that the model runs 104 billion active parameters per token (over three times K2’s 32.6 billion). Above and beyond the usual suspects you’d be paying for, all 2.8 trillion parameters must stay resident across your GPU cluster because the very next token might route to an entirely different set of devices.

• This means that you have to bend over backwards keeping high-count expert parallelism from choking on device communication.

• This is why you don’t see labs full send on an ungodly number of experts.

So what does Kimi do differently? Standard expert parallelism sends the full hidden representation to every selected expert across the network. If you select more experts, your all-to-all communication volume scales linearly with the model’s hidden dimension.

Moonshot decided that if the full representation was a problem, then they simply shouldn’t send the full representation:

• Two shared experts process the full 7,168-dimensional representation to handle universal transformations.

• The model projects the token representation down to a 3,584-dimensional latent space before routing.

• The 16 selected specialist experts run their computation entirely on this 50% compressed latent vector. (this is an interesting application of latent spaces that I think will eventually lead to the shared universal latent space for everyone to reason over that I think is inevitable, although I might perhaps be the man with a hammer in this case)

Splitting shared work from specialist work halves the tensor size moved across the network during routing, letting K3 consult 16 distinct experts without saturating inter-GPU interconnects. However, this still has to deal with a massive problem: Routing across 896 experts introduces severe training instability. When you push sparsity that far, a handful of popular experts attract most tokens while the rest starve, causing routing collapse and exploding activation values across deep projection layers.

Moonshot stabilized the setup with three specific mechanisms:

• Quantile Balancing: Updates router bias values directly from the score distribution needed to hit each expert’s target load, preventing load imbalances without using blunt auxiliary loss penalties.

• Pre-reconstruction Normalization: Normalizes the routed latent outputs before projecting them back to the full 7,168-dimensional space, preventing activation spikes.

• SiTU-GLU Activations: Replaces standard SwiGLU with a bounded activation function to clamp internal representation drift across deep layer stacks.

This super specialization of Kimi K3 is what unlocks it’s high performance on the intelligence while keeping the costs low (the 16 experts per run is still okay) (this part is my speculation).

Working across these 3 layers of solutions, you get a model that is able to think intelligently. However, our world only validates intelligence if it help your McKinsey Consultant or investment banker make their slide decks faster. So how does Moonshot ,actually get Kimi K3 to actually work?

How Was Kimi K3 Trained to Handle Long-Horizon Agent Tasks in Production?

Once pre-training gives K3 its raw model capacity, Moonshot uses reinforcement learning to shape it’s behavior towards multi-tool execution (take notes b/c this is the correct way to use RL/post-training, not trying to inject more domain knowledge).

Moonshot trained K3 across three domains: general reasoning, general agents, and coding agents. Instead of tuning on simple conversation logs, the agent environments ran inside live terminals, codebases, web browsers, and simulated tools like Slack and Notion. The training rewarded an execution loop: act, check compiler or tool errors, patch the code, and keep going until an external verifier clears the output.

An interesting side effect of this execution heavy RL is that b/c the model is trained to survive long runs, its default is to keep pushing boundaries(so when user intent is ambiguous, K3 often makes assumptions and executes instead of stopping to ask for clarification). It also likely contributes to the brittleness of Kimi when thinking history isn’t passed properly to it (both these limitations are mentioned in their release notes here: Kimi K3 Tech Blog: Open Frontier Intelligence)

All of this is reflected in the benchmarks:

• On AA-Briefcase, which tests multi-step execution converting raw files into complex deliverables, K3 scored near Claude Fable 5 on analytical quality.

• That performance carries a steep test-time tax: K3 averaged 83 turns, 120,000 output tokens, 56.4 minutes of wall-clock time, and $10.57 per task.

So where does this leave us? Why should you care about Kimi K3? Why did we spend all these words studying the architecture?

What Does K3 Tell Us About the Next Phase of Frontier Architectures?

Imo, Kimi K3 is a strong signal that the next generation of breakthroughs will come from unlocking better information flow within models. Building meta-cognition into architectures, so models can selectively filter, prioritize or queue look up will unlock the next generation of intelligence for models.

In other words, future architectures won’t process every token/layer uniformly. We are moving toward systems that dynamically decide what to keep in recurrent memory, what to pull across network depth, and which specialist weights to activate on the fly. Assuming that to be true, our new research frontier shifts from building intelligent systems to building reliable judges and state.

As models change focus from intelligence to useful work completed, our AI stack will haveto reorg it’s priorities. Here are some trends I expect:

• Hardware and interconnects: Peak FLOPS matter less than device communication. Clusters need higher bisection bandwidth and low-latency interconnects rather than just denser compute dies.

• Memory management: Serving runtimes can’t rely on monolithic KV caches alone. Systems need hybrid memory managers that track dynamic recurrent state vectors alongside paged KV blocks without fragmenting GPU RAM.

• Post-training data: Static text scrapes are losing value. The high-margin training asset is verified execution data — multi-turn terminal logs, sandbox trajectories, and external test harnesses where models learn to recover from actual tool failures.

Plan accordingly.

Thank you for being here, and I hope you have a wonderful day,

Dev <3

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819