Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6
A quick note before we begin: This article was largely written before Jensen Huang’s departure from NVIDIA on November 12. The circumstances surrounding his exit have received plenty of coverage elsewhere, so I won’t be revisiting them here. What interests me more is the remarkable business and engineering strategy he spent decades building, and how Nvidia has set themselves up to continue their position as the worlds most valuable company.
Today, we will be exploring one of the most misunderstood companies in the AI Industry right now. Despite their position at the top of the stock market, NVIDIA’s deeply technical business and unique customer dynamics make it hard for investors and analysts to put them in a box. NVIDIA also tends to make very intelligent long-term plays that are hard to understand for an industry typically obsessed with quarters. In this article we will be breaking down their strategy in great detail covering what they’re doing, why, how it can go wrong, and what would have be the case for things to work out. Keep reading if that interests you.
NVIDIA made $89 billion from data centers last quarter, growing 117% YoY, with company-wide gross margins of 75%. This is why their substantial investments in open-source AI confuse a lot of people:
Nemotron: Its new Bolt model squeezes 74 billion parameters into 24GB of memory, while an even smaller model fits into just 4GB. Both are designed to make capable AI accessible on local hardware.
Dynamo: NVIDIA’s open-source inference software promises up to 7× better inference performance on Blackwell GPUs, allowing customers to process far more requests with the same hardware.
Vera Rubin: NVIDIA reports up to 45× lower inference costs than Blackwell Ultra on a selected benchmark, continuing its push to make each generation substantially cheaper to operate.
Why would a company making most of its money from selling expensive computing hardware spend so much effort helping customers need less of it? Wouldn’t that eventually undermine its own business by letting people use less hardware for the same level of intelligence?
There’s actually a really smart reason for this, and it’s the kind of far-sighted strategic thinking that led NVIDIA to become the most valuable company in the world to begin with (remember that in the beginning the media and Wall Street relentlessly mocked CUDA and failed to understand NVIDIA’s ecosystem play). Presently, most of NVIDIA’s leather jacket money and growth is tied to a few “whale” companies that drop billions on AI infrastructure (one direct customer alone accounted for 16% of its revenue last quarter). These whales mostly pay the high margins to NVIDIA to ensure that their training runs proceed smoothly since the cost of paying for premium hardware is much lower than the cost of having your expensive engineering teams debug new hardware on frameworks they aren’t familiar with —
But this creates another issue. If a company can afford to spend billions buying GPUs, it can also (… cue the drum rolls) afford to spend billions finding ways to build their own baby formula instead of paying top dollar to suckle on Jensen Huang’s plentiful teats. And so you have all of NVIDIA’s biggest customers making their own chips: Google has its TPUs, Amazon has Trainium, OpenAI recently broke the market with their Jalapeno chips, all while AMD is competing for a share of the same market.
Essentially, it’s a war on all fronts for NVIDIA’s empire.
(If you’re interested, we broke down this dynamic in great detail in our article How AI will Change in 2026 predictions. It contains deep dives on how hardware is changing with so many new entrants, deep technical splits bw inference and training chips, and other detailed insights into how the AI industry is changing. Read it here) —
Back to our main article, this revenue concentration + incentive structure is why, we’ve seen so much capital and attention flow into the AI Inference Market. Unlike training frontier models, inference is something almost every business could eventually need. And reducing its cost doesn’t just save money on existing applications. It can make applications that were previously too expensive worth deploying in the first place. Consequently, in just 2026, six inference infrastructure providers have announced over $7 billion in financing, with NVIDIA itself participating in several rounds.
This sets up interesting question numero dos — why is NVIDIA investing in people that (at least superficially) bet against (their) GPUs?
The simple answer here is market expansion (also known as Jevons Paradox) — Jensen is betting that if the cost of inference goes sharply enough, it will open up whole new use cases for AI that will dramatically expand the demand for AI (we’ve seen this happen w/ tokens where the fall in token prices has made AI companies much more valuable since they’ve unlocked much more usage; the bet now is that we will unlock the efficiency involved with economies of scale to eventually make tokens more profitable as it did for other infra buildouts/commodities).
But does that work? What would make this successful, and when would this fail? In this article, we will be exploring 3 questions —
How (on a technical level) NVIDIA is making AI much cheaper from everyone.
The market expansion opportunities this creates.
What needs to happen for this to be a game-winning play instead of NVIDIA undercutting its own business.
If you’re unsure about what the long term play for NVIDIA is, this company map will help you understand one of tech’s most misunderstood companies.
Executive Highlights (TL;DR)
The core thesis: NVIDIA is deliberately making AI cheaper to expand the market for its hardware. But Jevons Paradox doesn’t guarantee that NVIDIA gets richer. For this strategy to work, customers must spend more on AI, that spending must require additional infrastructure, and NVIDIA must capture a profitable share of it.
The engineering is impressive, and it’s happening across the entire stack. LatentMoE offers up to 3.5× higher modeled throughput, Nemotron Bolt compresses 74B parameters into 24GB, and Dynamo improves utilization of existing clusters. Rubin and Groq LPX take this further by redesigning how inference runs across entire racks. NVIDIA isn’t just building faster GPUs; it’s trying to reduce the cost of intelligence at practically every level.
There is real evidence that cheaper AI creates more valuable work. Anthropic’s revenue grew 13× while compute expenses grew 4.3×, with over 11,900 customers spending $100K+ annually. But the opportunity varies by application. Routine workloads eventually become cheap commodities, while coding, research, and autonomous agents can justify increasingly large reasoning budgets. I suspect the latter will drive much of the industry’s profitable compute demand.
NVIDIA’s biggest customers are also its most dangerous competitors. Google’s TPU v7 beats GB300 by 1.85× on tokens per dollar in one FP8 comparison, but NVIDIA reverses that advantage through FP4 and software optimizations. Google’s next TPU introduces native FP4 support. NVIDIA’s moat increasingly depends on staying ahead in full-stack engineering, not simply producing better silicon.
The infrastructure buildout introduces a different kind of risk. CoreWeave’s depreciation and amortization already represent 54% of quarterly revenue, while power and construction delays restrict deployments. NVIDIA has committed $279B to manufacturing and supply capacity and provided up to $105B in contingent guarantees for OpenAI-linked infrastructure. These commitments can secure future demand, but become dangerous if paying workloads fail to absorb the increasingly productive hardware.
My current bet is that the market splits. Routine, predictable inference increasingly moves toward cheaper accelerators and depreciated GPUs, while frontier reasoning and complex agents remain hungry for NVIDIA’s latest systems. Three things will tell us whether this holds: how quickly enterprise AI spending grows, whether competing chips close NVIDIA’s software-performance gap, and what happens to older GPU rental prices as new data centers come online in 2027.
Hey there. I keep this newsletter ad-free, w/o sponsors, and mostly w/o paywalls to ensure that everyone has access to high-quality deep-dives on the most important ideas in the AI Industry without any bias. B/w my research, independent verification, and the chocolate milk needed to produce these works, these deep-dives take a lot of time, resources, and infrastructure. For just 10 USD/month or 100 USD/year, you can help me continue my vision of democratising access to the most important ideas in AI by becoming a paid subscriber over here —
PS — Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.
Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.
How is NVIDIA Making AI Cheaper
When it comes to making AI Cheaper, there are actually several ways you can eat that horse —
First, you can reduce the amount of raw math required to generate each token. Refer here to our exploration of How Much an AI Token Costs to get the full details.
Second, you might choose to cut down the amount of memory you have to buy just to hold the model’s weights and conversation history.
Third, you may reduce the standard autoregressive generation so that people aren’t wasting your high end GPUs on token by token generation. The most extreme version of this are diffusion models, which are I believe will be the future of Language/Multimodal Models for the reasons mentioned here.
Fourth, there’s cluster utilization, because a data center full of expensive GPUs still loses money if half the chips sit idle waiting on network transfers. When we studied WEKA a few months back, we discovered that this problem isn’t something that can be solved by simply spamming more hardware, and there since then I have come across a ton of interesting engineering focused on cluster utilziation. I think this spac is generally a bit overlooked by analysts studying it (it is a somewhat crowded market now, but there is a lot of moving dynamics in there constantly).
And fifth — often the biggest bill of all — is the months of human engineering required to clean data, wire up pipelines, and build the application before you ever serve a single request.
As we will learn, NVIDIA has shipped a wave of models, software schedulers, and rack-scale chips to attack these bottlenecks. We’re going to go through each layer, one by one, starting with how Nemotron 3.5 rewires the Mixture-of-Experts architecture to drive more efficiency.
(Fair warning, this will be a long section since the goal is to give you a really good understanding of the engineering fundamentals of their solutions and how they impact various industry focal points. If you’re busy and not interested in the engineering, the TLDR + this image map is a good enough to skip ahead, but I always recommending building your intuition into the platforms so you can build your own assessments).
How Does LatentMoE Fix the Hidden Data-Movement Costs in Nemotron 3.5?
NVIDIA’s Nemotron 3.5 release dropped two different models to match two operating budgets:
Nemotron 3.5 Super: A 120-billion-parameter multimodal model that activates roughly 12 billion parameters per token, built for heavy reasoning, long contexts, and multi-step agentic workflows.
Nemotron 3.5 Lightning: A 30-billion-parameter text model that activates roughly 3 billion parameters per token, stripped down for high-speed, low-cost inference.
Both models combine hybrid Mamba-Transformer layers, native speculative decoding, and Mixture-of-Experts (MoE) routing, the last of which comes with an interesting seasoning. Traditionally, MoE cuts your costs by routing each token to only a small fraction of the network’s experts; helping you get the knowledge capacity of a massive model while paying for the inference of a smaller one. That’s why MoE has become a mainstay in all modern LLMs (to the point that it was actually surprising that Meta refused to MoE for as long as they did for their Llama models).
However, even when you cut the FLOPs with MoE, you have to pay for the freeloading inactive experts in memory (AI models got free housing before you did), read their weights into the compute units, and — when the model is split across multiple GPUs — ship token activations back and forth across the network. That data movement creates two nasty bottlenecks depending on your traffic regime:
When concurrency is low (say, a few interactive users), the GPU spends most of its time waiting on high-bandwidth memory (HBM) wires just to stream in the selected expert weights.
When concurrency is high across a multi-GPU cluster, the interconnect saturates as the system dispatches full-width token vectors to whichever remote GPUs hold the chosen experts. While MoE reduces the total number of calculations for you but you’re still streaming lots of body positive bytes across your network, which adds a cost of it’s own.
This is what LatentMoE was built to solve. Simply put, it reduces the dimensions of your input vectors, making processing much cheaper (look at the image above). More mathametically, given an input representation h_in of width d the layer multiplies h_in by a shared down-projection matrix W_down to compress it into a much smaller latent dimension l (where l << d). All of the routed experts do their math inside that tiny l-dimensional space, and a shared up-projection W_up restores the result back to dimension d at the end:
h_out = W_up * Sum_i( g_i(h_in) * Expert_i(W_down * h_in) )
I gave you the longer math b/c I want you to look closely at the router gating function g_i(h_in) in that equation. While the experts operate on the compressed vector W_down * h_in, the router itself still looks at the uncompressed hidden state in full dimension d. This makes sure your routing decisions don’t lose important fine-grained features that get wiped out during compression.
(As an aside, notice how more and more companies are embracing reasoning in the latent space?? When this becomes mainstream, don’t forget that I was the first one to call it)
With these cost savings, each latent expert becomes much cheaper to read from HBM much lighter on the network; so Nemotron can pack in far more total experts and activate a much higher top-k combination of experts per token without spiking serving costs. This allows them to hit the perfect top-bins half volley of intelligence per dollar, the metric that the entire AI Industry is now chasing when they realized that AI models eventually need to make money AND that perhaps you don’t astra level intelligence to edit your spreadsheets based on your macros. The results speak for themselves: conventional MoE would need roughly 350 billion extra parameters just to match LatentMoE’s accuracy. Under modeled distributed serving assumptions, LatentMoE delivers up to 3.5x higher throughput at matched accuracy.
From NVIDIA: Standard MoE versus LatentMoE architecture
Naturally, when model calls get cheaper, man is tempted to spam them in autonomous agents. So we gotta look at the the next thing that breaks when you deploy Nemotron as an autonomous agent?
The answer is Context.
How NVIDIA reduces the cost of long context reasoning.
Imagine a software-engineering agent that has been running for four hours — reading repository trees, inspecting pull requests, running compilers, parsing build logs, and calling tools. As one might imagine, you are going to end up burning through a lot of tokens.
As we’ve discussed at length, long context reasoning in vanilla transformers has two major costs:
During prefill, full attention scales quadratically as O(N²) in sequence length N.
During token generation, caching past keys and values avoids recomputing the whole sequence, but that KV cache still grows linearly as O(N) in HBM — eating up gigabytes of memory per active agent and choking memory bandwidth on every single decode step.
This is why the entire AI industry has been moving away from vanilla transformers and instead putting all their attention on building kinky transformers (this isn’t the scientific term yet, but I’m doing my best to get it to catch on).
NVIDIA’s version of kinky transformers involves interleaving two different techniques together (generally a good approach to life) —
Mamba models fold each new token into a fixed-size O(1) recurrent state S_t = A_t * S_(t-1) + B_t * x_t. Because that state never grows, processing token 800,000 takes the exact same memory footprint as processing token 100. This raised a lot of eyebrows when it was released, but it never caught on as much since this recurrent is very lossy, especially when precise recall over time is needed.
Vanilla transformer layers rely on full self-attention on the input. These have perfect recall within their context window (technically a very dubious claim, but we will let it slide for now). So Nemotron takes a page out of newer edge models like Gemma 4 and Liquid AI to interleave full Transformer attention layers at periodic intervals — 4 Mamba layers to full self attention layer so that it can cut down the cost of long context inference.
Naturally, accommodating such a big change also requires them to rework their training pipeline a bit to make sure they can still work over a million tokens—
NVIDIA started by distilling Nemotron 3.5 Super across roughly one trillion tokens from a larger teacher model, then ran 65.4 billion tokens of specialized software-engineering mid-training over code repositories, pull requests, build logs, and agent trajectories.
Hammering the model on dense software-engineering trajectories made it a much better coder, but caused catastrophic forgetting on long-range retrieval.
To fix that damage, NVIDIA ran an extra nine-billion-token long-context recovery stage, which pulled Super’s RULER-1M long-context score back from a degraded 33.5 to 81.0.
Their acquisition of Gretel and subsequent investments into Synthetic Data (something we flagged in our March Market Report for 2025) probably helped them get the extra billions of training tokens relatively cheaply. This extra training (cheap) lets them cut the cost of inference dramatically —
Nemotron 3.5 Super hits 51.78% on SWE-bench Pro, beating the previous Nemotron 3 Ultra while activating 4.6x fewer parameters per token.
Okay, so LatentMoE and hybrid Mamba-attention make a 120-billion-parameter model much cheaper to run per token. Are we done? Not if you want to run it on a single GPU. Storing 120 billion parameters in standard 16-bit BF16 still takes 240GB of memory just for the static weights — before you allocate a single byte for the KV cache or runtime activations.
To get frontier-class models onto local workstations and single accelerators, NVIDIA had to figure out how to crush the weights themselves without lobotomizing the model.
How Nvidia uses Star Elastic and Two-Bit Quantization to Fit a 74B Model into 24GB?
In October, NVIDIA released Nemotron 3.5 Bolt, a model with 74 billion total parameters and 9 billion active parameters, which they somehow fit inside a 24GB memory budget (for context, storing 74 billion params in standard BF16 takes roughly 148GB of VRAM, so they squeezed them params 6x). Let’s understand how they pulled off this black magic.
First we got a lil something they call Star Elastic. Instead of chopping of layers randomly (which tanks your performance b/c some layers carry more importance than others), Star Elastic runs a router-guided neural architecture search constrained by your target hardware budget. It looks at how the router actually uses different layers, then varies both the total number of experts and their intermediate widths layer by layer to keep maximum capacity where it counts. SE pruned Nemotron 3.5 Super down from 120B total parameters (12B active, 22 routed experts per token) to the 74B total parameters (9B active, 9 routed experts per token) in Bolt, followed by recovery training to restore vision, reasoning, and agentic tool use.
Step two is where things get aggressive. Rather than quantizing the whole 74B checkpoint to a uniform bit width, NVIDIA matches the bit precision of each layer family to how badly it reacts to quantization noise:
Major feed-forward expert matrices (which eat up the vast majority of total weight storage) are crushed down to IQ2_XXS (this was a new one for me, crazy how many detailed there are in AI these days) at an average of 2.0625 bits per element.
Mamba linear layers are stored in four-bit NVFP4.
Attention weights, token embeddings, the language-model output head, and LatentMoE projection/router weights are kept at eight-bit Q8_0, protecting the sensitive pathways where rounding errors would otherwise snowball across layers.
Combining mixed-precision assignments with quantization-aware distillation reveals a non-obvious rule about memory-constrained AI: if you only have 24GB of VRAM, a massive 74B model crushed down to ~2 bits can actually beat a smaller model stored cleanly at 4 bits. The larger network learns high-level reasoning circuits during pretraining that survive aggressive weight compression better than a smaller network can ever learn from scratch.
It’s worth noting that unpacking ultra-low-bit weights and running 9B active parameters means that Bolt runs at roughly 84 tokens per second, vs smaller four-bit competitors like Qwen3.6 (~97 tok/s) and Gemma 4 (101 tok/s). You’re trading raw token velocity for heavyweight reasoning inside a single 24GB card.
Running two-bit lookup tables works well on a local workstation, but in a massive data center, you don’t want compute units wasting cycles unpacking software quantization formats. This is where our next round of optimizations becomes very handy.
How Does NVFP4 Preserve BF16 Accuracy at Four-Bit Precision?
Shrinking weights from 16 bits to 4 bits only gives you throughput benefits if the GPU’s Tensor Cores can multiply four-bit numbers directly without destroying model accuracy. Unfortunately, neural network tensors are full of massive outlier values; try to stretch a single scaling factor across a whole tensor (to accomodate the outliers), and all the normal values get squashed into the rounding bucket.
With its Blackwell architecture, NVIDIA baked a hardware fix for outliers directly into the silicon via NVFP4. Each weight uses a tiny 4-bit E2M1 floating-point format (1 sign bit, 2 exponent bits, 1 mantissa bit), backed by a two-level hierarchical scaling scheme:
Every tiny microblock of 16 contiguous values gets its own dedicated 8-bit FP8 scale factor.
The entire tensor shares a second, higher-precision global scale factor.
Because every 16-number block gets its own local scale, an outlier in position 4 only stretches the scale for its 15 immediate neighbors, leaving the rest of the matrix untouched.
NVFP4’s two-level scaling mechanism
Storing those extra microblock scales eats up a bit of memory (adding 0.5 bits per weight of scale metadata, so you don’t get a literal 4.0x footprint cut from BF16). This means that on Nemotron 3.5 Super, the NVFP4 checkpoint holds 99.62% of median BF16 accuracy across 36 benchmarks while running at 2.43x higher interactive serving throughput.
If you want to go even lower, NVIDIA pushed the same compression playbook down to the edge with Nemotron 3.5 Bolt E4B by taking a Gemma 4 E4B base and running NVIDIA post-training, to make Bolt E4B fits inside a strict 4GB memory budget. This geodude build model packs a 64K context window, tool calling, and toggleable reasoning, beating NVIDIA’s previous small model by roughly 40% across conversational-agent benchmarks and letting developers run continuous local intelligence.
Read more: Bolt E4B benchmarks and local-agent demonstrations
At this point, we’ve shrunk active FLOPs with LatentMoE, tamed the KV cache with Mamba, and compressed the weights with Star Elastic and NVFP4. At this point, there’s very little you can do to speed up the generation of tokens without reducing their quality. So can we finally go home now?
Lol, no. Remember that individual tokens are largely worthless. Valuable outputs require many tokens. So if we can’t speed up one token, perhaps the next frontier is to explore how we can speed up the generation of groups of tokens.
Read more about Spec Decoding and other Inference Optimization here.
How Do LiLiCorr and Dory Trade Extra Compute for Faster Tokens and Deeper Reasoning?
Even if a single forward pass is fast, generating a 400-token response requires 400 back-to-back forward passes. Speculative decoding breaks that serial chain by using a tiny, fast “drafter” model to guess several future tokens at once, then letting the big target model verify the whole batch in a single parallel pass. Spec Decoding relies on the assumption that, on average, a good portion of the draft model’s speculations are correct. This means that a good portion of the inference is done by a much smaller and cheaper model. Instead of paying for k target model passes, we pay for 1 target model pass and k (much cheaper) draft model passes. Since the draft model is significantly smaller and faster, the overall cost per generated token is substantially reduced.
There is an interesting dilemna with Spec Decoding:
If the drafter predicts tokens sequentially, drafting itself becomes slow.
If the drafter predicts multiple future positions in parallel, it runs fast — but because each position is guessed independently, the words often clash with each other. For example, imagine that position t+1 assigns high probability to both dog and cat, while position t+2 assigns high probability to both barks and meows. A parallel drafter picking the top token at each slot independently can easily output dog meows — a nonsensical pair that the target model immediately rejects, wasting the draft.
LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding — NVIDIA Nemotron fixes parallel incoherence without going back to slow step-by-step neural drafting. Instead of picking one winner at each draft position right away, LiLiCorr keeps a small shortlist of top candidate tokens at every position, forming a candidate lattice. Next, a tiny two-layer Transformer sweeps across that entire lattice in a single parallel pass. Each candidate token gets a representation enriched by its neighbors in the lattice, the drafter’s confidence, and the target model’s last verified hidden state. That single pass outputs compatibility vectors for every candidate, allowing the system to score how well neighboring words fit together in parallel, followed by a dirt-cheap sequential pass over those precomputed numbers to pick the best path.
On Nemotron 3.5 Bolt, a 1.13-billion-parameter LiLiCorr drafter (excluding shared embeddings and output head) proposing 7 tokens at a time averages 4.68 accepted tokens under greedy decoding and 4.25 under probabilistic sampling — a massive latency win when serving interactive users at low concurrency.
While LiLiCorr spends a bit of extra compute horizontally to draft future tokens faster, Nemotron Labs Dory 1.0 spends extra compute vertically to make a small model punch above its weight class. Dory is a 4-billion-parameter NVFP4 model built with input layers, a shared recurrent middle block, and output layers. Instead of passing through the middle layers once, Dory can loop the same hidden state through that shared block multiple times before emitting a token.
Why loop a 4B model instead of just loading a 12B model? Looping lets you keep a tiny 4B footprint in memory while you can unlock a lot of the intelligence in small models by exploring their internal representations better (which perfectly aligns with our latent space reasoning thesis that most models don’t fail due to a lack of intelligence but from inadequate exploration of their internal representations, more details here)
Notice that going from 1 loop to 2 loops adds over 9 points on SWE-bench Verified and 7.6 points on SciCode, while doubling to 4 loops hits diminishing returns on those two benchmarks (though Terminal-Bench 2.1 keeps climbing from 18.1% to 22.2%). To keep 4 loops from taking 4x the KV cache memory, Dory uses last-loop KV caching, overwriting the recurrent block’s cached states on each pass so memory stays completely flat.
Now step back and look at what happens when you take these fast, compressed models and deploy them across a thousand GPUs in a production data center. Even the most efficient model checkpoint will bleed money if the cluster software schedules it poorly.
Dynamo: Getting More Out of Existing GPUs
Even the fastest GPUs waste money when workloads are poorly scheduled. LLM inference has two very different phases: prefill, which processes prompts in parallel and is compute-intensive, and decode, which generates tokens sequentially and is largely constrained by memory bandwidth. Running both on the same GPUs creates unnecessary contention. NVIDIA’s Dynamo separates these workloads into specialized GPU pools, uses NIXL to transfer KV caches between them, and routes requests toward machines that already hold relevant context. This becomes particularly valuable for AI agents making dozens of tool calls, where repeatedly processing the same conversation history wastes substantial compute.
Thanks to their optimizations NVIDIA reports up to 4x faster time-to-first-token and 1.5x greater throughput in agentic workloads. Combined with TensorRT-LLM’s kernel-level optimizations, Dynamo effectively increases the useful capacity of existing GPU clusters without requiring more hardware.
Once Dynamo and TensorRT-LLM squeeze the software bubbles out of the cluster, you hit the hard physical limits of the hardware itself: how fast wires can move activations between chips, and how many megawatts of electricity the local utility will let you pull from the grid. This is where our final optimization comes in.
Why Is NVIDIA Pairing Vera Rubin GPUs with Groq 3 LPX Accelerators?
Vera Rubin takes this system-level optimization into hardware. Its NVL72 rack connects 72 GPUs through NVLink 6, delivering up to 3.6 TB/s of GPU-to-GPU bandwidth to reduce communication bottlenecks in large Mixture-of-Experts models. Rack-level power smoothing also helps operators use more of their available electrical capacity.
NVIDIA reports roughly 10x higher DeepSeek-R1 inference throughput per megawatt than Grace Blackwell, although this doesn’t translate directly into a 10x reduction in total costs. The more interesting development is its integration with Groq 3 LPX, a specialized accelerator system offering 128GB of on-chip SRAM and 40 PB/s of aggregate bandwidth. Rather than forcing GPUs to handle everything, NVIDIA proposes using Rubin for compute-intensive prefill and attention, while Groq handles latency-sensitive feed-forward and expert operations. The combined architecture reportedly delivers up to 35x higher throughput per megawatt than GB200 NVL72 under NVIDIA’s benchmark conditions. The strategic implication is more important than the benchmark: NVIDIA is preparing for a world where GPUs aren’t necessarily the best processors for every inference operation. By supplying the interconnect, specialized hardware, and orchestration software, it can retain control of the computing system even as workloads become increasingly heterogeneous.
There are several other engineering optimizations they make to really drive down the cost of intelligence, but those are lot more serving/topic specific (or are just not that major/novel), so I don’t think we should spend a lot of time covering them. Instead, we must now take all of this is a setup for the question that most of you are likely here for: why should any of this actually make NVIDIA more money?
When Does Making AI Cheaper Actually Make NVIDIA Richer?
Whenever you ask bulls this question, they immediately point to Jevons Paradox: Make a resource cheaper, people use way more of it (and find completely new uses for it), and thus the total spending goes up around the new industries that spawned out of this new prosperity.
I think that before we really get into the whole AI buildout discussion, it’s important to study a bit of history and see why infrastructure operators bet the farm on that dynamic. When we study how heavy infrastructure (think railroads, fiber-optic networks, and hopefully AI data centers) have made money in the past, a surprisingly consistent pattern emerges —
In the beginning your capital heavy investment (say railway system) is used by a few people. This means that every individual person has to also carry the cost of paying for the RnD, building, operations etc. To keep trains profitable, you might have to pay 100 USD on a 99 USD cost, giving the railway line a measly 1% margin. At this cost, people would rather Uber or drive.
Now imagine that the government drops the cost by 10x to promote more train usage. This isn’t the financial suicide it seems like on the surface:
First, fixed-cost amortization across elastic volume: Because tickets are 10x cheaper, people use the rails for everyday trips they never took at $100. So instead of growing 10x, your volume might grow by 15x (pushing top-line revenue from $100 to $150) while also spreading your fixed infrastructure (railway lines, ticketing systems, train cars etc, all of which would be very expensive one time payments) and customer-acquisition costs across 15x more rides.
Second, scale-driven systems innovation: Running at 15x volume exposes every inefficiency in your operations and forces you to innovate on the underlying plumbing (the exact role of LatentMoE, NVFP4, and Dynamo; this research would not have been worth it in a world that wasn’t processing hundreds of trillions of tokens regularly).
Between high utilization and engineering optimizations, your fully loaded cost per unit craters from $99/head down to $7/head. This means that your profits go up 45x (15x usage spike * 3x profit per head), even though your price 10x.
Tie this back to Nvidia and the parrallel becomes clear: crater the cost of intelligence (optimizations), ensure that the intelligence flows through your system (the CUDA ecosystem; in our rail example, worth noting that it’s not the train track maker that profits from Jevons, it’s the railways ), and you will ensure a long lasting dynasty.
So is that all? Not quite. To assume that this play will work out for Nvidia, 3 things have to go right: customers will have to spend more total dollars on AI, this spending will require net-new computing hardware, and that NVIDIA will be the one cashing the hardware checks.
Depending on which of those three links holds or breaks, we get three very different futures (all of which can play out simultaneously across different parts of the market):
NVIDIA creates new demand and captures the value: Cheaper inference unlocks high-value applications, customers buy more compute, and NVIDIA keeps its crown as the default infrastructure supplier of an expanded empire.
NVIDIA creates the demand, but someone else captures the margin: AI usage explodes, but custom hyperscaler ASICs, alternative accelerators, and cloud landlords pocket a growing share of the industry’s profit pool.
NVIDIA creates demand, but efficiency compounds even faster: The increased demand isn’t enough to offset the falling costs, essentially dropping profit margins.
For the rest of this article, we will study the three outcomes which ones might play out (in what contexts since all 3 can happen at once in different degrees and different subspaces) to help us identify where the market is headed and how we can ahead of.
Outcome 1: NVIDIA Creates New Demand and Captures the Value
One of the biggest mistakes analysts make when modeling cheaper inference is assuming all AI workloads have the same price elasticity. In reality, enterprise AI splits into two completely different economic buckets:
Capped-value workloads (like a customer-support chatbot): Once a bot is smart enough to resolve routine refund tickets, making the model twice as smart doesn’t justify spending twice as much. When inference gets 50% cheaper, the enterprise simply pockets the savings and moves on.
Uncapped-value workloads (like an autonomous coding agent): Give a coding agent 10x more test-time compute to inspect repository dependencies, draft candidate patches, run unit tests, and verify its own pull requests, and it starts solving engineering tickets it previously failed on. Here, extra tokens aren’t an overhead cost; they directly manufacture higher-value labor.
Given the conversations around Swarms breaking into systems, I think we might even see a future subgroup of uncapped workloads, which essentially operate on the insurance model — did you really invest resources into checking for vulnerabilities in your base, with certification agencies for the security measures conducted (so even if you get hacked, insurance would cover you). The reason I’m calling this out differently is because unlike standard uncapped value workloads (which are economically beneficial), these are more insurance/downside protection coded ( can you really afford to get hacked and possibly lose billions from reputational loss; and if you do get hacked wouldn’t you want to be able to say that you did what was industry best so you’re not completely ruined by bad PR). In a way, this kind of growth would be the AI industry monetizing a problem it helped create (although I’m sure some will argue that the problem was the human written bugs from older times).
We don’t have to guess whether these uncapped workloads are driving real hardware demand; Anthropic’s November S-1 amendment gives us the receipts:
In the first nine months of 2026, enterprise demand for coding and agentic workflows pushed Anthropic’s revenue up 13x to 33.24 billion USD (driven by 11,900+ enterprises spending over 100K USD a year, including 250 whales spending over 10M USD).
To feed that usage, Anthropic had to grow its compute bill 4.3x to 18.81 billion USD. In fact, demand outran their contracted clusters so badly in Q3 that Anthropic had to drop 5 billion USD on overpriced, short-term spot compute just to keep the lights on (temporarily swinging them from a 974M USD Q2 operating profit to a 567M USD Q3 loss).
At one level, Ant eating that 5 Billion Penalty to get energized racks online ASAP is a HUGE positive signal. However, there is a less flattering alternative interpretation here. Because Anthropic’s revenue grew 13x while its compute bill only grew 4.3x, the revenue Anthropic extracts from every dollar of compute jumped from 0.58 USD to 1.77 USD (even with training and R&D baked in). When NVIDIA’s engineering makes clusters more productive during a hardware shortage, model providers and clouds don’t hand 100% of that efficiency back to NVIDIA in new chip orders — they keep prices steady and pocket a big slice of the efficiency delta as their own margin expansion.
Now here, one might argue that Nvidia’s ecosystem that lets it engineer efficiency at scale will create a premium for Nvidia products in the short term, similar to their existing premium in Cuda, but I think the overall trend is toward customers pocketing efficiency gains, simply b/ of the value chain involved and how far back Nvidia has to sit:
Useful AI work → Customer spending → Net-new infrastructure required → NVIDIA’s share of hardware orders → NVIDIA’s gross profit
Once Anthropic and its 11,900 enterprise accounts (who generate 74% of its revenue) stare at an 18.81 billion USD annual compute bill, their number one financial priority becomes breaking that fourth link in the chain — which brings us directly to Outcome 2.
Outcome 2: NVIDIA Creates the Market, but Someone Else Gets the Money
Every billion dollars a hyperscaler or frontier lab sends to Santa Clara gives them another billion reasons to build an escape hatch ( even Anthropic, which has historically only been laser focused on models and crying wolf to not have to compete against open models, explictly laid out a 4-hardware plan across NVIDIA GPUs, AWS Trainium, Google TPUs, and AMD accelerators to maximize their intelligence per dollar).
Unfortunately, this investment in cost reduction, might not trickle down to the customers. When it comes down to it, if a model/agent company can drop their costs of intelligence 10x, what stops them from keeping the margins for themselves. Especially in a world where competitors will not undercut them (compute is constrained, so there is an upper limit to the amount of demand you can capture there). Too see this phenomenon in action, let’s take a look at some of the research done by the Gs at SemiAnalysis and their breakdown of the cost of Google’s TPU v7 —
Developing the chip internally costs Google roughly 1.03 USD per chip-hour (or 1.21 USD on an external ownership basis), which Google then rents out on Google Cloud for ~2.00 USD per chip-hour on a three-year commitment. Google can essentially price just below the GPU rental cost, drive demand to themselves, and pocket the margins for themselves.
So why hasn’t everyone dumped NVIDIA for cheaper ASICs already? This is where Nvidia’s software ecosystem becomes clutch. Thanks to the ubiquity of CUDA and Nvidia GPUs, a lot of effort has been invested in improving the utilization of Nvidia’s hardware (both by internal and external folk), which lets them overcome the raw hardware differences. This is shown very clearly in the October SemiAnalysis InferenceX benchmarks on Qwen3.5 (at 200 tok/s/user):
On a pure hardware basis (both chips running standard FP8): Google’s cheaper TPU v7 beats NVIDIA’s GB300 NVL72 by 1.85x in tokens per dollar.
With NVIDIA’s full-stack optimizations turned on (NVFP4 + Dynamo + TensorRT-LLM): Because TPU v7 lacks native four-bit arithmetic, cutting the byte payload in half lets GB300 flip the scoreboard and beat TPU v7 by 1.84x in tokens per dollar (assuming your workload tolerates any unmeasured quality gap between FP4 and FP8; this is also why we spent so much time understanding the engineering efforts Nvidia is taking to keep intelligence high at much lower costs).
This inversion when software comes into play means that NVIDIA it’s relatively protected from costly wars of attrition, since it wins on quality, not cheaper bare metal. This has let them keep the margins for now, even as the ASIC boys have spawned off a whole zoo of competitors, and why I think that too much money is invested in chip designs and not enough in reshaping the ecosystem around it.
Back to Nvidia, this inversion and software moat does have it’s own kinds of risks however. Generally speaking software is easier to produce than hardware (especially as agents produce more and more of it), and unlike previous times, heavy weights have a massive incentive to attack the said moats. This means that Nvidia is in a perpetual arms race to keep their software ahead of the curve. We’re already seeing this in real time as Google’s next inference chip, TPU 8i, adds native FP4 support, erasing that specific advantage, which forces NVIDIA to invent the next system primitive (like Rubin + Groq LPX disaggregation) just to protect its pricing power.
Assuming worst case here, what happens if hyperscalers eventually close that systems gap? This leaves Nvidia with 3 options:
Taxing competitor racks (NVLink Fusion): By letting third-party chipmakers plug into NVIDIA’s rack fabric (cc it’s deal with Rebellions into NVLink Fusion for SK Telecom )NVIDIA can still sell switches and NICs when it loses the GPU socket. This will unfortunately be a pure volume play since networking plumbing doesn’t replace 75% flagship GPU margins. They will also lose some major customers since Google and AWS use their own proprietary fabrics anyway.
Charging for enterprise software: NVIDIA AI Enterprise lists at 4,500 USD per GPU per year (plus commercial NIM licensing). There is a huge market to address with enterprise software, and Nvidia’s brands will make it a preferred provider compared to the many other software grunts and new gen Palantir’s teeming in the market. However, to replace just 40 billion USD of annual hardware gross profit at an illustrative 90% software margin, NVIDIA would need nearly 10 million GPUs paying full list price (or ~20 million GPUs at a 50% paid attach rate before discounts).
Selling turnkey Sovereign AI factories: National buildouts in Japan and Korea love NVIDIA’s integrated racks and free open-weight models b/c governments lack internal ASIC teams. This can be a huge long term play as sovereign countries will want their ai to be protected in their own controls (so that they aren’t reliant on another countries policies; look at what happened when Fable was banned in other countries), but this will be a much longer buildout, and there is a chance that the other countries will eventually want to move their supply chain dependency off Nvidia onto local manufacturers.
Putting all this together tells us that whether Nvidia capturing margins, instead of becoming one of many, ultimately boils down to whether they can keep their full stack efficiency higher than everyone else. I parallel it almost to higher education — in an ecosystem teeming with colleges and online certifications, a Harvard/Stanford degree becomes more valuable since it acts as a signal (regardless whether it works or not). Similarly, as long as Nvidia keeps a strong full stack lead, I think the fragmentation in the AI hardware market will cause decision paralysis and make companies instinctively reach for Nvidia (even if there is a premium).
With all this discussed, let’s look at the weirdest risk of all: what happens if NVIDIA’s engineers are too good at their jobs?
Outcome 3: NVIDIA Creates Demand, but Efficiency Improves Even Faster
When a cloud provider installs Dynamo, switches to NVFP4, and makes its existing GPU cluster 30% more productive in the middle of a compute shortage, everybody celebrates. Waiting customers immediately soak up the extra capacity, and the cloud provider still buys every new GPU it can find.
Run that exact same 30% software upgrade when the cluster already has enough capacity to serve paying customers, however, and the math flips. Instead of buying more racks, the cloud provider just delays its next hardware order.
If effective compute supply starts growing faster than paying demand, NVIDIA runs into a competitor that is much harder to beat than AMD or Google: the millions of GPUs NVIDIA already sold two years ago (a brand-new Rubin rack has to generate enough revenue to pay back tens of thousands of dollars in fresh capital expenditure, while a depreciated H100 sitting in an already-wired rack only has to cover electricity, cooling, and basic maintenance to stay cash-flow positive +every time NVIDIA ships a free software update that speeds up inference, that paid-off H100 gets more productive for free).
We haven’t hit that saturation point yet. Silicon Data’s H100 rental index tells us that even the ancient H100s rent at around 3–3.1 USD per GPU-hour. As older fleets pay off their upfront capex, the productivity hurdle NVIDIA’s newest chips have to clear just to justify rip-and-replace upgrades gets steeper every year.
There’s another overlooked wrinkle that makes this analysis messy: software moves on internet time, while the electrical grid moves on utility time (this is also why so many neoclouds are hitting credit crunches, since their investors anchored their expectations to software lifecyccles (they were tech investors), and completely overlooked that the short-medium term constraints would all be blocked by physical problems that the tech industry had never dealt with, due to which growth hasn’t been as smooth as anticipated).
This means that while you can drop a 2x model optimization overnight, you can’t summon high-voltage transformers or substation hookups on command (the IEA estimates grid bottlenecks could delay roughly 20% of planned data-center capacity through 2030). That physical bottleneck pulls operators in two completely opposite directions depending on their balance sheet:
If you already own an energized building with a fixed megawatt cap: Power efficiency is king. Swapping older GPUs for Rubin racks that squeeze 10x more tokens out of the same electrical allocation lets you serve way more customers without waiting three years for the utility company — assuming you have paying customers lined up for those extra tokens.
If you are borrowing billions to build new sites from scratch: NVIDIA’s frantic annual hardware cycle and power delays put you in a brutal vise. Look at CoreWeave’s Q2 2026 numbers: the company brought in 2.575 billion USD in quarterly revenue, paid 640 million USD in net interest, and watched 1.393 billion USD (54% of its entire top-line revenue) vanish straight into depreciation and amortization — all while waiting on 748MW of undelivered power across two leased site groups phasing in through 2028.
Pause on that 54% depreciation number for a second. When more than half of your revenue goes toward hardware losing value on the shelf, timing can kill you. Buy GPUs too early before your power is hooked up, and NVIDIA launches a 2x faster architecture before your chips earn their first dollar; wait until the substation is live, and you risk hardware shortages or blown customer deadlines.
This is where NVIDIA’s incentives clash directly with its infrastructure partners: NVIDIA wants to make its own hardware obsolete every 12 months to force new purchases, while its debt-financed cloud customers desperately need older generations to stay economically relevant long enough to pay back their lenders (this is also why Coreweave and co have very generous depreciation schedules).
In 2027, that timing gamble will reach a boiling point. SemiAnalysis forecasts a massive wave of delayed data-center shells finally coming online in 2027 —
If empty buildings suddenly outpace chip supply — especially with the IEA’s November 2026 analysis projecting high-bandwidth memory (HBM) shortages through at least the end of 2027 — the bottleneck swings from power right back to silicon. NVIDIA has already positioned its balance sheet for that exact flip, locking in roughly 279 billion USD in supply and manufacturing commitments (per its latest 10-Q) so it controls the memory and packaging when customers finally have rooms ready to fill.
But what happens if all those new 2027 buildings come online at the exact same time as 10x hardware jumps, 3.5x LatentMoE gains, and 4x Dynamo scheduling speedups? Effective computing capacity could multiply far faster than raw megawatts or GPU shipment counts suggest — even while total token usage keeps climbing.
And if effective supply overshoots paying demand, NVIDIA can’t simply tee hee and walk away, because it has been using its own balance sheet to finance the buildout. In August, NVIDIA agreed to provide up to 105 billion USD in contingent guarantees (disclosed in its 10-Q) backing land, power, and building shells for OpenAI-linked infrastructure in Ohio, phasing in starting fiscal 2029.
On one hand, guaranteeing 20-year power shells is a smart strategic play:
Securing the Ohio campus locks in sockets for several future generations of NVIDIA racks.
Funding independent neoclouds expands NVIDIAs customer base so it isn’t held hostage by three hyperscalers.
But at the same time, it’s undeniable that those neoclouds are currently pass-through landlords leasing compute back to the exact same handful of frontier labs (like Anthropic, which carries roughly 895 billion USD in multi-year compute commitments across hardware suppliers, with 69.8 billion USD due in the next 12 months). So Nvidia is still left to deal with the reality that while they can help finance their customers’ infrastructure to speed up adoption, this will only work until their end-user revenue fails to cover the rent. This is another reason why Nvidia is so frantic about market expansion, since the Neoclouds it’s investing into can only survive if they see a lot of customer growth outside of the valley.
Where Does All of This Leave Us?
The biggest takeaway from studying NVIDIA’s engineering alongside its balance sheet is that “AI demand” and “GPU demand” are no longer the same line on a chart. Every optimization we covered in Part 1 drives a wedge between the two, meaning AI usage can 10x from here while hardware profits go through a massive regime shift.
If I had to bet on how our three outcomes play out, they won’t happen as separate timelines — they’ll split the market in two:
Routine, capped-value AI (basic enterprise workflows, standard RAG, customer support) will slide rapidly into Outcomes 2 and 3. Once a workload stabilizes, hyperscaler ASICs and paid-off H100s are more than good enough to run it, squeezing NVIDIA out of the commodity tier.
Frontier, uncapped-value AI (autonomous coding, continuous cyber-defense swarms, physical robotics) will stay locked in Outcome 1, because those workloads are so compute-starved that they need NVIDIA’s bleeding-edge system tricks just to be economically viable.
In other words, NVIDIA is quietly trading away the commodity floor of the AI market to keep a monopoly on the ceiling. Whether that trade makes NVIDIA richer over the next 24 months comes down to three signals I’ll be tracking:
Enterprise cohort graduation: Are 100K USD customer accounts graduating into the 1M+ USD tier fast enough to pay off the industry’s nearly 1 trillion USD in forward compute commitments?
Real-world ASIC software parity: How quickly do chips like TPU 8i close the production TCO gap once NVIDIA’s software lead is factored in?
Used H100 rental rates in 2027: When the delayed wave of 2027 data-center power finally flips on, spot rental prices on depreciated GPUs will be the first canary in the coal mine telling us if software efficiency has finally outrun paying demand.
Fun, fun times ahead.
Thank you for being here, and I hope you have a wonderful day,
Dev <3
If you liked this article and wish to share it, please refer to the following guidelines.
That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.
I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you’re interested in keeping up with my learnings.
Reach out to me
Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.
Small Snippets about Tech, AI and Machine Learning over here
AI Newsletter- https://artificialintelligencemadesimple.substack.com/
My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/
My (imaginary) sister’s favorite MLOps Podcast-
Check out my other articles on Medium. :
https://machine-learning-made-simple.medium.com/
My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/
Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/
My Instagram: https://www.instagram.com/iseethings404/
My Twitter: https://twitter.com/Machine01776819