jain.com — data centers, connectivity and security, analyzed daily

Tag: GPU economics

  • Everyone’s Watching New AI Silicon. CoreWeave Found 19.8% in Racks It Already Had.

    Everyone’s Watching New AI Silicon. CoreWeave Found 19.8% in Racks It Already Had.

    TL;DR · 30-second read

    The Short Version

    CoreWeave rents out computers that run artificial intelligence. On September 16 it published results from an independent industry test and said its machines handled AI requests faster than those of rival rental companies.

    The more interesting result is quieter: on machines it installed months ago, it got about 20 percent more work out of each chip purely by improving its own software. No new hardware.

    That matters because those chips cost a fortune. Squeezing more out of the ones you own, instead of buying more, is where the profit in this business actually sits.

    CoreWeave on September 16, 2026 published its submissions to MLPerf Inference v6.1, the industry benchmark round released the same day by MLCommons. Writing on the company blog, Shadi Saba said CoreWeave entered the Datacenter Closed division’s Available category across four NVIDIA platforms — HGX B200, HGX B300, GB200 NVL72 and GB300 NVL72 — and four model families: Qwen3-VL-235B-A22B, DeepSeek-R1-671B, GPT-OSS-120B and Llama 2 70B. Headline figures include 16,635 tokens per second per GPU on GPT-OSS-120B in the offline scenario, more than 1.19 million tokens per second from a single GB300 NVL72 rack, and 944,902 tokens per second on Llama 2 70B in the server scenario. NVIDIA, in a blog post of its own, announced that its next-generation Vera Rubin NVL72 made its MLPerf debut in the same round.

    One day later, on September 17, CoreWeave filed an 8-K disclosing a proposed $3.0 billion offering of convertible senior notes due 2033, with an option for a further $500 million, alongside an Equity Distribution Agreement covering at-the-market share sales and collared forward sales through eleven banks. The two disclosures, a day apart, describe the same business from opposite ends: what the fleet produces, and what it costs to keep building.

    Executive Summary

    CoreWeave’s MLPerf claims break into two categories, and they are not equally interesting. The absolute throughput records — 1,196 queries per second on the multimodal Qwen3-VL model, over 1.16 million tokens per second on GPT-OSS-120B from one rack — are a function of running NVIDIA’s newest shipping silicon, GB300 NVL72, at scale. Any operator with the same racks and the same power envelope is in that fight.

    The second category is harder to replicate. CoreWeave says that comparing its 72-GPU v6.1 submission against its 64-GPU v6.0 submission on GB200 NVL72, derived per-GPU server throughput on DeepSeek-R1-671B rose 19.8% in the five months between rounds. That gain came from serving-stack tuning, scheduling and operations, not from a new chip. If it holds in production, it changes the revenue a depreciating asset can generate across its life — which is the number that underwrites everything else.

    That includes the financing. The 8-K filed September 17 shows CoreWeave raising $3.0 billion in converts on top of an existing stack of senior notes carrying coupons from 8.500% to 9.750%. Capital that expensive is serviced by output per installed accelerator, and software gains on hardware already paid for are the cheapest output there is.

    The 19.8% That Didn’t Require a New Chip

    Strip the benchmark round down and one number does real work. CoreWeave reports that its DeepSeek-R1-671B per-GPU server throughput on NVIDIA GB200 NVL72 rose 19.8% between MLPerf Inference v6.0 and v6.1 — five months apart — through changes to its serving stack, scheduling and operations rather than new hardware. The mechanisms it names are specific: topology-aware scheduling through SUNK, which pins an inference workload inside a single high-bandwidth NVLink domain so that tokens moving between accelerators never leave the fastest interconnect in the rack, and fleet-level health monitoring through Mission Control. For a mixture-of-experts model like DeepSeek-R1 — an architecture that routes each token through a subset of specialised sub-networks, so traffic between chips is constant — placement is not a detail. It is the workload.

    Who this affects is everyone financing GPUs on a multi-year schedule. An accelerator bought in early 2026 has a fixed capital cost and a depreciation clock that does not care how efficiently it runs. If per-GPU output rises roughly a fifth without capex, the cost per token served falls by a comparable amount across the remaining life of that asset, and the residual economics of the installed base improve rather than decay. That is a materially different asset story from the one where value arrives only with the next generation of silicon.

    The claim deserves its caveats stated plainly, and CoreWeave states some of them itself. Per-GPU throughput is a derived metric — total system throughput divided by accelerator count — and the company notes it is not verified by MLCommons. The comparison also spans two differently sized submissions, 64 accelerators in v6.0 against 72 in v6.1, so scaling behaviour and configuration changes are folded into the same percentage as the software work. The direction is credible and the mechanism is named; the precision of the figure is CoreWeave’s own arithmetic, not the benchmark’s finding.

    The Unit of Account Moved From the Chip to the Rack

    The GB300 NVL72 is not a server. It is a rack in which 72 accelerators share one NVLink domain, meaning they address each other at memory-like speeds rather than over conventional networking. CoreWeave’s results read accordingly: over 1.16 million tokens per second in the server scenario and over 1.19 million offline on GPT-OSS-120B from a single rack, 944,902 and 1,136,100 tokens per second on Llama 2 70B, and 1,196 queries per second on Qwen3-VL-235B-A22B. The company’s GB200 NVL72 submission reached 901,058 tokens per second on GPT-OSS-120B in the server scenario — the same architecture, one generation back, and a visible step down.

    The consequence lands on the people who build and power facilities rather than the people who buy models. When the smallest sensible deployment unit is a rack drawing far more than a legacy hall was designed for, throughput per rack becomes a proxy for how much revenue a given square metre and a given megawatt can carry. Operators comparing a rack-scale NVL72 deployment against clusters of HGX B200 or B300 servers — which CoreWeave also submitted, and which it says led all HGX entries from cloud providers — are really comparing two different power-density and liquid-cooling commitments, not two spec sheets.

    Available Is the Word Doing the Most Work

    CoreWeave’s submissions sit in the Datacenter Closed division’s Available category. Closed means every entrant runs the same reference implementation, so results are comparable rather than a contest in custom optimisation; Available means the system can actually be obtained. That second qualifier matters this round, because NVIDIA used the same release to announce that its next-generation Vera Rubin NVL72 made its MLPerf debut. A benchmark headline generated by a platform that is debuting is not a price a buyer can procure against today; CoreWeave’s four platforms are all Blackwell and Blackwell Ultra, the generations in production now.

    For anyone running a procurement process, the practical read is to check three fields before treating any v6.1 result as a quote: which silicon generation produced it, whether the entry sits in the Available category, and whether the metric quoted is a submitted number or a derived per-GPU figure. CoreWeave also stresses that it ran on the same production clusters and images customers get rather than a benchmark-tuned rig — a genuinely meaningful distinction, though one the benchmark itself does not audit.

    Throughput Is the Collateral Story

    The day after the benchmark post, CoreWeave’s 8-K disclosed a proposed $3.0 billion offering of convertible senior notes due April 1, 2033, sold under Rule 144A to qualified institutional buyers, with an option for up to $500 million more and capped call transactions to limit dilution on conversion. A concurrent Equity Distribution Agreement adds at-the-market share sales and collared forward sales through a syndicate that includes Deutsche Bank, Goldman Sachs, J.P. Morgan, Morgan Stanley and Citigroup. Interest and conversion terms, the release states, are set at pricing.

    The comparison worth making is internal. CoreWeave’s own press release lists the existing obligations the new notes will rank alongside: senior notes at 9.250%, 9.000%, 9.750%, 9.625% and 8.500%, and earlier convertibles at 1.75%. Converts price far below straight debt because the buyer is paid partly in equity optionality rather than cash coupon — cheaper carry now, potential dilution later, which the capped calls are bought to blunt. For a business whose costs are dominated by accelerators, the servicing capacity for any of it comes back to tokens per installed GPU.

    That is the honest link between two documents filed a day apart, and it is a structural one rather than a causal one: a benchmark result is a performance claim, not a revenue disclosure, and nothing in the filings ties the offering to the MLPerf round. But the neocloud model runs on both halves at once — demonstrable output per accelerator to justify the capital, and access to capital to buy more accelerators. Software gains of the kind CoreWeave reported on already-deployed GB200 racks are the half that does not require either.

    Background

    CoreWeave, based in Livingston, New Jersey and listed on Nasdaq as CRWV, is a specialist cloud provider built around NVIDIA accelerators for AI training and inference — a category often called a “neocloud,” distinguished from general-purpose hyperscalers by its narrow focus and its heavy reliance on debt and equity markets to fund hardware. The company was recently named a Visionary in the Gartner Magic Quadrant for Cloud AI Infrastructure.

    MLPerf, administered by the MLCommons consortium, has become the closest thing the industry has to an audited comparison of AI serving performance, which is why vendors submit and why the categories matter. Inference — running a trained model to answer requests — is now the larger and more cost-sensitive half of the AI workload, because a model is trained once but served continuously. That shift is what makes tokens per accelerator, rather than raw chip specifications, the metric operators and their lenders increasingly watch.

    Sources

    Source: MLPerf® Inference v6.1 Results: CoreWeave Leads Providers — CoreWeave’s September 16, 2026 post detailing its Datacenter Closed, Available submissions across four NVIDIA Blackwell platforms. See also NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut, NVIDIA’s account of the same benchmark round.

    Primary sources: CoreWeave, Inc. Form 8-K filed September 17, 2026 (Items 7.01 and 8.01, disclosing the proposed convertible notes offering and the Equity Distribution Agreement); Exhibit 99.1 — CoreWeave Announces Proposed $3.0 Billion Convertible Senior Notes Offering; and Exhibit 99.2 — CoreWeave Investor Presentation, September 2026. Full benchmark tables are published by MLCommons at mlcommons.org.

  • The Inference Shift: Why AI’s Economics Are Moving From Training to Serving

    The Inference Shift: Why AI’s Economics Are Moving From Training to Serving

    On May 11, 2026, technology analyst Ben Thompson published an essay on his influential Stratechery newsletter titled “The Inference Shift,” arguing that the economic center of gravity in artificial intelligence is moving from training — the one-time, compute-intensive process of building a model — to inference, the ongoing work of running that model every time a user asks it a question.

    Thompson’s framing matters because Stratechery is widely read by technology executives and investors, and because the training-versus-inference balance directly shapes where the next wave of infrastructure spending — chips, data centers, power, and networks — actually lands.

    Executive Summary

    The essay’s core contention, as its title signals, is that the AI buildout’s defining workload is changing. Training a frontier model is a bounded project: enormous, but finite, concentrated in a handful of massive facilities run by a handful of well-capitalized labs. Inference is different in kind. It scales with usage — every chatbot session, coding assistant, and AI-powered search query consumes compute — so as AI products find real adoption, serving them becomes a continuous, growing operating cost rather than a one-time capital project.

    For infrastructure providers, that distinction is not academic. Training demand rewards maximum-density campuses wherever cheap power and land exist, with latency largely irrelevant. Inference demand rewards something closer to the traditional internet: capacity distributed nearer to users, resilient connectivity, and economics measured in cost per query rather than cost per training run.

    Because the full essay sits behind Stratechery’s subscription, this analysis works from the thesis itself — the shift from training to inference economics — rather than from the piece’s specific figures or examples, and examines what that shift would re-rank across the infrastructure landscape.

    Two Very Different Kinds of Compute Demand

    Training and inference stress infrastructure in almost opposite ways. Training jobs run for weeks or months across thousands of tightly interconnected accelerators, which pushes builders toward gigantic single-site campuses where power is cheap and abundant — remoteness is a feature, not a bug. Inference workloads are short, bursty, and user-facing: a response has to come back in a second or two, which puts a premium on proximity to population centers, redundancy, and network quality.

    The economics diverge just as sharply. Training is capital expenditure that a company chooses to make; it can be deferred, right-sized, or cancelled. Inference is tied to revenue-generating usage — if customers are querying your model, you must serve them, and your margins depend on how cheaply you can do it. A market organized around inference is one where efficiency per query, not raw peak capacity, becomes the competitive battleground.

    What Gets Re-Ranked in Infrastructure Demand

    If Thompson’s thesis holds, several categories of infrastructure move up the priority list. Metro and regional data centers — including colocation capacity near enterprise users — regain relevance after a period in which headlines were dominated by remote gigawatt-scale training campuses. Connectivity providers benefit, because distributed inference multiplies traffic between users, edge sites, and core facilities. Power demand becomes more geographically dispersed and steadier in profile, a different planning problem for utilities than a handful of enormous point loads.

    The chip layer re-ranks too. Training has been dominated by the most powerful general-purpose GPUs, where flexibility justifies premium pricing. Inference, being a more predictable and repetitive workload, is friendlier to specialized silicon and to cost-optimized accelerators — which is precisely why cloud providers have invested in custom inference chips and why competition at this layer is more open than in training hardware.

    Winners, Losers, and the Margin Question

    The clearest beneficiaries of an inference-led market are operators with distributed footprints, strong interconnection, and the ability to sell capacity in smaller, latency-sensitive increments — along with any vendor that reduces cost per query, from silicon designers to cooling and power-efficiency specialists. The more exposed parties are those whose plans assume training demand grows indefinitely on its current trajectory: single-tenant mega-campuses purpose-built for one lab’s training runs carry concentration risk if that lab’s training appetite plateaus while its serving needs move elsewhere.

    There is also a margin story embedded in the shift. When inference is the dominant cost, AI application companies face a squeeze between what users pay and what serving costs — which pressures them to negotiate hard with infrastructure suppliers, adopt cheaper hardware, and shrink models where quality allows. Infrastructure revenue may keep growing, but the pricing power within the stack could redistribute.

    Reasons for Caution

    The thesis has honest counterarguments, and they deserve equal scrutiny. Frontier labs continue to spend heavily on training, and newer techniques that make models “think longer” at answer time blur the line — they raise inference costs, supporting the thesis, but also keep demand for dense, training-class hardware high. It is also possible that both curves rise together, in which case “shift” overstates a rebalancing. And headline-level analysis of a subscription essay cannot verify which evidence Thompson marshals; readers should treat the thesis as a framework to test against disclosed capital-spending and usage data, not as settled fact.

    Background

    Stratechery, founded by Ben Thompson in 2013, is a subscription publication analyzing the strategy and economics of the technology industry, and it has been one of the more influential independent voices in debates over the AI buildout. The training-versus-inference question it takes up here has become central to that buildout: the industry’s first phase was defined by a race to train ever-larger foundation models, concentrating spending on top-end GPUs and massive single-site campuses.

    As AI products have moved from demos to daily tools, attention has turned to the cost of actually serving them at scale. Cloud providers have developed custom inference chips, model developers have released smaller and cheaper model variants, and newer ‘reasoning’ models that consume extra compute per answer have pushed inference costs up further — all of which forms the backdrop against which Thompson’s May 2026 essay lands.

    Source: The Inference Shift — Stratechery by Ben Thompson, an analytical essay published May 11, 2026, arguing that AI economics are moving from model training to inference.

  • Cerebras Puts Trillion-Parameter Kimi K2.6 in Front of Enterprises

    Cerebras Puts Trillion-Parameter Kimi K2.6 in Front of Enterprises

    Cerebras Systems announced on 6 May 2026 that it is making inference on Kimi K2.6 — a trillion-parameter-class large language model from Moonshot AI — available to enterprise customers on its wafer-scale hardware. The announcement positions Cerebras as a route for companies that want to run a frontier-scale open-weight model without assembling their own GPU fleet.

    The material available with the announcement is essentially the headline claim. Cerebras has not published, in the source reviewed here, the pricing, sustained throughput, context length, regional availability or capacity commitments that would let a buyer compare the offer directly against GPU-based inference providers.

    Executive Summary

    The substance of the news is straightforward: a specialist silicon vendor is putting a very large open-weight model in front of enterprise buyers on its own accelerators. The strategic question underneath it is larger. For most of the current AI build-out, the marginal dollar went into training — the one-time, capital-heavy process of creating a model. Spending is now shifting toward inference, the repeated act of running that model to answer requests, which behaves less like a construction project and more like a utility with a per-token meter attached.

    That shift changes which hardware properties matter. Training rewards raw arithmetic throughput across enormous clusters. Generating text one token at a time rewards something different: how fast a machine can move model weights to its compute units. Cerebras builds a processor the size of an entire silicon wafer and keeps weights in fast on-chip memory rather than in the off-chip high-bandwidth memory GPUs rely on, an architecture aimed squarely at that bottleneck.

    Whether that translates into better economics — not just faster demos — is unresolved by this announcement. Speed per token and cost per token are different metrics, and a trillion-parameter model stresses memory capacity in a way that cuts against wafer-scale’s main advantage. Enterprises evaluating the offer should treat it as a credible architectural bet that has not yet been priced in public.

    Inference Is Becoming the Data Center’s Recurring Bill

    Training a frontier model is a project: it has a start date, a budget and an end. Inference is an operating expense that scales with usage and never stops. As enterprises move AI features from pilots into products, the cost centre migrates from the training run to the serving fleet, and the buying criteria migrate with it — from peak cluster performance to cost per million tokens, tail latency and the ability to hold capacity when demand spikes.

    This matters for the reasoning and agentic workloads enterprises are now deploying. A model that thinks step by step before answering emits a long chain of intermediate tokens the user never sees. If generation runs at a modest rate, a query that produces thousands of hidden tokens becomes a wait measured in tens of seconds — which rules out interactive use. Token generation speed stops being a benchmark curiosity and becomes the difference between a product and a demo.

    That is the market Cerebras is aiming at, and it is a defensible one. It is also a narrower claim than it first appears: being fastest at generating tokens does not automatically mean being cheapest, because cost depends on how many concurrent requests a system can serve while staying fast. The announcement does not address that trade-off.

    The Wafer-Scale Bet: Bandwidth Over Everything Else

    Conventional accelerators are cut from a silicon wafer into many small chips, each paired with stacks of high-bandwidth memory (HBM) that hold the model’s weights. Every token generated requires reading those weights across that memory interface, so the interface, not the arithmetic units, usually sets the pace. Cerebras takes the opposite approach: it leaves the wafer whole, producing a single processor roughly the size of a dinner plate, and stores weights in memory distributed across the die itself. On-chip memory is dramatically faster to reach than off-chip memory, which is why the architecture has produced striking token-per-second figures on open models.

    The catch is capacity. On-chip memory is fast but comparatively scarce per unit of silicon, while HBM is slower but plentiful. A trillion-parameter model is precisely the case where that asymmetry bites, because all of the model’s weights must be resident somewhere before a request can be served. Serving one at wafer scale implies spreading the model across multiple systems and moving activations between them — which reintroduces exactly the kind of interconnect cost the architecture was designed to avoid.

    None of this makes the approach unworkable; Cerebras has run large models this way before, and mixture-of-experts designs help by activating only a fraction of parameters for any given token. But it means the headline claim — trillion-parameter inference — is where the engineering difficulty is concentrated, not where it is resolved. The disclosure that would settle the economics is how many systems constitute one serving instance, and the announcement does not provide it.

    An Open-Weight Model Changes the Procurement Conversation

    Kimi K2.6 comes from Moonshot AI, a Chinese lab whose K2 family has been released with open weights — the trained parameters are published, so anyone with sufficient hardware can run the model themselves. That property is what makes this announcement possible at all: a hardware vendor cannot offer a proprietary frontier model as a service, but it can offer an open one, and open weights have become the mechanism by which non-Nvidia silicon reaches enterprise buyers.

    For buyers, open weights cut in two directions. They reduce lock-in, because the same model can in principle be moved between providers or brought in-house, which makes a specialist accelerator less of a one-way door. They also shift the governance question from the model’s origin to the serving arrangement: where inference physically runs, who retains prompts and outputs, and what the licence permits commercially. A model developed in one jurisdiction and served on infrastructure in another is a common and legitimate arrangement, but it is one enterprise compliance teams will want documented rather than assumed.

    It is fair to note the competitive asymmetry this creates. Open releases from Chinese labs have given Western hardware challengers a supply of frontier-class models they would otherwise lack, while proprietary US models remain concentrated on GPU infrastructure. That is a genuine structural feature of the market, and it is worth stating without treating either the models or their provenance as inherently suspect.

    Winners, Losers and the Benchmark Problem

    If the offering performs as positioned, the clearest beneficiaries are enterprises with latency-sensitive AI products who currently face long queues for GPU capacity, and Cerebras itself, which has publicly disclosed heavy revenue concentration in a small number of customers and needs a broad enterprise base to diversify. Rival specialists pursuing similar high-speed inference strategies face more direct comparison. Incumbent GPU vendors are not meaningfully threatened by a single model launch, but they are affected by the general argument that inference and training may not want the same silicon.

    The losers, if any, are harder to identify from an announcement this thin. A serving offer is only as good as its capacity, and capacity is a function of how much wafer-scale hardware exists and is deployed — a supply constraint that specialist vendors have historically found harder to solve than performance.

    Buyers should also be alert to the benchmark problem. Tokens per second for a single request, cost per million tokens at realistic concurrency, and latency at the 99th percentile under load are three different numbers, and vendor materials across this entire market tend to lead with whichever is most flattering. That is not a criticism unique to Cerebras. It is the reason independent, workload-specific evaluation remains the only reliable basis for a purchasing decision here.

    Background

    Cerebras Systems, founded in 2016, took a contrarian approach to AI hardware: rather than dicing a silicon wafer into many chips, it manufactures a single processor spanning nearly the whole wafer, with memory and compute distributed across the surface. Successive generations of its Wafer Scale Engine have targeted first training and, more recently, high-speed inference sold as a cloud service. The company filed publicly to list its shares in 2024 and, in doing so, disclosed a heavy dependence on a small number of customers — a concentration that a broad enterprise inference business would help address.

    Moonshot AI is a Chinese AI lab whose Kimi K2 family arrived as one of the largest openly released model lines available, built as a mixture of experts — a design in which only a subset of the model’s parameters is activated for any given token, making very large models cheaper to run than their headline parameter count suggests. Open-weight releases of this kind have become the principal way that alternative accelerator vendors gain access to frontier-scale models, since proprietary models are generally tied to their developers’ own infrastructure.

    Source: Cerebras Brings Trillion Parameter Inference to Enterprises with Kimi K2.6 — Cerebras announcement dated 6 May 2026 making the trillion-parameter Kimi K2.6 model available to enterprise customers on its wafer-scale inference platform.