IBM, the century-old computing company, has signed a $240 million, multi-year deal with Together AI, a younger firm that runs freely available artificial intelligence programs for other businesses.
IBM’s cloud, meaning its rentable computers in data centers, will host a large set of powerful Nvidia chips set aside for one job: answering people’s questions to those programs, the everyday work of a chatbot.
The surprising part is how it is bought. Instead of paying per use, the computing power is booked years ahead, like leasing a whole building instead of renting hotel rooms by the night.
IBM and Together AI have signed a $240 million multi-year agreement for an Nvidia-powered AI inference cluster that will run on IBM Cloud, Reuters reported on August 11, 2026. Inference is the stage where a trained AI model answers requests, as opposed to training, where the model is built.
Together AI operates a cloud platform that serves open-weight AI models, meaning models whose parameters are publicly released, to developers and enterprises. Under the deal, a dedicated cluster of Nvidia accelerators on IBM’s infrastructure is committed to that inference work for several years.
Executive Summary
The headline number is $240 million, but the structure matters more. This is not a customer swiping a card for GPU hours. It is a multi-year commitment to a specific, dedicated cluster, which is how data center space and power have long been contracted, and increasingly how AI compute is contracted too.
For IBM, the deal places IBM Cloud in the business of hosting third-party AI inference at scale, a market where the large hyperscalers and specialist GPU clouds have been most visible. For Together AI, it secures a block of capacity for serving open models at a time when access to Nvidia hardware still depends heavily on who has committed to it in advance.
The companies have not disclosed the hardware generation, the size of the cluster, where it sits, or how the $240 million is paid over time. Those details determine whether this is a modest capacity reservation or a meaningful new source of inference supply.
Inference Is Being Booked Like a Lease
Most people meet AI inference as a metered service: a developer sends a prompt, a provider returns an answer, and the bill is calculated per token, a token being a small chunk of text. That pricing looks elastic, but the hardware underneath it is not. Nvidia accelerator clusters are expensive to buy, take time to deliver and install, and lose value as each new chip generation arrives. Someone has to commit to them before the first token is served.
A $240 million, multi-year contract for a dedicated cluster is that commitment made explicit. It shifts inference from the on-demand model of cloud computing toward the reserved-capacity model long familiar from colocation and wholesale data center leasing, where a tenant contracts a defined block of infrastructure for a defined term. The provider gets revenue visibility that justifies deploying hardware; the tenant gets guaranteed supply it can resell to its own customers with predictable performance.
This is one deal, and it does not prove an industry-wide shift on its own. But it shows clearly that open-model inference, often described as a commodity service racing to the lowest per-token price, is being underwritten by the same kind of long-dated capacity contracts as training clusters. The per-token price a developer sees sits on top of a fixed, multi-year obligation somewhere in the stack.
What IBM Gains by Hosting Someone Else’s Models
IBM has spent recent years positioning around open and hybrid AI: its Granite models are openly released, and its Red Hat business is built on open-source software. Hosting a platform whose entire proposition is serving open-weight models fits that posture more naturally than competing head-on with the largest clouds for frontier-lab training contracts.
It also gives IBM Cloud a substantial, recurring AI workload with an anchor tenant. For a cloud provider, a committed customer reduces the risk of buying accelerators that might otherwise sit underused. For enterprise buyers who already run on IBM infrastructure, the arrangement could, in principle, place open-model inference closer to their existing systems, though neither company has described how enterprise customers would access the cluster.
The Risk Sits in the Utilisation Curve
Reserved capacity moves risk; it does not remove it. Inference demand can shift quickly as new models arrive, and per-token prices for open models have been falling as providers compete. A multi-year commitment to a fixed pool of hardware exposes whichever party carries it to two pressures: whether the cluster stays busy, and whether the hardware stays competitive as Nvidia releases newer, more efficient generations.
There is also a physical side. Dense Nvidia clusters draw significant power and increasingly require liquid cooling, which constrains which data halls can host them and how quickly. The value of a contract like this depends on the cluster actually being energised and in service on schedule, and none of those milestones have been published.
Background
IBM has operated a public cloud for more than a decade and in recent years has concentrated its AI strategy on enterprise and hybrid deployments, including its watsonx platform, its openly released Granite models and the open-source software business of Red Hat. IBM Cloud offers Nvidia GPU infrastructure alongside its traditional enterprise services.
Together AI emerged during the generative AI boom as a platform for running open-weight models, giving developers an alternative to closed models from the largest AI labs. Like other inference providers, it depends on securing large volumes of Nvidia accelerators, whose supply has been shaped by long-term commitments from cloud providers and AI companies.
Businesses are on track to nearly double what they spend renting powerful computers for artificial intelligence this year, to about $42 billion, according to research firm Gartner.
The bigger change is what that money pays for. For the first time, more will go to running artificial intelligence tools every day — answering customers, handling tasks at work — than to building new ones.
Building happens in bursts. Running never stops. So the giant computer warehouses behind these tools increasingly have to stay busy, and powered, around the clock.
Gartner said in a forecast published August 10, 2026 that worldwide spending on AI-optimized infrastructure as a service (IaaS) — rented cloud computing built for artificial intelligence workloads — will grow 96.4% in 2026 to $42.3 billion, up from $21.5 billion in 2025, and reach $66.1 billion in 2027, a further 56.5% increase.
The research firm also forecast that in 2026 spending on inference (running trained AI models in production) will reach $23.3 billion and surpass spending on training (building those models) at $19 billion. Inference is projected to account for 55% of AI-optimized IaaS spending in 2026 and 59% in 2027.
Executive Summary
The headline number is the near-doubling of rented AI cloud spending, but the more consequential figure is the crossover: in Gartner’s forecast, 2026 is the year inference overtakes training as the larger use of AI-optimized cloud capacity. Hardeep Singh, Sr Principal Research Analyst at Gartner, attributes the shift to fine-tuned and domain-specific models being built into customer-facing and operational systems that require “continuous, real-time execution rather than periodic training.”
That distinction matters for anyone who builds, powers or connects AI infrastructure. Training demand arrives as large, time-bound projects; inference demand scales with everyday usage and persists. A market increasingly weighted toward inference behaves less like a series of construction jobs and more like a utility load — steady, growing with adoption, and tied to where users and applications are.
Gartner’s numbers also put the AI boom in proportion: AI-optimized services are forecast at 14.7% of total IaaS spending in 2026, and conventional cloud infrastructure keeps growing at roughly 20% a year alongside it. Data center operators face rising demand on both fronts, not a wholesale swap of one for the other.
Inference Overtakes Training, and Demand Stops Arriving in Bursts
Training is the process of building an AI model by feeding it vast amounts of data; inference is the work of using that finished model to answer a question, draft a document or complete a task. Gartner forecasts inference spending of $23.3 billion in 2026 against $19 billion for training, with inference’s share of AI-optimized IaaS rising from 55% in 2026 to 59% in 2027. Singh ties the shift to models moving into production systems that require “continuous, real-time execution rather than periodic training.”
The mechanism is the shape of the demand. A training run is a project: a large block of accelerators — the specialized chips, usually GPUs, that do AI math — is reserved, runs for weeks, and is then released. Inference consumption instead scales with use: every customer query and every step an AI agent takes draws compute. Gartner specifically flags agentic AI, software that carries out multistep tasks on its own, as amplifying compute intensity. Demand that tracks usage behaves like a utility load — it persists, rises with adoption and follows the rhythm of the businesses it serves.
That changes the planning problem for several groups. Cloud providers get fleets that are more consistently busy, which improves the economics of expensive accelerators. Data center operators and utilities see more load that draws close to its contracted power much of the time, rather than in project-driven peaks. And because inference feeding customer-facing systems has to reach those customers quickly, well-connected sites near users and networks tend to gain value relative to remote campuses chosen purely for cheap power.
The Percentages Are Slowing. The Dollars Are Not.
On growth rates alone, AI-optimized IaaS looks like it is cooling: 180% in 2025, 96.4% in 2026, 56.5% in 2027. But the absolute increments rise each year — about $20.7 billion of new annual spending in 2026 and roughly $23.9 billion in 2027. For people who build capacity, the increment is what counts: under Gartner’s forecast, each year requires more new AI-ready capacity than the year before, not less.
The forecast also keeps AI in proportion. AI-optimized services are about 9.7% of total IaaS spending in 2025, 14.7% in 2026 and 18.4% in 2027. They account for roughly a third of the market’s growth in each forecast year. The remainder — general-purpose rented infrastructure — still grows by about 22% in 2026 and about 20% in 2027, on Gartner’s figures. AI is not crowding conventional cloud out of these numbers; builders face demand for both high-density AI halls and standard cloud capacity, which have very different power and cooling requirements.
Rented AI Capacity Is Not the Same as Built Capacity
Gartner’s figures measure what customers pay to rent AI-optimized infrastructure, not what cloud providers spend building it or what enterprises spend on hardware for their own facilities. The forecast therefore shows strong and rising demand for rented AI capacity, but on its own it cannot show AI investment shifting away from ownership toward the cloud, because it does not measure the ownership side.
What it does indicate is where enterprise AI is being consumed as it moves into production. Singh says the shift toward deployment is “accelerating the cloud consumption patterns.” Rental spending is the revenue that providers weigh their construction against, and a forecast in which that revenue nearly doubles and tilts toward always-on inference strengthens the case for facilities designed for sustained, high-density load — heavier power delivery and the advanced cooling dense AI racks often require. The caveat is that this is a forecast: whether chips, power and finished data center space arrive fast enough to meet it is a separate question.
Background
Gartner, Inc. (NYSE: IT) is a Stamford, Connecticut-based research and advisory firm whose market spending forecasts are widely used by technology buyers, vendors and investors. Infrastructure as a service is the part of cloud computing in which customers rent raw computing, storage and networking from providers such as Amazon Web Services, Microsoft Azure and Google Cloud, paying for what they use instead of running their own hardware.
The generative AI boom created a distinct segment within that market: cloud capacity built around accelerator chips and the high-speed networking needed to train and run large language models. Gartner’s figures show that segment grew 180% in 2025 to about $21.5 billion, and the firm now expects spending to shift from building models toward running them in everyday business applications.
Anthropic, the company behind the Claude chatbot, is starting to design its own computer chips for running its AI. Samsung is reportedly in talks to build them.
Today most AI runs on chips from Nvidia, which are expensive and use a lot of electricity. Google, Amazon, Meta, Microsoft and OpenAI already make their own chips.
What most people miss is that these custom chips are built to answer people’s questions, not to create new AI models. Creating new models still mostly happens on Nvidia hardware. Nvidia’s rivals are winning ground on the everyday work of answering requests.
Anthropic is building an in-house chip development team to co-design custom ASICs (application-specific integrated circuits, chips built for one kind of job) to run its Claude models. The company told Business Insider it is co-designing the chips so that Claude can run faster and more efficiently at the scale its customers need. It is hiring engineers now, and the job listing stresses delivering the design on schedule. Tom’s Hardware reported the development on August 7, 2026. It added that The Information had reported in July that Anthropic was in talks with Samsung to manufacture the chips.
Anthropic has not named its design partner. No chip specifications, production volumes or deployment dates have been disclosed.
Executive Summary
Anthropic is the latest frontier AI developer to pursue its own silicon. Google, Amazon, Meta, Microsoft and OpenAI already have custom chip programs. The chips are aimed at inference, which is the work of running a trained model to answer requests. They are not aimed at training, the far more compute-intensive process of building the model in the first place.
That distinction matters more than the headline rivalry with Nvidia. Training advanced models still depends heavily on Nvidia GPUs. Inference can run on a much wider range of hardware, and its cost recurs with every prompt a customer sends. For a company whose usage is growing quickly, especially from agentic AI that consumes many more tokens per task than a single chat exchange, inference is the line item that scales. It is therefore the one worth engineering down.
The reported Samsung link adds a supply-chain dimension. Samsung is a small player in custom chip co-design, but it brings manufacturing capacity and memory. Memory is one of the scarcest inputs in AI hardware right now.
Why Nvidia’s Exposure Sits in Inference, Not Training
Every custom AI chip program disclosed so far aims first at running models, not building them. The economics explain why. Training a frontier model is an episodic, enormous job, and Nvidia’s GPUs and software ecosystem remain the default for it. Inference is continuous. Each question a user asks, and each step an AI agent takes, consumes compute. As usage grows, inference spending grows with it. Agentic workloads multiply the token count per task, so the growth is faster still.
Inference also tolerates specialization. A chip that only needs to execute a known family of models efficiently can drop much of a general-purpose GPU’s flexibility. In exchange it gains lower power draw and lower cost per request. Estimates put custom silicon’s total cost of ownership advantage at up to 65% compared with Nvidia GPUs. That figure is a ceiling across the industry, not a disclosed number for Anthropic’s chip, which has no published specifications. Anthropic’s own stated goal is narrower and consistent with this reasoning: making Claude run faster and more efficiently at customer scale.
For Nvidia, this means the pressure is concentrated in inference deployments. Each custom chip installed for inference displaces a GPU that would otherwise have done that work. Training, where Nvidia’s position is strongest, is not what these programs target. The large AI labs are also unlikely to become fully self-sufficient. Demand is growing too fast, and Nvidia retains strong control over access to the supply chain. The realistic outcome is mixed fleets, with GPUs for training and flexible workloads and in-house chips for high-volume serving.
The Real Cost Is the Software, and the Years
Designing a chip is only the visible part of the effort. A custom accelerator needs a full software stack, including compilers, runtimes and kernels, before it can run production models. The models themselves need tuning to exploit the hardware. That burden is why most companies sensibly buy general-purpose GPUs. Custom silicon only pays back at very large scale, where small per-request savings multiply across billions of requests.
The reference point is sobering. Google has built its Tensor Processing Units for 12 years, with Broadcom as co-design partner on each generation. Amazon runs parallel Trainium and Inferentia lines. Meta has announced several MTIA designs for deployment through 2027. Anthropic is at the hiring stage, and its first chip will arrive into a market where rivals are several generations in. The job listing’s emphasis on schedule suggests the company understands this, but no delivery target has been disclosed.
Samsung, Memory and a Concentrated Co-Design Market
Custom AI chips are rarely designed alone. Broadcom and Marvell account for roughly 95% of the ASIC co-design market. Broadcom works with Google, OpenAI and Meta, as well as ByteDance and Fujitsu. It claims a $73 billion order backlog and expects more than $100 billion in annual AI chip revenue by the end of 2027. Marvell supports Amazon’s Trainium and Microsoft’s Maia and is expected to earn upward of $11 billion from co-design work in 2026. Whichever partner Anthropic has chosen for design, that partner is serving a crowded order book.
Samsung’s reported role is manufacturing, and its appeal is not mainly design share. Samsung would bring fabrication capacity and chip design expertise. It would also bring access to memory, which is in short global supply. Modern AI accelerators depend on high-bandwidth memory (HBM) stacked alongside the processor. That memory, along with advanced packaging such as TSMC’s CoWoS, has been a practical limit on how many AI chips the industry can ship. A manufacturing partner that also makes memory could ease one of the tightest constraints a new chip program faces. Neither company has confirmed such an arrangement.
What Changes for Data Center Operators
For the facilities that host this hardware, custom inference chips add variety to planning. GPUs are power-hungry, and inference ASICs are built specifically to cut power per request. A fleet that mixes Nvidia GPUs with several in-house accelerators could bring different rack densities, cooling needs and networking layouts to the same campus. Operators serving the large AI labs should expect less standardization over the next few hardware cycles, not more.
Background
Anthropic develops the Claude family of AI models and sells access to consumers, businesses and government customers. It has grown quickly over the past year, and agentic AI, where models carry out multi-step tasks, has sharply increased the volume of computation each customer uses. Like other frontier labs, Anthropic runs its models on hardware from several suppliers, with Nvidia GPUs the industry’s default.
Custom AI silicon has become standard practice among the largest AI companies and cloud providers, often called hyperscalers because of the scale of their data center fleets. The market for designing these chips is dominated by Broadcom and Marvell. Much of the manufacturing and advanced packaging runs through TSMC, and memory supply has become a key constraint.
Chipmaker AMD is buying Taalas, a Toronto startup founded only in 2023. Taalas designs chips shaped around the artificial intelligence model they will run, instead of chips built to run anything.
Every time someone asks a chatbot a question, a computer in a data center does the work, and that happens constantly. Flexible, do-everything chips waste time and electricity on that job. AMD is betting that purpose-built chips can answer faster and use less power.
The price and the first products have not been announced.
AMD announced on August 6, 2026 that it has reached a definitive agreement to acquire Taalas, a Toronto-based company founded in 2023 that develops specialized silicon for AI inference, the work of running a trained model to answer requests. AMD said it plans to integrate Taalas’ technology into its accelerator roadmap and build system-level products that pair it with AMD Instinct GPUs.
Financial terms were not disclosed. The deal is subject to customary closing conditions and regulatory approvals, and AMD gave no expected closing date.
Executive Summary
AMD, one of two major suppliers of general-purpose GPUs for AI data centers, is buying a company whose stated purpose is to get around the limits of general-purpose hardware. AMD’s own announcement describes Taalas’ technology as reducing “compute and memory bottlenecks associated with general-purpose architectures.” Taalas CEO and co-founder Ljubisa Bajic described the company’s approach as “building the hardware around the model.”
The significance is directional. AMD is not abandoning GPUs. It frames Taalas as a complement to Instinct accelerators, EPYC CPUs, ROCm software and Helios rack-scale systems. But a leading GPU vendor is treating inference as a workload that justifies dedicated silicon on its own roadmap. That matters for anyone planning data center capacity, power budgets or AI hardware purchases over the next several years.
What the announcement does not yet provide is proof. There is no price, no performance figure behind the “breakthrough” claim, no named customer and no product timeline. The deal is best read as a strategic commitment whose payoff remains to be demonstrated.
A GPU Maker Buying Non-GPU Silicon Is the Signal
AMD’s flagship AI product, the Instinct GPU, is a general-purpose accelerator. It can train models, run them, and handle scientific computing. So it is notable that AMD’s release justifies this acquisition by pointing to the bottlenecks of general-purpose architectures. AMD says it will integrate Taalas’ technology into its accelerator roadmap and pair it with Instinct GPUs in system-level solutions. In other words, AMD is adding inference-specific hardware alongside the GPU rather than asking the GPU to do everything.
The mechanism behind that bet comes from how the two main AI workloads differ. Training teaches a model and happens periodically, at enormous scale, on hardware that benefits from flexibility. Inference runs the finished model every time a user sends a request. It is repetitive: the same model performs the same operations, over and over, at high volume. Inference is often limited less by raw arithmetic than by moving the model’s data between memory and the compute units, the “memory bottleneck” AMD’s release refers to. Taalas says it optimizes inference dataflows, the path data takes through the chip, for the model being served. That is what Bajic means by building the hardware around the model, an approach sometimes described as hard-wiring a model into silicon.
If the approach delivers, the people most affected are those who pay for inference at scale: cloud providers, AI service companies and the data center operators who host them. Inference runs continuously, so efficiency per answer compounds into power, cooling and floor-space requirements. More answers per unit of electricity would let operators serve more demand within a fixed grid connection, and grid connections are increasingly the constraint on new AI capacity.
The Price of Specialization Is Flexibility
Hardware tuned to a model is efficient for as long as that model is in use. AI models are revised and replaced quickly, while chips take years to design, validate and manufacture. A chip fitted too tightly to today’s model risks being stranded when customers move on. This likely explains why AMD positions Taalas as a complement rather than a replacement. Boppana’s framing, the “right compute solutions for every AI workload,” implies a division of labor. GPUs would handle training and fast-changing models, and specialized silicon would take stable, high-volume inference.
For infrastructure buyers, that points toward mixed fleets: racks combining different accelerator types, each doing the job it is best at. Mixed fleets complicate capacity planning, procurement and software. Workloads have to be routed to the right hardware, and utilization has to stay high across all of it. How Taalas’ technology is exposed through ROCm, AMD’s software platform for programming its accelerators, will largely decide whether customers can adopt it without rewriting their deployment stacks.
What AMD Is Actually Buying
Taalas was founded in 2023, making it roughly three years old at signing. The release names no Taalas product, customer or performance figure. Instead, it leans heavily on the “world-class engineering team” and on retaining and growing Canadian talent. That suggests AMD is acquiring a design approach and the people behind it as much as any finished product line. Acquiring engineering teams to accelerate a chip roadmap is a well-established move in semiconductors, and it is a reasonable way to enter a design space quickly.
Read even-handedly, the claims in the release fall into two groups. Some are substantiated by the fact of the deal itself: AMD has committed to acquiring the company and has stated where the technology will go. Others remain assertions until products ship, including “breakthrough inference performance and efficiency” and “significantly reducing” bottlenecks. The competitive logic is clear enough. AMD is building a full-stack AI platform of CPUs, GPUs, networking, software and rack-scale systems, and inference-specific silicon would fill a gap in that stack. Whether it closes that gap depends on execution the market cannot yet evaluate.
Background
AMD, based in Santa Clara, California, sells EPYC server CPUs and Instinct GPU accelerators for AI and high-performance computing. It also sells ROCm, its software platform for programming those accelerators, and Helios, a rack-scale system that packages them for data centers. The company has used acquisitions to broaden its data center portfolio before, most notably its purchase of programmable-chip maker Xilinx. It positions itself as a full-stack alternative in AI infrastructure.
Taalas was founded in 2023 in Toronto to design silicon specifically for AI inference. Inference is the stage where trained models serve real users, and AMD calls it one of the fastest-growing segments of the AI market. As inference volumes rise, the cost and power consumed per answer have become central concerns for cloud providers and data center operators. That has drawn interest in specialized chips designed to do that one job more efficiently than general-purpose hardware.
Chipmaker Marvell unveiled new parts aimed at a quiet problem in artificial intelligence: the expensive chips that run chatbots often sit idle, waiting for data.
When an assistant works through a long conversation or a multi-step task, it keeps a running memory of everything it has already read. That memory outgrows the chip. Marvell’s answer is a shared memory pool, reached by light over cables up to 50 meters long, that holds about as much as 32 laptops.
Marvell says this lets the same building produce two to three times more answers on the same electricity. That figure has not yet been tested by anyone else.
Marvell Technology announced on August 4, 2026, at the FMS 2026 memory and storage conference in Santa Clara, California, a set of products aimed at AI inference, the stage where trained models answer requests. The lineup spans three levels of the data center: the Bravera SC6 PCIe 6.0 SSD controller for individual servers, Structera X CXL memory expansion for racks, and Photonic Fabric memory modules, network cards and chiplets that form an optical shared-memory tier across multiple racks.
Marvell says Photonic Fabric reaches up to 50 meters, supports up to 32TB of warm KV cache offload and delivers up to 2-3x higher token throughput within existing footprints and power budgets. It says Bravera SC6 doubles the performance of its Bravera SC5 predecessor and is expected to begin sampling in Q4 2026. Marvell gave no availability dates for the Structera X or Photonic Fabric products.
Executive Summary
Marvell is presenting memory, not processors, as the next place where AI inference performance is won or lost. All three products in the announcement target the same object: the KV cache. This is the working memory a language model builds as it reads a prompt, and it grows with longer conversations and with more simultaneous users. The company argues that expanding, pooling and sharing memory separately from GPUs reduces the time those GPUs spend idle, which it calls stalls, and raises output per watt.
This matters to anyone who pays for AI infrastructure by the megawatt. If the claims hold, operators could get more usable work from GPU fleets they have already bought and powered, without adding floor space or grid capacity. The strongest number, the 2-3x throughput gain, is framed as an “up to” figure with no stated baseline or workload. Only one of the products, the SC6 controller, has a date, and that date is for sampling rather than volume shipment.
Three Tiers Aimed at One Problem
The announcement is best read as a memory hierarchy rather than three separate products. At the top sits high-bandwidth memory (HBM), the fast but scarce memory stacked beside each GPU. Below it, Structera X uses CXL (Compute Express Link, an open standard for attaching extra memory to processors over the same physical lanes as PCIe) to add and pool DRAM at the rack level. Photonic Fabric then extends a shared memory tier across racks over optical links of up to 50 meters. At the bottom, the Bravera SC6 controller makes SSDs fast enough that, in Marvell’s words, more KV cache can “move from high-bandwidth memory to SSD.”
Every tier has the same stated purpose: holding the KV cache somewhere cheaper and larger than HBM, while keeping it close enough that the GPU does not wait. Marvell’s framing, that the portfolio helps hyperscalers “scale memory more independently from compute,” is a statement about data center design. It describes memory as a resource provisioned at the rack and pod level, not a fixed amount soldered into each server.
Why KV Cache Placement Drives Throughput
The case for the headline rests on how inference works. When a model processes a prompt, it computes intermediate values for every token and stores them in the KV cache so it does not have to recompute them for each new word it generates. That cache grows with context length and with the number of concurrent sessions. Agentic AI, where a model runs multi-step tasks and carries long histories between steps, pushes both factors up at the same time. Once HBM fills, the operator has three options. It can evict the cache and recompute it later, which spends GPU cycles on repeated work. It can fetch the cache from storage, which is slow. Or it can fetch it from a faster intermediate tier.
Marvell’s throughput claim sits on that third option. The company says Photonic Fabric enables KV cache “to be loaded from a shared memory tier rather than storage,” offering up to 32TB of warm cache (recently used but not currently active) across multiple XPUs (Marvell’s umbrella term for GPUs and custom accelerators). From that it claims up to 2-3x higher token throughput within the same footprint and power envelope. The GPUs are unchanged. What changes is how often they sit idle waiting for context to be rebuilt or retrieved. Operators most affected are those running long-context, many-user inference on power-capped sites, where adding GPUs is not an option but raising their utilization is.
This supports a narrower claim than “memory has replaced GPUs as the bottleneck,” and the release itself does not go that far. It says memory is becoming “as critical as compute.” 650 Group analyst Alan Weckel, quoted in the release, calls memory capacity, bandwidth, latency and data movement “primary constraints.” The defensible reading is that for inference workloads with large KV caches, gains now come from memory placement as much as from raw compute.
What the Numbers Do and Do Not Establish
The concrete figures are the 32TB capacity, the 50-meter reach, the 2x performance step from SC5 to SC6, and the Q4 2026 sampling date for SC6. The 50-meter reach is significant for facility planners. It means the shared tier spans multiple racks, enough to cover a GPU pod, and it adds optical cabling inside the pod.
The 2-3x figure is the least specified. It carries an “up to” qualifier and no stated model size, context length, batch size or comparison system. Throughput gains from cache offload depend heavily on how often the cache is reused, so the figure is best read as an upper bound for favorable workloads, not a fleet-wide expectation. Likewise, “doubling the performance” of SC5 fits the generational jump from PCIe 5.0 to 6.0 but does not specify which metric doubles. “Extremely low latency” for the optical tier is not quantified at all.
Sourcing Flexibility and the Buyer’s Calculus
One of the more practical details concerns procurement. Bravera SC6 is NAND-agnostic, meaning it works with flash memory from multiple suppliers. For cloud providers, that lowers dependence on a single flash vendor, a real consideration in a market prone to supply swings. Structera X is pitched in similar terms, as a way to extend “the value of existing memory investments” rather than forcing wholesale replacement.
The trade-off is architectural complexity. Each tier adds devices, interconnects and software orchestration that must decide, in real time, where each session’s cache should live. The added power draw of CXL expanders, optical network cards and memory modules also counts against the power envelope the throughput gains are meant to protect. Buyers will need measured results on their own workloads before the net efficiency claim can be verified.
Background
Marvell Technology (NASDAQ: MRVL) is a Santa Clara-based semiconductor company focused on data infrastructure: chips that move, store, process and secure data for cloud, enterprise and carrier networks. It says it has served leading technology companies for over 30 years. Its Bravera line covers storage controllers, its Structera line covers CXL memory, and Photonic Fabric is its optical interconnect and memory technology.
As AI shifts from training models to serving them at scale, the memory attached to each GPU has become a practical limit. Models that handle long documents, long conversations or multi-step agent tasks build large KV caches that can exceed on-package high-bandwidth memory. Chipmakers and cloud providers are responding with tiered designs that spill that cache into pooled DRAM, shared memory and fast flash storage. FMS, the annual memory and storage conference, is a common venue for these announcements.
Two chipmakers, AMD and Cerebras, are teaming up so that a single chatbot answer is produced by both companies’ hardware working together.
When you ask an artificial intelligence assistant something, the computer does two jobs: it reads your question, then writes the reply word by word. AMD’s machines will do the reading. Cerebras’ dinner-plate-sized chip will do the writing.
The companies say the pairing could produce up to five times more output for the same electricity. That matters because power is one of the biggest limits on building the giant computer buildings behind these services. It is due later this year.
AMD and Cerebras Systems announced on July 23, 2026, at AMD’s Advancing AI 2026 event, a technical partnership that combines AMD’s Helios rack-scale systems, built on AMD Instinct GPUs, with the Cerebras Wafer-Scale Engine in a single disaggregated inference workflow. In the companies’ joint announcement, Helios handles prompt processing and large context windows, while the Wafer-Scale Engine handles token generation, the step that streams the answer back to the user.
The companies say the combined system is expected to deliver up to 5x higher tokens per second per watt. Cerebras plans to deploy Helios systems in its own data centers, with the joint solution expected to be available first through Cerebras Cloud in the second half of 2026.
Executive Summary
The announcement pairs two very different kinds of AI hardware inside one request. AMD’s GPU racks take the first, compute-heavy pass over a prompt. Cerebras’ wafer-scale processor then generates the output, a phase limited mainly by how fast data can be pulled from memory. The companies aim the product at the ultra-low-latency segment of inference: coding tools, real-time copilots, live agents and agentic workflows, where response time directly shapes how useful the system is.
It matters for three reasons. First, the headline efficiency figure is measured per watt, and electricity is increasingly the scarcest input for new AI capacity. Second, it makes disaggregated inference, the practice of running the two phases of a request on separate hardware, a cross-vendor design rather than something done inside one company’s GPU fleet. Third, it turns Cerebras into an AMD customer and operator of AMD systems, with the integration burden sitting in Cerebras’ data centers rather than with end buyers.
What is not yet established is the performance itself. The 5x figure is an ‘up to’ projection. The product is forward-looking, with availability targeted for the second half of 2026, and neither company has disclosed deployment scale or commercial terms.
The 5x Claim Rests on the Split, Not on Either Chip
Every response from a large language model is produced in two phases. Prefill reads the whole prompt in one parallel pass, whether that prompt is a question, a pasted contract or an entire codebase. That work suits hardware built for massive arithmetic throughput. Decode then produces the answer one token, roughly a word fragment, at a time. Each step has to re-read the model’s weights and the stored context of the conversation from memory, so decode speed is capped by memory bandwidth more than by raw compute. The AMD–Cerebras announcement describes exactly this division of labour. Helios processes ‘prompts and large context windows’, while the Wafer-Scale Engine ‘accelerates the memory-bandwidth-intensive token generation.’
That division is the mechanism behind the headline number. The companies say the two engines together are expected to deliver up to 5x higher tokens per second per watt. They present the gain as a property of the pairing, not of either system alone. A GPU rack running both phases spends much of decode waiting on memory. A wafer-scale system running both phases spends capacity on prefill work that GPUs batch efficiently. Assigning each phase to the hardware whose strength matches that phase’s bottleneck is how a combined system could outperform either one, provided the handoff between the two machines is fast and cheap enough.
The choice of metric matters for the people who build and power these facilities. Tokens per watt means output per unit of electricity, and grid power, more often than floor space, now caps how much AI capacity an operator can bring online. The words ‘up to’ mark the figure as a best case. Until AMD and Cerebras publish the configuration and baseline behind it, it is a design target rather than a measured operating result.
One Answer, Two Vendors’ Hardware: What Operators Inherit
Splitting prefill from decode is not new in itself; large inference providers have separated the phases across different pools of GPUs. The new element here is that every request crosses a vendor boundary. When Helios finishes prefill, the prompt’s working state, known as the key-value cache (the model’s stored record of everything it has just read), has to move to the Cerebras system before the first output token appears. That transfer sits directly on the latency path the product is being sold on, so the network link between the two systems effectively becomes part of the product.
It also puts two hardware platforms, two software stacks and two support relationships inside one serving pipeline. It means two distinct system designs, likely with different power densities and cooling requirements, sharing the same data hall. Scheduling is harder too, because prefill and decode capacity must be sized against each other: too few Helios racks starve the wafer-scale engines, and too few wafer-scale engines leave GPU output queued.
The launch model shields buyers from most of that. Cerebras plans to deploy Helios in its own data centers and offer the joint solution first through Cerebras Cloud in the second half of 2026, so customers would consume it as a service rather than integrate it themselves. Cerebras, not the end customer, absorbs the integration risk, along with the capital cost of buying, powering and hosting AMD racks.
What Each Company Gets Out of It
For AMD, the partnership extends Helios into the most latency-sensitive end of the inference market without AMD building a dedicated decode chip. Helios stays positioned, in the announcement’s words, as ‘the foundation for high-throughput and balanced inference workloads across the data center.’ It also makes Cerebras a buyer and operator of AMD systems, adding another deployment of Helios in a production cloud.
For Cerebras, the arrangement concentrates its wafer-scale hardware on the phase where it says it is strongest. Prompt processing, which grows heavier as context windows lengthen and agentic workflows chain many calls together, goes to GPU racks built for throughput. Andrew Feldman framed the benefit as reach: partnering with AMD lets Cerebras ‘bring that performance to even more customers.’ That fits Cerebras’ own risk disclosures in the announcement. It lists dependence on a small number of significant customers, including OpenAI, Group 42 Holding, Mohamed bin Zayed University of Artificial Intelligence and AWS, and the need to secure data center capacity and capital for its cloud offerings. The partnership broadens the product, but it also adds a capital-intensive deployment commitment.
For the wider market, this is one partnership, not proof of an industry shift. It does put a concrete test in front of buyers: whether heterogeneous inference, which matches different chips to different phases of a single request, delivers enough latency and energy gains to justify the added operational complexity. If Cerebras Cloud demonstrates those gains in production, similar cross-vendor pairings become more likely. If it does not, single-platform stacks keep their simplicity advantage.
Background
Cerebras Systems, listed on Nasdaq under the ticker CBRS, builds AI systems around its Wafer-Scale Engine, a processor made from a whole silicon wafer rather than many chips cut from it. The company sells hardware on premises and, increasingly, inference as a cloud service, and it markets itself on speed of token generation. Its disclosures name OpenAI, Group 42 Holding, Mohamed bin Zayed University of Artificial Intelligence and AWS as significant customers.
AMD is the main GPU alternative to Nvidia in AI data centers through its Instinct accelerator line, and Helios is its rack-scale system, which packages GPUs, processors and networking into a single deployable unit. The partnership reflects a wider industry move toward splitting inference into prefill and decode stages and optimising each separately, an approach previously used mainly within single-vendor GPU clusters.
A young company called Fireworks just raised $1.5 billion from investors. It doesn’t invent its own artificial intelligence. It runs freely available artificial intelligence programs for other businesses, using computer chips spread across more than a dozen cloud companies.
Its yearly sales pace reportedly jumped from $100 million to over $1 billion in 16 months. Shopify and Twilio took four to five years to make the same climb.
Why it matters: the money in artificial intelligence isn’t only going to chipmakers and tech giants. The middleman who keeps these systems fast and cheap is now big enough to fund on its own.
Fireworks, the training and inference platform for open-weight AI models, has raised a $1.5 billion Series D, according to Bessemer Venture Partners, which said on July 17, 2026 that it had joined the round as an existing investor. Bessemer said Fireworks has passed $1 billion in annual recurring revenue (ARR), up from $100 million 16 months earlier. It also said the company processes about 43 trillion tokens a day. Tokens are the word-fragments AI models read and write.
The company runs on more than a dozen cloud providers across over 20 regions. It is a first-party partner shipping natively inside Microsoft Foundry on Azure. It is led by CEO Lin Qiao, who led the Meta engineering organization that built the PyTorch framework, and by President George Hu, who joined in April 2026.
Executive Summary
Fireworks does not sell a chatbot or a proprietary model. It runs other people’s open-weight models, and the fine-tuned versions customers build on them, as fast and as cheaply as possible. It charges by the token. A $1.5 billion round for that business, at more than $1 billion in ARR, is a large bet that inference is a market of its own. Inference means running a trained model to answer requests, and on this bet it is neither a feature of the chip clouds nor of the model labs.
For infrastructure operators, the important detail is the architecture. Fireworks describes its platform as a virtual GPU cloud. It pools graphics processors from more than a dozen providers and sits between those providers and the enterprises consuming AI. The more demand flows through layers like this, the more GPU owners compete as suppliers to software platforms rather than selling directly to end customers.
The growth figures are striking. So far, though, they are headline numbers without the detail needed to judge durability. That detail includes margins, customer concentration, how much capacity is contracted, and how ARR is defined for a usage-based business.
A Platform That Sits Above the Chips
What Fireworks sells is not compute capacity in the usual sense. Its platform is described as a virtual GPU cloud. That is software pooling graphics processors (GPUs, the chips that do most AI work) from more than a dozen cloud providers in over 20 regions. On top sits a proprietary inference engine that decides how to run each model at maximum speed and efficiency. Customers buy output, measured in tokens, not servers. At roughly 43 trillion tokens a day and more than $1 billion in ARR, that software layer is now a business of real scale. The $1.5 billion Series D finances it as one.
That is the operational shift. In the familiar cloud model, whoever owned the machines owned the customer. In this model, GPU owners become suppliers. Those owners include hyperscalers (the largest cloud providers, such as Azure) and neoclouds (newer GPU-specialist clouds). The platform holds the customer relationship, decides where each workload runs, and keeps the margin from running it efficiently. Bessemer states the ambition directly: Fireworks wants to be the software layer that abstracts away inference running on “heterogeneous silicon, hyperscalers, neoclouds, and air-gapped environments” — the inference platform “running on every GPU in the world.”
For GPU cloud and data center operators, this cuts both ways. An aggregator pools demand from thousands of customers, which can make it a large and steady buyer of capacity. But demand routed through a multi-cloud layer is also portable: it can move toward whichever provider offers the best price, availability or location. The Microsoft relationship shows the layer can coexist with hyperscalers rather than simply competing with them. Azure distributes Fireworks natively through Foundry. Still, this is one company’s round. It shows the independent inference layer can be financed at scale, not yet that it will hold its margin once hyperscalers price their own inference services against it.
Open Models Move the Spend From the Model to the Serving
The investment thesis rests on a claim about models, not chips. Open-weight models, whose trained parameters are published for anyone to run, now match closed frontier models on the workloads most enterprises care about, at a fraction of the cost. On that premise, enterprises will increasingly post-train an open model on their own proprietary data instead of paying per call for a closed API. They would, in Bessemer’s phrase, stop wanting to “rent” their core intelligence.
If that premise holds, value shifts toward whoever serves and tunes those open models well. Fireworks’ pitch is that it does both in one place. Its platform combines fine-tuning and reinforcement learning (methods for adapting a model with examples and feedback) with production serving. Every interaction can feed the next version of a customer’s model. That combination also creates switching costs. A customer whose tuned model, data pipelines and serving setup all live on one platform has reasons to stay. That stickiness is what justifies valuing the layer as more than a GPU reseller.
The load-bearing premise, however, is asserted rather than demonstrated. No benchmarks or customer workloads accompany the claim that open models match frontier quality. The answer will vary by task. Buyers weighing an open-model strategy should test it on their own workloads rather than assume parity.
What the Growth Numbers Establish, and What They Don’t
Growth from $100 million to more than $1 billion in ARR in 16 months is, by Bessemer’s comparison, a climb that took Twilio and Shopify four to five years. It establishes that demand for hosted open-model inference is very large right now. For a usage-based business, however, ARR usually annualizes current consumption. It can fall if customers optimize their prompts, switch to smaller models, or move volume to another provider. That makes it a different quantity from contracted subscription revenue.
Revenue is also not margin. A platform built on capacity drawn from other clouds has to pay for that capacity. Its economics depend on how many tokens its engine can produce per GPU-hour relative to what it pays for that hour. The inference engine is the core asset for that reason: efficiency is the margin. The broader market figures cited alongside the round are context, not evidence about Fireworks. They include hyperscaler capital spending of more than $800 billion in 2026 and token consumption expected to rise more than 30-fold by the end of the decade, and no source is given for either projection.
Background
Fireworks was founded by engineers who built and scaled PyTorch at Meta, the open-source framework on which much of today’s AI software is built. CEO Lin Qiao led that engineering organization. The company positions itself as a training and inference platform for open-weight models. It combines a proprietary inference engine, a multi-cloud GPU pool and tools for fine-tuning and reinforcement learning. Bessemer Venture Partners was already an investor before the Series D.
The round lands as AI spending shifts from building models toward running them. Enterprises that first used closed frontier-model APIs are weighing open models they can tune on private data and serve more cheaply. A growing set of providers now competes to run those models: hyperscalers, GPU-specialist neoclouds and independent inference platforms.
According to a July 1, 2026 report by The Information, OpenAI has discovered a new technique to cut its inference costs — the cost of running trained AI models to answer user queries — roughly in half. The report, surfaced via Google News, offers few public technical details, but the headline claim alone is significant: inference is the dominant recurring expense of operating large AI services at scale.
Executive Summary
The Information reports that OpenAI has found a way to halve inference costs. Inference — the compute consumed every time a model generates a response — is distinct from training, the one-time (though enormous) cost of building a model. As AI products reach hundreds of millions of users, inference has become the larger and faster-growing line item, and the one that determines whether AI services can ever be sold profitably at mass-market prices.
If the reported claim holds across OpenAI’s production workloads, it matters far beyond one company. Inference cost per query is the denominator in nearly every AI business model, and it also drives how much data-center capacity, power, and silicon the industry believes it needs. A genuine 50% reduction would ripple through capacity forecasts, chip demand assumptions, and cloud pricing. What is publicly available so far, however, is a headline and attribution to a single outlet — the technique itself, its scope, and its verification remain undisclosed. Readers should treat the magnitude as reported, not confirmed.
Inference Is Where AI Economics Are Won or Lost
Training a frontier model is a capital project; serving it is an operating expense that scales with every user and every query. For a company operating at OpenAI’s scale, inference compute is widely understood to be the largest recurring cost of the business. That is why efficiency work — better model architectures, quantization (running models at lower numerical precision), caching, batching, and smarter routing of queries to smaller models — has become as strategically important as raw capability gains.
A 50% cost reduction, if real and durable, changes the unit economics of every product built on the platform. Features that were too expensive to offer free users become viable. Margins on paid tiers widen, or prices fall to win share. Either way, the historical pattern in computing is consistent: when the cost of a unit of compute drops, providers do not pocket the savings for long — competition passes them through.
Cheaper Inference Rarely Means Less Infrastructure
A natural first reading is that halving inference costs halves the data-center capacity AI requires. History argues the opposite. This is the Jevons paradox — the economic observation, dating to 19th-century coal markets, that efficiency gains tend to increase total consumption of a resource, because lower cost unlocks new demand. Cheaper inference makes it economical to embed AI in more products, run longer reasoning chains, serve more users, and process more modalities like video and voice.
For data-center operators, connectivity providers, and power planners, the practical takeaway is that efficiency breakthroughs shift the composition of demand more than they shrink it. Inference-optimized capacity — which prizes power efficiency, proximity to users, and network performance over the raw density of training clusters — becomes relatively more valuable. Announcements like this one strengthen, rather than undercut, the case for distributed inference-serving footprints.
Winners, Losers, and the Silicon Question
Who benefits depends on what the technique actually is, which the public reporting does not say. A software-level advance (better serving algorithms, sparsity, or distillation) would be broadly replicable and would compress costs industry-wide over time — good for AI application builders and enterprise buyers, more ambiguous for chipmakers whose demand forecasts assume ever-growing compute per query. A hardware-dependent advance tied to specific accelerators would instead concentrate advantage in whoever controls that silicon.
For competitors — Anthropic, Google, Meta, and open-model providers — the report raises the efficiency bar. Inference cost per token has become a headline competitive metric alongside benchmark scores. For enterprise buyers, the sensible posture is patience: if the largest AI provider has found a way to halve its serving costs, downstream API price reductions have historically followed within quarters, and procurement teams negotiating long-term AI contracts should factor that trajectory in.
Background
OpenAI, founded in 2015 and best known for ChatGPT, operates one of the largest AI services in the world and has been a primary driver of the surge in demand for GPUs, data-center capacity, and power since 2023. The company’s spending on compute — for both training new models and serving existing ones — is central to debates about AI economics, because analysts have long questioned whether revenue from AI products can outpace the cost of delivering them.
Efficiency work is not new: the industry has steadily driven down cost per token through techniques like quantization, distillation, and better serving software, while The Information has built a track record of detailed reporting on OpenAI’s internal finances. What makes this report notable is the claimed magnitude — a one-time halving, rather than incremental gains — arriving amid historically large infrastructure commitments across the AI sector.
Etched, a startup building chips specialized for AI inference, has emerged from stealth with $800 million in funding and unveiled a working chip, according to a June 30, 2026 report by Data Center Dynamics. The announcement positions the company as one of the best-capitalized challengers to general-purpose GPUs in the fast-growing market for running — rather than training — AI models.
Executive Summary
The headline facts are two: a very large capital raise, and functional silicon. In the chip industry those milestones matter in combination. Hundreds of startups have raised money on architectural promises; far fewer have demonstrated a working chip, the point at which a design has survived the multi-year, multi-hundred-million-dollar gauntlet of tape-out and fabrication. An $800 million round — among the largest ever disclosed for an AI chip startup — signals that investors believe Etched has cleared that bar.
Why it matters: the economics of AI are shifting from training (building models) to inference (serving them to users), which recurs with every query and now dominates many operators’ compute bills. Etched’s core thesis, articulated publicly since 2024, is that a chip hard-wired for the transformer architecture underlying today’s large language models can deliver dramatically better throughput per dollar and per watt than a flexible GPU. If that holds in production, it pressures the pricing of incumbent accelerators and reshapes data center power and cooling planning. The release, as reported, does not yet prove it holds.
Inference Is Where the Money Now Flows
Training a frontier AI model is a one-time (if enormous) expense; inference — actually answering user queries — is a cost incurred billions of times a day, forever. As AI products reach mass adoption, inference has become the dominant and recurring line item in operators’ compute budgets, and every percentage point of efficiency compounds. That is the market Etched is aiming at, and it explains investor appetite: a supplier that meaningfully cuts the cost per generated token addresses one of the largest and fastest-growing spend categories in technology.
It also explains the timing. GPU supply has been constrained and expensive throughout the AI boom, and the power those GPUs draw has become the binding constraint on data center construction. Any credible chip that promises more inference per megawatt speaks directly to the industry’s scarcest resource.
The Specialization Bet: What an ASIC Gains and Risks
Etched builds what the industry calls an ASIC — an application-specific integrated circuit. Where a GPU is a general-purpose parallel processor that can run almost any AI architecture, Etched’s design bakes the transformer architecture directly into the silicon, spending its transistor budget on exactly one workload. The company has previously claimed this yields order-of-magnitude gains in throughput. The gain is real in principle — specialization has repeatedly beaten generality in mature workloads, from Bitcoin mining to video encoding — but it carries a matching risk: if the dominant model architecture shifts away from transformers, a transformer-only chip has nowhere to go, while a GPU simply runs the new thing.
Etched’s implicit wager is that transformers are now infrastructure, stable enough to hard-wire. Several years into the transformer era, with every major frontier model still built on the architecture, that wager looks stronger than it did at the company’s founding. But it remains a wager, and buyers weighing multi-year deployments will price that architectural lock-in accordingly.
$800 Million Buys Credibility, Not Victory
Leading-edge chip development routinely consumes hundreds of millions of dollars per generation before a single unit ships in volume, which is why the AI accelerator field has narrowed to companies with either deep pockets or hyperscaler patrons. An $800 million round puts Etched in rare company among independents and funds the unglamorous phase ahead: yield ramp, volume manufacturing, server integration, and — critically — software. Nvidia’s real moat is less its silicon than CUDA, the software ecosystem that millions of developers already use. Every challenger, from Groq to Cerebras to the hyperscalers’ in-house chips, has learned that a fast chip without a mature software stack and cloud availability wins benchmarks but not budgets.
One framing note deserves scrutiny: Etched has not been literally unknown — the company publicly announced a $120 million Series A in mid-2024 and marketed its Sohu chip concept openly. The ‘stealth’ language in the reported headline most plausibly refers to the silence surrounding its silicon progress since then. That distinction matters, because the genuinely new, load-bearing claim here is the working chip — and as reported, it arrives without published benchmarks, customer names, or availability dates.
What It Means for Data Center Operators and Buyers
For data center operators, credible inference ASICs change capacity math. Higher throughput per watt means more revenue-generating tokens per megawatt of grid connection — the metric that increasingly governs siting and construction decisions. For enterprise buyers, a well-funded second source of inference compute is leverage in GPU negotiations even before a single Etched server ships. The practical near-term effect of announcements like this one is often pricing pressure on incumbents rather than immediate displacement; displacement requires the proof points this release does not yet contain.
Background
Etched was founded in 2022 by a group of Harvard dropouts and stepped into public view in June 2024 with a $120 million Series A and an audacious pitch: its Sohu chip would abandon GPU-style flexibility and etch the transformer architecture — the mathematical structure behind essentially all modern large language models — directly into silicon, claiming order-of-magnitude throughput gains over contemporary GPUs. At the time the company had no working chip, and skeptics noted both the architectural lock-in risk and the graveyard of past AI chip challengers.
The intervening two years transformed the market it targets. Inference spending overtook training as the growth engine of AI compute, power availability became the industry’s defining constraint, and hyperscalers validated the specialization thesis by pouring billions into their own custom inference silicon. Etched’s reported $800 million raise and working chip land in that context: a market actively searching for alternatives to GPU economics, but one that has also repeatedly shown how hard it is to convert a fast chip into a shipping business.
AI inference platform Baseten is nearing a funding round of roughly $1.5 billion, according to a June 19, 2026 report from PYMNTS. The report ties the raise directly to surging demand for inference — the work of running trained AI models in production — rather than for model training.
Terms, investors, and valuation were not detailed in the headline-level report, and the round had not been confirmed as closed at publication time.
Executive Summary
According to the report, Baseten — a company that helps businesses deploy and serve AI models at scale — is close to raising approximately $1.5 billion in new capital. For a company that was a mid-sized startup only two years earlier, a raise of this magnitude would rank among the largest ever for a dedicated inference provider.
The significance is less about one company than about where AI infrastructure money is now flowing. For the first few years of the generative-AI boom, capital chased training: the enormous one-time compute jobs that create frontier models. A $1.5 billion round for an inference specialist signals that investors now see the recurring, usage-driven business of serving models to end users as the larger and more durable prize.
That said, the source is thin. A single report of a round that is ‘near’ closing establishes investor intent and market temperature, but not final terms, valuation, or how the money will be spent. Those distinctions matter for anyone reading this as a market signal.
Inference Becomes the Center of Gravity
Training a large AI model is a one-time capital event; inference is a bill that arrives every time anyone uses the model. As AI applications have moved from demos into daily production use, the aggregate compute spent answering queries has grown continuously, while training runs remain episodic and concentrated among a handful of frontier labs. A near-$1.5 billion bet on an inference specialist is a bet that this recurring workload — not the headline-grabbing training runs — is where sustained revenue accumulates.
This inversion matters for the whole infrastructure stack. Training clusters favor a few gigantic, tightly coupled GPU installations. Inference favors distributed capacity closer to users, high utilization, and relentless cost-per-token optimization. If the money is following inference, demand patterns for data center capacity, networking, and power will follow it too.
Why Inference Platforms Command This Kind of Capital
Inference sounds simple — run the model, return the answer — but doing it profitably at scale is an engineering discipline of its own: batching requests, compiling models to specific chips, autoscaling against spiky traffic, and squeezing latency low enough for real-time products. Companies like Baseten sell that discipline as a service, sitting between raw GPU suppliers and application builders who don’t want to run their own model-serving operation.
The catch is that the business is capital-hungry in both directions. Serving customers requires reserving expensive GPU capacity ahead of demand, and competing on price requires continuous optimization investment. A $1.5 billion war chest, if the round closes as reported, is plausibly less about runway than about locking up compute supply and engineering talent before rivals do.
Winners, Losers, and the Squeeze in the Middle
The clearest beneficiaries of an inference-led cycle are the layers underneath: GPU vendors, specialized AI clouds, and the data center and power providers that host distributed serving capacity. The most exposed parties are undifferentiated middlemen — inference is a market where hyperscalers (Amazon, Google, Microsoft), well-funded independents, and open-source serving stacks all compete, and per-token prices have fallen steadily across the industry.
That competitive pressure cuts both ways for Baseten. A massive raise validates the category but also raises the stakes: the company would need to convert capital into durable advantages — proprietary optimizations, enterprise trust, sticky deployments — faster than falling inference prices erode margins. Investors appear to be betting that scale itself becomes the moat. That thesis is credible but unproven, and the report offers no revenue or margin data to test it against.
Background
Baseten was founded in 2019 in San Francisco, initially building tools that let software teams deploy machine-learning models without specialized infrastructure staff. The generative-AI boom transformed that niche into one of the industry’s fastest-growing markets, and the company raised successive venture rounds through 2025 that reportedly pushed its valuation past $2 billion.
The broader market context is a widely discussed shift in AI economics: as chatbots, coding assistants, and AI-powered products moved into everyday production use, industry attention moved from training models to serving them. Inference specialists — alongside GPU clouds and the data center operators beneath them — became prime beneficiaries of that shift, setting the stage for the mega-round reported here.
