TL;DR · 30-second read
The Short Version
A French startup called Kog claims software alone can make artificial intelligence respond 30 times faster, using chips companies already own.
Its demonstration produced about 3,000 small chunks of text every second. The catch: it used a deliberately small system, and customers want much bigger ones.
This matters because these chips are scarce and use a lot of electricity, and people relying on such tools at work can wait hours for results. If the approach works on bigger systems, existing equipment could become far more useful.
TechCrunch reported on August 14, 2026 that French startup Kog is developing software, the Kog Inference Engine (KIE), to run AI models faster on conventional data center GPUs, the graphics processors that do most AI computing. A May tech preview on AMD MI300X and Nvidia H200 GPUs reached 3,000 tokens per second for a single request, using a purpose-built model of about 2 billion parameters called Laneformer 2B, which is now open source. CEO and solo founder Gaël Delalleau said the preview produced “200 tangible business leads.”
Delalleau expects software engineering to be the first use case, aimed at professionals frustrated by delays that can stretch to hours for Claude Code users. Kog is now focused on accelerating larger models, because prospective customers are not prepared to fine-tune small ones.
Executive Summary
Kog is promising “30x faster LLM inference” through software on GPUs that enterprises already operate. Inference is the stage when a trained AI model generates answers, and an LLM, or large language model, is the type of system behind tools such as chatbots and coding assistants. The appeal is direct: if software can deliver large speed gains, buyers may get more value from hardware they have already paid for.
The evidence so far is narrower than the promise. Kog’s 3,000 tokens-per-second result came from a small model built for the purpose, while the demand Kog has encountered points to larger models. That gap is the central issue for anyone evaluating the technology.
The story also matters beyond one startup. Cerebras, which sells purpose-built AI chips, received a warm market reception at its May IPO. Kog represents a different bet: that conventional GPUs still have substantial untapped performance, and that software can release it.
Model Size Is Where the Speed Claim Gets Tested
Kog’s headline figure is 3,000 tokens per second for a single request. A token is a small chunk of text, often a word or part of one, so the number describes very fast output. But it came from Laneformer 2B, a purpose-built model with about 2 billion parameters, the numerical values a model uses to turn an input into an output.
The mechanism behind the challenge is straightforward. Generating each token requires computation across a model’s parameters, and larger models generally demand more arithmetic and more data movement through memory at every step. A result achieved on a small, purpose-built model therefore cannot simply be projected onto a much larger one. Kog has also found that its prospective customers are not prepared to fine-tune small models, a finding that moved the company’s priorities. “That’s why since the launch, we’ve been fully focused on accelerating the development of larger models to meet the demand we’ve seen,” Delalleau said.
This affects both sides of a potential deal. Enterprises that own GPUs need to know the speed of the model they would actually deploy, not the speed of a demonstration model. For Kog, the next benchmark will say more about the “30x” promise than the first one did, because it would test the engine under the conditions buyers have asked for.
Faster Requests Do Not Automatically Mean More Capacity
Software efficiency is an attractive answer to scarce, power-hungry AI compute. Extracting more from installed GPUs would, in principle, ease pressure without new chips, racks or electrical capacity. That is the larger question Kog’s work raises: how far can software stretch hardware that is already deployed?
Kog’s demonstrated claim, however, concerns single-request speed. Data center operators also care about total output across many simultaneous users. The two measures can pull apart, because serving systems often group requests together to raise overall throughput, which can slow any individual response. An engine could give one user striking speed while total tokens per GPU, or per unit of electricity, rise modestly, stay flat or even fall.
Fast responses have real value of their own, particularly when a person or an automated workflow is waiting. But shorter waits and capacity relief are different benefits with different buyers. The case for stretching existing compute requires a second set of measurements: aggregate throughput and energy per token, at realistic numbers of concurrent users.
A Software Bet Beside a Silicon Bet
The contrast with Cerebras frames the competitive question. Purpose-built chips pursue speed through new hardware, which buyers must acquire and integrate. Kog’s pitch is that “extremely fast single-request decoding is possible on the standard datacenter GPUs enterprises already own.” For organizations with GPUs already installed, software could be an easier decision than a hardware change, if the performance holds on the models they need.
Kog’s use of both AMD and Nvidia hardware in its demo is also notable. If comparable results carry over to larger models on both, buyers could gain speed without tying that improvement to a single chip supplier. That remains a conditional benefit until larger-model results are available.
The two approaches need not be mutually exclusive, and one preview does not settle the contest. If Kog delivers at larger scale, GPU owners stand to benefit most. If speed falls sharply as models grow, the argument for specialized inference hardware becomes stronger.
Demand Signals Are Real but Early
There are signs that speed has commercial value. Anthropic charges a price multiple for Claude’s Fast Mode, which shows that at least one major AI provider treats speed as something customers will pay extra for. Kog also has design partners whose users generate games and apps from a prompt, and Delalleau said faster results through KIE would mean more revenue for them.
The 200 leads show interest, and Delalleau’s description of them as “tangible” suggests more than casual curiosity. Still, a lead is an early step rather than a signed contract or recurring revenue. Kog itself recognizes that this market is not yet mature, which is a useful caution when weighing the size of the near-term opportunity.
Background
AI inference has become a critical bottleneck on both speed and cost, as more software depends on large models that must respond quickly. GPUs are the workhorse hardware for this computing, but they are scarce and power-hungry, so there is strong interest in getting more from them. Cerebras has taken the hardware route with purpose-built chips, and markets welcomed its IPO debut in May 2026.
Kog drew attention the same month when its tech preview reached the front page of Hacker News, a widely read technology forum. The preview aimed to show that extremely fast single-request decoding, the process of generating a model’s output, is possible on data center GPUs that enterprises already own. Source: Kog is going deeper to squeeze more inference out of GPUs, TechCrunch’s August 14, 2026 report on the French startup’s effort to speed AI inference on conventional GPUs.Sources