Nvidia Groq 3 LPX: 3,400 Tokens/Second & Agency Costs

One week after OpenAI published the first results for its Jalapeño inference chip, Nvidia answered with hardware of its own. At Hot Chips 2026, Nvidia announced that Groq 3 LPX — an inference accelerator built on the Groq architecture it licensed in December 2025 — is now in full production, and it published its first third-party inference benchmark to back the claim. It is the Nvidia side of the story we covered with OpenAI's Jalapeño chip: token generation, not training, is where the next wave of AI inference economics gets decided.

What happened: full production at Hot Chips 2026

Nvidia's release is direct: "NVIDIA today announced that NVIDIA Groq 3 LPX, the interactive AI inference accelerator, is now in full production." Groq 3 LPX is an extension of the NVIDIA Vera Rubin platform — the rack-scale line that succeeds Grace Blackwell — designed to dramatically increase token generation rates on Vera Rubin NVL72 systems. The target workload is agentic AI, where agents "generate massive volumes of tokens across hundreds or thousands of inference steps" and generation speed determines how quickly they can reason, inspect files, call tools, and iterate. Jensen Huang framed it as the next stage of the inference strategy: "Inference is the growth engine of AI... advancing the performance frontier with LPX for ultrafast token generation."

The $20 billion backstory: Groq's architecture becomes Nvidia silicon

The most striking part is provenance. The chip Nvidia presented was designed by the team that used to compete with it: Igor Arsovski — Groq's former chief architect, now Nvidia's VP of hardware — presented the LPX rack's architecture on stage, calling it "a pinch me moment for the Groq team that's now integrated into the Nvidia group." The lineage runs through the Tensor Streaming Processor that Groq — founded by ex-Google TPU engineer Jonathan Ross — described in its 2020 ISCA paper "Think Fast." In December 2025, Nvidia paid Groq Inc. $20 billion for a non-exclusive IP license, hiring Ross, president Sunny Madra, and most of Groq's engineers in a structure that avoided a formal merger review. The deal drew FTC scrutiny in early 2026 — Senators Warren and Blumenthal argued it acquired Groq "in all but name" — and Nvidia pulled its GDDR7-based Rubin CPX accelerator off the roadmap to focus on the LPU this year.

One naming note for agencies: Groq 3 LPX is Nvidia silicon, not Groq Cloud. The startup's own AI cloud now plans to be among the earliest adopters of Nvidia's LPX rack; the chip is unrelated to xAI's Grok model entirely.

Groq 3 LPX at a glance

Why SRAM-only decode is fast

The architecture is a deliberate trade. Each LP30 carries roughly 500MB of on-die SRAM and no HBM, so a full 256-chip rack holds 128GB — compared with 288GB of HBM4 on a single Rubin GPU. What the design lacks in capacity it makes up in latency: keeping weights resident in SRAM removes the memory-access latency that dominates single-token decode. That is why the LPU is pitched as a decode accelerator, not a general-purpose GPU replacement. Three engineering choices drive the speed:

• Fully deterministic pipeline. The design drops caches, branch prediction, and out-of-order execution for a pipeline the compiler schedules at clock-cycle granularity — cutting voltage droop by more than 60% and overshoot by more than 70%.
• Thermal equalization. Per-block scheduling equalizes heat instead of throttling to the hottest tile — roughly 10–11% more performance under a fixed thermal limit.
• A plesiosynchronous network. Chips synchronize to a single virtual clock, each acting as both processor and router — no adaptive routing or congestion sensing needed.

The first third-party benchmark: 3,431 tokens per second — with caveats

This is the first third-party inference benchmark Nvidia has published for the hardware. Artificial Analysis measured 3,431 output tokens per second on a 100K-context Gemma 4 31B reasoning workload — roughly four times the 870 tokens per second of the next-fastest public endpoint in the same benchmark family. Nvidia's release rounds it to 3,400, calls it a record for the model, and claims "4x faster responsiveness... than the nearest alternative platform," with coding tasks completing "in minutes versus hours."

Read the caveats before you build quotes on it:

• It is a private, single-tenant endpoint. Artificial Analysis ran the comparison on a private, pre-release Gemma 4 31B endpoint served through Google Cloud — median of 50 sequential requests at a concurrency of one — against public providers on shared production serverless endpoints. That is the highest per-user rate the hardware can post, not a multi-tenant guarantee.
• The demo number is bigger and self-reported. Nvidia's on-stage demo showed 10,996 tokens per second — flagged by Arsovski himself as "self-reported" before he said the goal was "third-party verified independent benchmarks that you guys can trust."
• Gemma 4 31B is a small, dense model that fits inside a single LPX rack; the trillion-parameter MoE picture, where memory capacity is the main constraint, went unaddressed.

None of this makes the number meaningless — a ~4x decode lead over the fastest public endpoint is where the market has been heading. But it is a hardware ceiling under ideal conditions, not a service-level guarantee you can price against yet.

How LPX splits inference with Vera Rubin

Nvidia is selling LPX as a co-processor, not a standalone GPU replacement: Rubin GPUs handle the compute-heavy prefill phase and build the KV cache, while the LPUs generate output tokens, in three integration modes: disaggregated prefill and decode; attention–FFN disaggregation (attention and its cache stay on GPU HBM while the LPU runs the feed-forward layers); and external-draft speculative decoding (a small model on the LPU proposes tokens that the GPU verifies in parallel). An FPGA bridges the synchronous LPU domain and host I/O, and Nvidia's Dynamo runtime — with an LPU extension to CUDA — orchestrates the split. Nvidia put the gains at roughly three-to-five times over Rubin alone on a two-trillion-parameter workload with a 400K cached context — Nvidia-measured, not third-party.

Who's buying first: Nebius, Groq, and the AI-cloud roadmap

Nvidia named its first customer at the announcement. Nebius is the first AI cloud to adopt Groq 3 LPX, planning to bring it to Nebius Token Factory with a commitment to having it online before December 31, 2026. CTO Danila Shtan put it in the terms agencies care about: "we're making sure every step of an agent's loop feels instant — through the same API developers are already using, with no migration to a new stack." Following Nebius, the purpose-built AI inference cloud Groq plans to be among the earliest adopters. The same Hot Chips session revealed SpaceX as the latest flagship Vera Rubin customer, and Cerebras presented its CS4 wafer-scale system — with a July agreement to pair AMD Helios GPUs for prefill, the same division of labor Nvidia now builds in-house.

The pricing gap: hardware in production, tokens not yet for sale

The part that matters most for your quotes: Groq 3 LPX is in full production, but no one can buy a token of it today. Nvidia has not announced an API date, and Nebius's public pricing does not identify LPX capacity or a separate rate card. The only committed date is Nebius's "before December 31, 2026." When pricing does land, hold these expectations:

• Expect a speed premium, not a discount. SRAM-only decode is expensive per byte of capacity — a 31B model needs ~62 LPUs, and large MoE models need thousands of chips across racks. Speed tiers have historically priced far above commodity rates (Cerebras's ultrafast tier has served GPT-5.6 Sol at roughly 14x the standard rate); expect LPX to follow that pattern.
• Speed changes product design, not token count. A 4x-faster decode does not reduce the tokens an agent consumes; it reduces wall-clock time per loop iteration — a latency lever for what you can build, not a cost lever. The cost lever remains model routing (see our AI agent workload routing guide on Find AI Agency).
• The first published rate card is the trigger. When Nebius Token Factory or Groq Cloud publishes LPX pricing, a speed tier enters your calculator — a per-token premium for output-token-heavy, latency-sensitive agent tasks only.

What this changes in the calculator

We have not added a Groq 3 LPX model row to the AI Agency Pricing Calculator — there is no published price to model, and inventing one would be worse than waiting. What changes is the trend assumption underneath your existing rows:

• The "inference cost keeps falling" story now has both sides. OpenAI's Jalapeño (Aug 25) showed the efficiency case; Nvidia's Groq 3 LPX (Aug 24) showed the merchant-silicon case with a third-party benchmark. Re-baseline token-cost assumptions quarterly, not annually.
• The speed tier is coming; budget for it as premium. A "3,400 tokens/sec" endpoint will not be priced like DeepSeek. Plan client quotes around two tiers: commodity long-context models for most agent work, premium speed tiers for the interactive front.
• Watch Nebius Token Factory before Dec 31, 2026 — the first price-discovery event for LPX-class decode. The AI Agent API Cost Calculator lets you model the workload shapes that would justify a speed tier before the rate card exists.

Practical takeaways

• Hardware is real; pricing is not. Groq 3 LPX is in full production with a third-party 3,431 tok/s benchmark, but no API price exists yet — do not quote it, do not ignore it.
• Pair the Nvidia and OpenAI stories for clients. Jalapeño (efficiency) plus Groq 3 LPX (decode speed) is a coherent "token costs keep falling" narrative — position it as margin you reinvest in scope.
• Speed premium is the likely outcome. SRAM-only capacity costs mean LPX tokens will launch priced like Cerebras-style speed tiers, not commodity rates. Budget for a premium tier, not a price war.
• Latency, not token count, is the win. Faster decode shortens agent loops and unlocks real-time products; it does not cut token consumption. Buy speed only where interactivity demands it.
• Set the trigger. The first Nebius Token Factory or Groq Cloud rate card is the signal to add an LPX speed tier to your quotes. Watch for it before Dec 31, 2026.

Frequently asked questions

Sources

• NVIDIA Newsroom, "NVIDIA Groq 3 LPX Now in Full Production With World-Class Speed for Agentic AI" (Aug 24, 2026): nvidianews.nvidia.com
• SiliconANGLE, "Nvidia's dedicated inference accelerator Groq 3 LPX enters full production to supercharge AI agents" (Aug 24, 2026): siliconangle.com
• Tom's Hardware, "Hot Chips 2026: Nvidia presents Groq 3 LPX architecture and unveils its first third-party inference benchmark" (Aug 26, 2026): tomshardware.com
• Nebius, "Nebius Token Factory Becomes First AI Cloud to Adopt NVIDIA Groq 3 LPX" (Aug 2026): nebius.com
• NVIDIA Blog, "NVIDIA Advances Vera Rubin Inference With New LPX, Spectrum-X Multiplane and More" (Aug 2026): blogs.nvidia.com
• TechTimes, "Groq 3 LPX Hits Full Production: SRAM Decode Chip Reaches 3,400 Tokens Per Second" (Aug 25, 2026): techtimes.com
• WindowsForum, "NVIDIA Groq 3 LPX Enters Production, No API Date Yet" (Aug 2026): windowsforum.com
• Tom's Hardware, "OpenAI's 700W Jalapeño ASIC outpaces 1,400W Nvidia flagship GPU" (Aug 2026, context for the paired inference story): tomshardware.com

Accuracy note: The full-production announcement, Vera Rubin platform framing, Nebius first-adopter status, and Jensen Huang quotes are from NVIDIA's Aug 24, 2026 press release and blog, cross-checked against SiliconANGLE (Aug 24) and Tom's Hardware (Aug 26). The 3,431 output tokens/sec figure is Artificial Analysis' third-party measurement as reported by Tom's Hardware (NVIDIA's release rounds it to 3,400); the 10,996 tok/s demo figure was explicitly flagged self-reported by Nvidia's Igor Arsovski. The Artificial Analysis benchmark ran on a private, pre-release, single-concurrency endpoint served through Google Cloud and is not directly comparable to multi-tenant public endpoints — treat it as a hardware ceiling, not a service guarantee. The $20 billion December 2025 Groq license/hiring deal, the Rubin CPX cancellation, the Warren/Blumenthal FTC letter, and the ~500MB SRAM / 256-chip / 40 PB/s / 315 PFLOPS / 350 ns / scales-past-1,000-LPU rack specs are from Tom's Hardware's Hot Chips architecture coverage. The "3–5x over Rubin alone" figure is Nvidia-measured, not third-party. Nebius's "before December 31, 2026" commitment is per TechTimes; the absence of published LPX pricing is per WindowsForum's review of Nebius public pricing as of Aug 2026. Cerebras CS4 and SpaceX Vera Rubin claims are from the same Hot Chips session as reported by Tom's Hardware/SiliconANGLE. No LPX API price exists as of Aug 26, 2026 — this post deliberately does not model one; re-verify rates before quoting clients.