Nvidia says the best way to make tokens cheaper is increasing performance per watt. The market says: just use a cheaper model

Ian Buck, Nvidia's VP of hyperscale, at the AI infra summit:

Agentic workloads are 100x heavier than 2023 chat – 142K input tokens, million-token KV caches, 65 agent turns. Vera Rubin delivers 30x throughput per megawatt vs Blackwell on SemiAnalysis AgentX. The MaxLPS software fits 40% more GPUs into the same power budget. Lambda validated it: 19 nodes in the budget of 16.

But here's the tension Nvidia doesn't address: while they optimize hardware to cut cost per token generation over generation, the token market itself is collapsing faster.

Cost per million input tokens right now:

- DeepSeek V4 Flash – $0.14
- GPT-5.6 Luna – $0.20
- GPT-5.6 Sol – $4
- Claude Opus 5 – $5
- Claude Fable 5 – $10

Similar capability on coding benchmarks. 70x price spread.

Nvidia's own customers are building around them:

- AWS Trainium 3 – claims 50% lower inference cost vs Nvidia
- Google TPU Ironwood – inference-optimized, captive to GCP
- AMD MI455X – 50% more memory per GPU, ships this year

Every hyperscaler is designing custom silicon to reduce Nvidia dependency.

Nvidia's moat is the full stack – CUDA, NVLink, annual cadence from Vera Rubin to Feynman Ultra. The question is whether the cost-per-token floor drops faster than Nvidia can improve performance-per-watt. Right now, the floor is winning.