Every time a new model drops, I open Artificial Analysis, find the model, and check where it sits on the Intelligence Index, the price chart, and the speed leaderboard. Then I check which inference provider is serving it fastest. I suspect a lot of you do the same thing (if you’re as eager as me).

Over the last few months, my take on a certain metric has changed slightly. That metric is tokens per second. I want to be careful here, because I am not anti-benchmarks or inference engineering; in fact, all of my research to date at the Sky Computing Lab has been focused around efficient inference! Benchmarks are in fact how we know anything at all about our systems. My claim is narrower than this:

Past a certain point, faster inference means nothing to the person using the system and research time might be better spent elsewhere.

So let me first explain what I mean by benchmarks and metrics, and then why I think we got here in the first place.

Benchmarks, metrics, and the leaderboard we all check

A benchmark, put simply, is a way to measure the performance of a system. The most famous one that I’ve used extensively is SWE-bench, which hands a model a real codebase plus a GitHub issue and asks it to write a patch that makes the tests pass. When it came out in 2023, the best model solved under 2% of the issues. Today, that number climbs higher and higher every few months and is often a headline for releases.

Another favorite of mine is the needle in a haystack test. Take a long document synthetically generated, bury a random fact somewhere in the middle (say, the number 37443341), and then ask the model at the end what the secret number was. When you vary the document length and where the needle sits inside of it, you get a very clean picture of how well a model processes long contexts. I know that models today are amazing and do really well on long-context type benchmarks (hence the shift to agentic benchmarks), but the example here is still illustrative generally.

Benchmarks are great at one thing: establishing base capability. How else would an organization know whether to use Cognition’s Devin as opposed to using GPT-4o (in a sense this is an exaggeration, but in another sense it is a very real possibility for the naive).

Metrics are slightly different . A metric measures some dimension of performance that sits on top of capability. Two metrics:

• Output tokens per second (tokens/s). A token is roughly three-quarters of a word or four characters in English. Tokens/second measures how fast a model returns these chunks when it is generating text as an answer to some input.

• Cost per task. The bill incurred when you run the model through some benchmark. Ideally, divide by the number of tasks in that benchmark.

A great place to see these numbers is Artificial Analysis. It aggregates an Intelligence Index built from a basket of evals, plus output speed, time to first token, and cost per Intelligence Index task, for every major model and provider. When a new model releases, it is effectively treated as ground truth (there are some debates on this, but I personally do not believe that their benchmarking is any more fradulent than other providers).

The reading-speed ceiling

I dug around and found a 2019 meta-analysis by Marc Brysbaert which pooled 190 studies and landed on the following:

• 238 wpm for silent reading of nonfiction

• 260 wpm for silent reading of fiction

[More curious is why these numbers are different, but unfortunately not in scope for this article: cc Reddit, Quora]

238 wpm → 4 words per second. At .75 words per token (my heuristic), that’s about 5 tokens per second. Even a very aggressive skimmer (which I am ashamed to say I was when I was a kid) at 500 wpm is only consuming around 11 tokens per second. Now, look at what inference providers are actually serving (cc our good friends at Artificial Analysis):

Sources: gpt-oss-120b providers, Inkling Small providers.

Even the slowest entries are already many times faster than you can read. Cerebras getting 1,700 tokens/sec is insane (shoutout wafer-scale chips; if you can’t comprehend this speed, see here for a slower demo). But for a person reading a chatbot answer to “why is the sky blue?”, the difference between 250 and 300 tokens per second is the difference between 50x and 60x faster than eyes can move.

This begs the question: if you take a fictional model served at 200 tokens per second by provider A, 250 tok/s by provider B, and 300 tok/s by provider C, what does that mean to a consumer?

Clarification: I am talking about general-purpose consumer inference, ie the chatbot on your phone. I am not talking about software engineering agents, batch jobs, or high-frequency training and RL workloads, where no human is reading the tokens at all.

Where the engineering hours should go instead

Squeezing out the last 50 tokens per second is not free. It takes a lot of GPU hours for profiling/ benchmarking, and a lot of very expensive engineering hours on kernels, speculative decoding, scheduling, and in some cases extreme hardware-software codesign. I think that this engineering is beautiful, and it will surely teach us a lot downstream (shoutout Baseten, Together, Modal, and more for actually being open about how they’re doing this). I just do not think it is the most pressing problem for an inference provider in 2026.

So if I were Baseten or Together (and this is very naively me thinking about it), here is where I would rather see those hours go:

• Developer experience. How long does it take a new customer to go from signup to a working endpoint with their own fine-tune? How good are the docs, the SDK, the error messages, the observability?

• Cold starts and container spin-up. Modal is a great example here: they have written publicly about booting containers in about a second and using memory snapshots to more than halve cold start latency. A request that waits 40 seconds for a scale-from-zero container does not care whether decode then runs at 250 or 300 tokens per second.

• Adoption across tasks. Multimodal inputs, structured outputs, tool calling that actually works, fine-tuning and serving in one place. Thinking Machines shipping Inkling alongside first-party fine-tuning on Tinker is woohoo!

• A harness. I think an inference provider could do a really good job building an agent harness, the scaffolding of tools, context management, and retries around a model, instead of doing a really good job shaving a few milliseconds off inference. As I argued in SaaS is Well and Alive, the hard part of real software is everything around the benchmark. The same is true for inference.

If you are Together or Baseten: is your moat going to be a marginal tokens-per-second lead that the next kernel release might erase, or is it going to be how much developer experience is baked into the product? My naive read is that the intangible developer experience is worth something like 20x the tangible micro-increase in tokens per second.

One caveat: cost per task is different. Faster, more efficient inference often means cheaper inference, and cheaper is a benefit everyone feels, consumer or enterprise. Artificial Analysis already reports cost per Intelligence Index task for every model, and I think that is a far more defensible number to compete on than raw speed. If your inference engineering is really cost engineering in disguise, I am much more sympathetic.

Why metricmaxxing took hold

Think about Slack ten years ago. When Microsoft announced Teams, Slack did not (and could not, unless they had Steve Jobs’ creativity) respond with a speed chart. It took out a full-page ad in the New York Times on November 2, 2016, and the very first piece of “friendly advice” it gave Microsoft was that “it’s not the features that matter.” The rest of the letter is about craft, care, and how the product makes teams feel.

Slack could not have said “we process 30% more channels per second than Teams” even if they wanted to. So it competed the only way it could - on an intangible, value-based pitch. The point is that the fight was about the experience, not metrics.

AI infra is the opposite, because every model and every provider gets a public score on intelligence, speed, latency, and price within days of launch (and today, on day 0). In a way this is great for businesses: for the first time you have a clean, legible way to prove you are winning. If you are an inference provider, a tokens/s lead is what helps you sleep at night (and a sustained one is probably the reason why you can raise like this)

But this is Goodhart’s law in action: when a measure becomes a target, it ceases to be a good measure. Toks/s started as a reasonable proxy for “this provider is good at serving models.” Once everyone started optimizing it directly, it stopped telling us much about what a user actually experiences. We have a way to show we win, but it isn’t telling us that much anymore.

Counterarguments?

Here are the strongest objections I can think of.

Alternative #1: Reasoning models mean users wait for tokens they never read.

This is the best counterargument, and I think it is partially right. Most consumer chatbots now think before they answer, and those thinking tokens are generated at the same decode speed but never shown to you. The Inkling Small numbers above make this painfully clear: on Artificial Analysis, Baseten returns the first answer token in about 6 seconds while Thinking Machines’ own API takes about 12. That gap is almost entirely decode speed, and a user absolutely feels 6 seconds.

But notice two things. First, the metric that captures this is time to first answer token, not output speed. We should be putting that number on the slide. Second, returns flatten. If a model thinks for 2,000 tokens, going from 50 to 250 tokens per second saves you 32 seconds. Going from 250 to 300 saves you about 1.3. See the difference?

Alternative #2: Agents and coding workloads don’t have a human in the loop.

Completely agree, which is why I scoped them out. When a coding agent chains fifty tool calls together, or an RL pipeline generates millions of rollouts, speed compounds and nobody is reading anything. Cerebras makes exactly this case for its gpt-oss-120b launch. My critique is about marketing consumer chat on a number that consumers cannot perceive, not about agentic infrastructure.

Alternative #3: Faster means cheaper.

Sometimes. Efficiency work that raises throughput per GPU does lower cost, and I have already said cost per task is the number I would rather see providers compete on. But per-user speed and aggregate throughput are not the same thing, and they often trade off: serving each user faster usually means smaller batches, which means fewer users per GPU. Maxing the leaderboard number can actually make serving more expensive. If the speed is a byproduct of cost engineering, great. If the cost is a byproduct of speed-maxing, that is the thing I am worried about.

I want to be clear that inference optimization is an important problem. Kernel scheduling, hardware-software codesign, speculative decoding is all really cool and will pay off in ways we cannot predict yet. My argument is not that we should stop. It is that, for consumer chat, the marginal tokens per second has become a scoreboard we optimize because it exists.

Sources

• Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (ICLR 2024)

• Greg Kamradt, Needle In A Haystack

• OpenAI, What are tokens and how to count them?

• Brysbaert (2019), How many words do we read per minute? A review and meta-analysis of reading rate, Journal of Memory and Language

• Artificial Analysis, model comparisons and methodology

• Artificial Analysis, gpt-oss-120b API providers

• Artificial Analysis, Inkling Small API providers

• Cerebras, OpenAI GPT OSS 120B Runs Fastest on Cerebras

• Thinking Machines Lab, Inkling: Our Open-Weights Model

• Modal, Cold start performance and Memory Snapshots

• Slack, Dear Microsoft (New York Times ad, November 2, 2016)

Originally published on Inference, Goodhart’s Law, and reading speed.