While I spend a lot of time thinking about infrastructure, there are a lot of interesting questions on the application side that I find myself constantly asking people. One of those questions is loosely “who should be in charge of model routing?” In this context, I mean routing in the sense of picking the best model for a given task. In technology, consumers have always been blessed with choice, but the cost variability was never so high. The difference between spinning up an AWS or GCP microservice might vary a few cents for the same feature set (more or less). On the other hand, the difference between Sonnet and Fable pricing is quite high for a somewhat different feature set.

Let’s understand what model routing looks like right now from a few different vantage points. On the foundation model layer, users have all the choice. If you go to Claude, ChatGPT, or Gemini, you will have the ability to choose which model you want to ask questions to and chat with (or use via the API). Here, the choice sits with the consumer fully. If a user wants to pay for the best model for every query no matter how trivial, they have the choice to do so.

On the other hand, you have the emergence of routing platforms (OpenRouter as an example) which will handle the heavy lifting of routing your query to the right model based on the task complexity and pricing, as well as what they have available at that point. I haven’t used Perplexity in quite some time, but I recall they were a similar (and first) platform doing the same thing - dynamic model routing based on aspects of the task and what was available. In fact, this is what they were called a “ChatGPT wrapper” for.

The question I’d like to pose is whether users should have access to routing or whether the companies should do it automatically. Here’s the case for both sides.

As a consumer, we are constantly shaped by what we see in the news. If we see posts in the morning praising Fable 5 and highlights of one-shotted projects with it, it’s very likely that later that day if given the opportunity to use that model we might want to. It’s natural for human beings to be curious about things that they have heard are promising. On the other hand, if we see posts in the morning saying that Fable 5 is no better than its predecessors, benchmarks are lying, and it’s too expensive, we might not be as interested in using the model later that day. That’s also how humans work - we let externalities bias the way we think.

The problem here is that this ends up forcing the consumer into thinking that they are making a rational decision about models based on their task, but rather they’re making a slightly more emotional decision given all of the information floating around in their head. I’m not saying that consumers become completely irrational, but people definitely become less rational than they otherwise might have been in a vacuum. Given this, we can safely ask, “is the end-user the right person to choose their model?” In some sense, not giving them that choice might even save them from using a more capable model when their job could have been done cheaper and faster.

Now that we’ve talked about why end users maybe shouldn’t route themselves, let’s think about why companies shouldn’t. The obvious answer here is that they have a vested interest. Companies want to cut costs and prioritize certain enterprise consumers. I firmly believe that if given the choice, foundation model companies would likely rather provide consumers with slower and less capable models in order to give enterprise customers with more “potential to innovate” their more capable models. That’s just a hunch, but at the end of the day Fable for me is running on the same GPUs that it’s running on for Anthropic’s biggest customers.

Companies also want to control consumers and run experiments. If a company wants to run an A/B experiment, they might use the right to route to carry that out, effectively forcing end user choices for models. Given that piece of evidence and the idea that as a customer we want control, it makes sense why model companies give us the right to route our own models. However, I do see a world where companies route and end-users have no control over the process.

Imagine if companies routed and provided an expandable reasoning block for why they routed a certain way. Then, the end user wouldn’t really care. Say Anthropic said “we routed to Opus 4.8 because your query requires long reasoning and structured, creative output, but it doesn’t require an end-to-end implementation of something (which is why we didn’t use Fable)”. This would be quite cheap on their end (the smallest and fastest 1B models could do this, or some heuristic engine might be capable) and the end-user would ultimately have permission to override if they felt like that was necessary for their use case.

I think the foundation model companies will eventually move to this model, and it might prove to be quite effective for their expenses. While we’d like to think that we’re the most rational creatures, unfortunately we are biased by what we see and hear.