Ori 会为一次运行解析一个测试框架和一个模型，并在该运行的所有测试中保持不变，因此同一评估文件的两次运行使用相同配置。它通过 OpenRouter 发送请求，因此一次比较可以包括来自多个供应商的模型。上述指令适用于临时目录。如果改为手动操作：https://openrouter.ai/docs/guides/ori/eval，评估文件会像普通代码一样保留在项目中，并且当新的模型发布后可以重新运行，可以使用 --baseline 对比早期运行结果，也可以在 CI 中按计划运行。
每次测试调用后，将生成 ID 传给 get-generation。你可以获得精确费用、提示和完成的令牌数、提供商以及延迟。在一组代表性提示上取平均值，调整任务成功频率，并将该数字记录在决策文档中。
To choose an AI model, define the task, shortlist candidates from live usage and benchmark data, compare price and latency across providers, then test the finalists on your own prompts. Judge them on cost per completed task rather than cost per token, and expect the answer to change as new models are released.
We won’t name a single best model. Any name we printed would be out of date within a month, and the right model depends on what you’re building and what you’re willing to pay for correct results.
This article describes the framework we use to answer the question, and how to run it without leaving your editor. Our MCP server：https://openrouter.ai/docs/guides/overview/mcp-server connects your assistant to live usage rankings, third-party benchmarks, per-provider pricing, and a way to send test prompts to candidate models.
There’s no single best AI model, only the best model for a given task, budget, and moment.
Different tasks require different strengths. Summarization and coding place different demands on a model. Extraction needs valid JSON on every call more than it needs good prose. A chat feature depends on how quickly the first token arrives, which is separate from the quality of the full answer. A model that ranks first on a coding benchmark can still perform poorly on long documents and cost too much for routine extraction.
A more useful question includes the specific job. Instead of “what is the best AI model,” ask “what is the best model for extracting line items from scanned invoices,” or “for reviewing a TypeScript pull request,” or “for summarizing a 90-minute call transcript.” Write your own version of that question, then answer it from current data rather than from an old ranking.
Our traffic shows how much the answer varies by task. We classify a sample of requests into 29 task types and publish the market share：https://openrouter.ai/rankings. Coding alone accounts for nine of them, covering code generation, debugging, code review, repo scanning, SQL work, and DevOps config. Those nine don’t share one leader. In the seven-day window ending 25 July 2026, one model led eight of them and a different one led code review and security. “Best model for coding” is too broad a question even within coding.
Benchmarks are useful for narrowing hundreds of options down to a few candidates you can test properly. We surface third-party scores from Artificial Analysis：https://artificialanalysis.ai and Design Arena：https://designarena.ai alongside our own usage data.
A leaderboard can’t make the final choice for you. Scores are noisy, popular benchmarks attract tuning, and none of them ran your prompts. Use the leaderboard to shortlist and your own tests to decide.
Different tasks require different strengths. Coding needs reasoning quality and reliable tool calls. Summarization needs a large context window and low input pricing. Extraction needs consistent adherence to a schema more than fluency. Chat needs low latency. Vision needs a model that accepts images at all, which narrows the field before quality matters.
No model leads every category. Choosing one model for everything produces an expensive default that performs well in demonstrations and poorly on the work you actually run.
Every step below runs through the MCP server, so connect it first.
The OpenRouter MCP server：https://openrouter.ai/docs/guides/overview/mcp-server is hosted by us, so there’s nothing to install locally. Any MCP client can connect. The setup below covers Claude Code, Cursor, and the Codex CLI, and the docs also cover OpenCode and Claude Desktop. You connect once, and your assistant can then pull live models, pricing, credits, rankings, benchmarks, and docs, and send test prompts, without you leaving the editor. Use it while you’re choosing a model. When you ship, call the API：https://openrouter.ai/docs/api_reference/overview as normal.
You can also authenticate from inside a session by running /mcp , selecting openrouter , and clicking Authenticate .
Cursor: add this to ~/.cursor/mcp.json , then verify with cursor-agent mcp list .
Authentication is one browser step and works the same way in all three editors. In Cursor it runs on your first request rather than from a login command. An unauthenticated request returns a 401 that starts our OAuth flow, and the approval screen states what you’re agreeing to before you agree to it.
We mint a key labelled OpenRouter MCP: , scoped to that client, with a seven-day expiry and a $10 credit limit you can change on that screen (MCP announcement：https://openrouter.ai/blog/announcements/openrouter-mcp-server). The key is short-lived and capped by default, you can disconnect at any time, and it’s revocable from your keys dashboard：https://openrouter.ai/settings/keys.
The flow redirects to localhost, which is normal for a desktop client like Claude Code or Cursor, but it means we can’t verify which local app receives the key. Approve it only if you started the connection yourself a moment ago.
Most are read-only lookups against live data. The exceptions are send-message , generate-image , transcribe-audio , and generate-speech , which make billable inference calls, and send-feedback , which writes feedback on one of your own generations (MCP docs：https://openrouter.ai/docs/guides/overview/mcp-server).
The assistant calls list-task-classifications for the code:general_impl tag and returns the leading models with their usage and token shares, then continues to per-provider pricing. No browser is involved.
Run these in order. Steps 2 through 5 each map to a specific call your assistant can make against live data. Steps 1 and 6 are your judgment: you define what you need and then decide what to ship.
Start with the job, not a model name. Write down the input, the output you expect, what counts as good, your latency target, and which way you lean when cost and quality conflict.
The last item affects every later step. A summary shown to customers justifies a higher price. A nightly extraction job across a million records justifies a lower price, even at some cost in quality, because volume dominates the bill. State which of those you’re building.
The shortlist comes from two questions: what are people using for this job, and what scores well on it?
For the first, call list-task-classifications . It returns our 29 task tags over a trailing seven-day window, each with its usage share and a ranked list of the models serving it, drawn from real traffic. For the second, call list-benchmarks with task_type set to coding, intelligence, or agentic, which returns Artificial Analysis and Design Arena scores alongside pricing. Those three categories are deliberately coarser than the 29 traffic tags, and the two calls are meant to be used together. The benchmark filter removes low-scoring models across a broad category, and the traffic tag then shows which models people use for your specific part of it.
list-daily-model-rankings is useful for trends. By default it returns daily token totals for the top 50 models overall, plus one aggregated other row per day. You can narrow it by use-case category such as programming or roleplay , by modality, or by tool-calling activity, but the category slices come from a sampled dataset aggregated weekly, so treat those totals as estimates. It tells you what’s growing, not what performs well on your job. The same view is available at openrouter.ai/rankings：https://openrouter.ai/rankings.
For each finalist, call list-model-endpoints . You get every provider serving that model with its price, context length, throughput and latency over the last thirty minutes, uptime, quantization, and supported parameters. The same model can differ between providers in price, speed, and reliability, and it’s better to find those differences here than in production.
The compare page：https://openrouter.ai/compare shows the same data in a browser when you need to share it with someone.
Benchmarks gave you the shortlist, and your own prompts make the final choice.
Run send-message against work you actually have: real tickets, real documents, real schemas, including the ones that usually cause failures. A clean evaluation set makes every model look competent, which is why it can’t separate them.
Three variants help while testing. :floor ：https://openrouter.ai/docs/guides/routing/provider-selection#floor-price-shortcut routes to the cheapest provider serving that model, which keeps evaluation costs down. :nitro ：https://openrouter.ai/docs/guides/routing/model-variants/nitro routes to the fastest, which is how you check a latency budget. :online ：https://openrouter.ai/docs/guides/routing/model-variants/online adds web search when the task needs current context.
Watch what :online costs. Running the same trivial prompt three ways, :floor came to $0.0000030 and :nitro to $0.0000024, while :online came to $0.0052576. That’s roughly two thousand times the plain call for a single prompt, so use it deliberately rather than leaving it enabled.
Ad-hoc test calls answer the question once. Ori Eval：https://openrouter.ai/docs/guides/ori/eval makes this step repeatable. You ask a question in plain words, for example “what is the best model for my support agent,” and your coding agent finds test material in your project, writes the eval as a *.eval.ts file, runs the candidate models, and recommends one with the scores, times, and costs behind it. To start it from your editor, give your coding agent this instruction:
Ori resolves one harness and one model for a run and holds them for every test in that run, so two runs of the same eval files use the same configuration. It sends its requests through OpenRouter, so one comparison can include models from many providers. The instruction above works in a temporary directory. If you run the manual steps：https://openrouter.ai/docs/guides/ori/eval instead, the eval files stay in your project as normal code, and you can re-run them when a new model is released, compare against an earlier run with --baseline , and run them in CI on a schedule.
After each test call, pass the generation ID to get-generation . You get the exact cost, the prompt and completion token counts, the provider that served it, and the latency. Average across a set of representative prompts, adjust for how often the task succeeds, and record that number in the decision doc.
Two things to note. The generation record isn’t queryable the instant the call returns, so a lookup made immediately afterward returns a 404 and resolves a few seconds later. Retry rather than treating that first 404 as a failure. Also, the completion response already carries usage.cost , so if the price of the call is all you need, skip the extra round trip. Use get-generation when you want the provider, the latency, or the native token counts as well.
If one model wins clearly across the work you run, use it.
If the results are close, if your traffic mixes several kinds of jobs, or if you don’t want to revisit the decision every time a better model is released, point at the Auto Router：https://openrouter.ai/docs/guides/routing/routers/auto-router with the model string openrouter/auto-beta . The older openrouter/auto still resolves but is documented as deprecated, so use the current one.
The router doesn’t pick at random. It classifies each request into roughly 30 fine-grained task types, ranks candidates by real-world spend share over a trailing seven-day window, applies your cost and quality preference, and routes with fallbacks (Auto Router docs：https://openrouter.ai/docs/guides/routing/routers/auto-router). That’s the framework above, running per request, on the same task-classification data you queried in Step 2.
Cost per task, not cost per token, is the correct unit for comparing model economics.
People compare on per-token price because it’s easy to compare, and because fully-loaded cost used to be difficult to measure. With get-generation returning the real number on every call, that difficulty is gone.
A model with a low unit price stops being cheap when it retries, produces completions longer than your token budget, or needs a stronger model behind it to catch its failures. A more expensive model that completes the task on the first attempt often costs less in total.
A 2026 study of reasoning-model pricing measured this. In 32% of model-pair comparisons the model with the lower listed price incurred the higher total cost, with the reversal reaching 28x at the extreme (Chen et al., “The Price Reversal Phenomenon”：https://arxiv.org/abs/2603.23971). The authors attribute it to how differently models spend tokens on thinking: on the same query one model can use 900% more than another, and repeated runs of a single query vary by as much as 9.7x. Listed price reflects none of that.
Most comparisons leave out expected attempts, and that term usually decides the result.
Here is the calculation with real prices, checked on 27 July 2026. GPT-5.4 mini：https://openrouter.ai/openai/gpt-5.4-mini listed at $0.75 per million input tokens and $4.50 per million output. Claude Sonnet 5：https://openrouter.ai/anthropic/claude-sonnet-5 listed at $2.00 and $10.00, roughly 2.4 times more. Take a task of 2,000 input and 800 output tokens. If Sonnet 5 succeeds on the first attempt 95% of the time, it costs about $12.63 per thousand completed tasks. For the mini to match that, it has to succeed on the first attempt 40% of the time. Below 40%, the model that is 2.4 times cheaper per token is the more expensive way to finish the work.
When you report this to others, use cost per 1,000 completed tasks. The model with the cheapest tokens is often not the cheapest way to finish the work.
The chart below plots the whole curve instead of the single point. The curve is the more useful thing to keep, because the break-even rate moves with the price gap between the two candidates you’re comparing. A wider gap lets the cheaper model tolerate a much lower success rate before it loses.
The same worked example, plotted across every success rate rather than one. Sonnet 5 is held flat at 95% for reference while the mini’s rate varies. List prices are as of 27 July 2026, the task is 2,000 input and 800 output tokens, and the success rate is the variable being swept rather than anything we measured.
This is a starting point rather than a ranking. Each row tells you what to optimize for and which call to make, and the live data supplies the name. We don’t print winners, because any list of winners would be out of date by the next release.
Coding: Decide which kind of coding you mean. Our task tags separate code generation from debugging, review, frontend, and repo scanning, and the leaders differ. Test candidates on a real ticket from your backlog, not on a toy problem.
Summarization: Read input price and context length together, because either one alone will mislead you. A cheaper model with a large window often beats a stronger one with high input pricing.
Extraction: A smaller model that returns valid JSON every time beats a stronger one that corrupts a field twice a day. Test the difficult inputs: missing fields, ambiguous records, and malformed source text.
Vision: Multimodal quality varies strongly by domain, so filter for image input and then run your own screenshots. A stock demo set makes every candidate look good.
In every case, run the query, look at this week’s numbers, and pick from those.
You don’t have to commit to one model at all.
One integration gets you the whole catalog：https://openrouter.ai/models across providers. When a better model is released next month, you change a model string instead of integrating another SDK and re-testing an integration path. Selection and execution sit on the same platform, and with MCP the selection data is available in the editor you already work in. You also get provider redundancy and automatic fallback：https://openrouter.ai/docs/guides/routing/model-fallbacks, and a per-request cost figure precise enough to make the cost-per-task calculation above accurate rather than estimated.
Going direct to a provider is a reasonable choice if you’re certain you want exactly one model from exactly one place and that won’t change. New models are released constantly, so consider how certain you are.
Most bad model decisions come from measuring the wrong thing, or measuring the right thing too late. These are the five we see most.
Treating leaderboard position as a production decision: A high rank on a public leaderboard earns a place on your shortlist, not production traffic. Run your prompts through the model first.
Shopping by price per token: A low unit price hides retries, long completions, and fallbacks. Until get-generation has told you the cost per completed task, you don’t know what the model costs.
扫码直达文章详情