OpenBenchmarks

OpenBenchmarks, branded on the site as Openbenchmarks for Agents, is a public benchmark hub built for a specific moment, when an AI agent or the team behind it has to decide which SaaS API to trust for a job. Load an agent up with tools and the hard part isn't calling them, it's knowing which one actually performs. This site tries to make that call with numbers rather than marketing.

The pitch is transparency first. The methodology is public, scoring rules are written down, and every metric is meant to be externally validated rather than self-reported by a vendor. Each benchmark page documents how the test was run, how it was scored, and what pricing assumptions went into any cost comparison, so a reader can check the reasoning instead of taking a leaderboard on faith. That posture is the whole point, since a benchmark you can't inspect isn't worth much when real money and real reliability are on the line. The reason that transparency is the headline is that most benchmarks arrive with a conflict of interest baked in. A vendor's own numbers tend to flatter the vendor, and a leaderboard you can't reproduce asks you to trust a result on faith. By publishing the method and the pricing assumptions on every page and leaning on external validation, the project is trying to make the numbers usable for a real decision rather than a marketing screenshot, which is a low bar in theory and a rare one in practice.

The first benchmark live on the platform is the Lookalike Benchmark, which compares providers on how well they surface similar companies. Results are reported by metric, so you can see who wins on a given measure rather than a single blended score, for example a provider leading on average precision at the top of the results list. The format is deliberately granular, because the best tool for relevance and the best tool for cost are often not the same one, and a per-metric view keeps that honest.

What makes it unusual is that it's designed to be read by agents, not just people. The same data is available through a REST API that needs no authentication, through an MCP server for tools that speak that protocol, and through a stack of machine-readable discovery formats including OpenAPI, an llms.txt file, an MCP manifest, and an agent card. In practice that means an autonomous system can query the benchmarks at decision time and route itself to the stronger option, rather than a human copying a number out of a web page. The access surface is deliberately broad. Beyond the plain REST endpoints, there's an MCP server on a dedicated subdomain using OAuth 2.1 with dynamic client registration, plus a set of discovery files including an llms.txt, a fuller variant, an MCP manifest, an agent card, an agent skills index, and an API catalog following the RFC 9727 convention. The point of all that plumbing is that an agent can find and read the benchmarks on its own, without a human hand-configuring an integration first.

In practice, an agent facing a choice can query the benchmarks directly, read which provider leads on the metric it cares about, and route the call accordingly, all without a person in the loop. Because the endpoints need no key and the data refreshes hourly, that lookup can run at decision time on current numbers rather than a figure someone pasted into a config weeks ago. It's a small shift, but it moves tool selection from a static setting into something an agent can reason about live. The freshness story is straightforward. Data refreshes on an hourly cadence and standard HTTP cache headers are honored, so both browsers and automated clients can behave well without hammering the service. Because access is open, there's no key to provision before an agent can start reading, which lowers the friction for wiring it into an existing pipeline.

The audience splits two ways. There are the humans doing build versus buy analysis, engineering leads and product teams weighing whether to build a capability in-house or pay for an API that already does it well. And there are the agents themselves, systems that need a trustworthy source of comparison to choose tools on their own. Both get the same underlying data through whichever surface suits them.

It's early, and worth reading as such. Coverage today centers on the Lookalike Benchmark rather than a broad catalog across every category, so how useful it is depends heavily on whether your decision overlaps with what's already measured. The tradeoff is that the transparency and the machine-readable access are unusually complete for a young project. Access is free to query for both humans and agents, across the website, the API, and the MCP server, with the details laid out on the site rather than hidden behind a signup.