FlexInference

FlexInference is a deadline-aware router for large language model requests. It sits between an application and the model providers it already calls, then tries to run the exact same request on a cheaper tier whenever there's enough time to pull it off. The site puts it plainly, "we find you cheaper inference within the time you provide." The problem it goes after is that inference bills grow with usage and most teams have very few levers to pull. You can downgrade to a weaker model, trim prompts, or cut output budgets, and all three cost you quality. FlexInference takes a different angle by leaving the request completely alone and changing only the tier that serves it. Your chosen model, output budget, thinking configuration, and tool use all stay exactly as you set them.

The mechanism is one new field called start_within. Set it to default, priority, auto, or a duration between five seconds and ten minutes, and FlexInference races a cheaper flex tier against the standard one inside that window. If flex can start in time, you get the same answer for less. If it can't, the request escalates to the standard tier before flex writes a single output token, so there's no half-finished response to pay for and no commission on the miss. Requests sent without a time budget skip the race entirely and go straight to standard at standard cost, which makes this something you opt into per request rather than a global behavior change you have to reason about later.

The demos on the site give a decent sense of where the trade actually lands. A Gemini image classification workload came out 38.9 percent cheaper for 20.2 percent more latency. An OpenAI deep research run landed 44.8 percent cheaper and, oddly enough, 30.1 percent faster. A browser agent came in 51.5 percent cheaper for 9.7 percent more latency. The pattern is that work which tolerates a short wait, meaning batch jobs, background agents, and research pipelines, benefits most, while anything a user is actively watching load is a poorer fit for a long deadline.

You bring your own provider key, so the provider still bills you directly and your credits, discounts, negotiated rates, and API tiers stay yours. The API is OpenAI-compatible, which means existing clients work by pointing the base URL at FlexInference and passing a FlexInference key. It covers OpenAI, Gemini, and Anthropic, though Anthropic is proxy-only and doesn't get the flex race or the savings that come with it. You can still call any Claude model through the unified endpoint or use the Anthropic Messages format directly, and you draw down your own Anthropic credits either way. Python and TypeScript SDKs exist for teams that want strict types, but they're optional rather than required.

The published numbers are specific enough to argue with. Median cost across 62,854 requests came in roughly 47 percent lower with about 16 percent more time to first token, and an earlier 10,000-request benchmark from June 2026 put blended savings at 40 to 50 percent for under 10 percent added latency. Routing runs on Cloudflare Workers across 300 or so cities and adds one to five milliseconds on a cold start. Prompts and replies pass through without being stored or read, and provider keys are envelope-encrypted with AES-256-GCM and locked to a single org and provider slot, so a key that won't decrypt gets treated as missing rather than quietly reused somewhere it shouldn't be.

Several design choices are aimed squarely at people who've been burned by other proxies. FlexInference fails fast and loud rather than silently stripping a parameter you set to force a request through, and provider rejections come back with the real status code attached so you debug against your actual intent. Errors arrive in the shape of the SDK you called and carry a machine-readable code, the exact fix, and a doc_url, which means an agent can often correct itself on the first try. There's an MCP server too, letting Claude, Cursor, and other MCP clients search the docs, look up error codes, and manage keys over OAuth without leaving the editor. It never runs inference and never takes a raw provider key. Optional upstream retries run from one to five attempts with exponential or linear backoff plus jitter, honor a provider's Retry-After header, and only fire before the response commits.

Pricing is unusual and worth understanding before wiring it in. Standard, priority, and auto routing carry no fee at all, and neither do Anthropic proxy requests, so there's no seat count, no tier, and no trial to get past. When a flex request actually beats the standard price, FlexInference keeps 20 percent of what it saved you, which the site illustrates as a $10 request dropping to $5 for a $1 fee and a $6 bill. Commission accrues per request and gets billed monthly once your balance passes $20, with smaller balances carried forward and settled at least every three months. If a payment goes past due, priced flex pauses and returns a 402 while free routing keeps working. Lexical caching and retries work today, while semantic caching, fallback providers, and automatic model selection are all listed as coming soon.