VernLLM

VernLLM is an open-source TypeScript framework that wraps calls to large language model providers with the reliability controls an application usually has to build itself. It sits inside the application's own process and keeps the existing provider client in place. Developers get one call interface plus configurable retries, timeouts, fallback, rate limiting, circuit breaking, caching, structured output, streaming, usage tracking, middleware, and observability. The goal is to make a production LLM call predictable without sending traffic through another hosted gateway or forcing a team to adopt a large application framework.

The basic setup is deliberately small. A developer installs the vern-llm package, adapts an existing provider client, chooses a model, and creates a VernLLM instance. A plain call gets JSON parsing, retries, and timeouts by default, while raw text is available by switching JSON mode off. Calls can override instance settings such as the model or reasoning effort, which supports patterns where routine work uses a cheaper default and difficult work escalates selectively. Streaming returns incremental chunks alongside a final validated result, so an interface can render output immediately without giving up the same result handling used by nonstreaming calls.

Its resilience features address several different failure modes instead of treating every error as a retry. Retries use exponential backoff with jitter and understand provider retry guidance, while parse errors, validation failures, and nonretryable responses can fail fast. A retry budget can cap the share of recent traffic spent on repeated attempts. Provider fallback tries declared backup targets in order when policy allows a transition. Client-side rate limiting queues work against requests, tokens, and concurrency ceilings before the provider rejects it. The circuit breaker stops sending calls to an unhealthy target after repeated failures and can recover after its cooldown conditions are met.

VernLLM also covers the work around a successful model response. A call can request structured output and validate it with Zod or another validator that exposes a compatible safeParse interface. Provider-native JSON Schema mode can constrain generation where the provider supports it. Tool calling is represented in the response but execution stays in the host application, which keeps side effects under the developer's control. The cachedCall helper can reuse completed work through an in-memory store or a custom backend such as Redis. Middleware can transform or redact outgoing requests, wrap a complete logical call, and observe outcomes for tracing, logging, or cost accounting.

Observability is built around a single event stream for retries, fallovers, circuit transitions, and rate-limit waits. Usage tracking reads token counts from provider responses, while optional usage metering can reserve capacity before a call and refund failed work. Logging is replaceable, and debug output can be redacted before it reaches the logger. These extension points follow small interfaces rather than one fixed deployment design. A team can bring a distributed rate limiter, a shared cache, its preferred logging stack, or a custom circuit policy without rewriting the rest of the call path.

Provider coverage is broad but doesn't hide the underlying provider. OpenAI-compatible services can use a near-direct adapter path, while official adapters translate Anthropic, Gemini, and AWS Bedrock into the common interface. A raw HTTP adapter handles providers that lack a packaged integration. The project lists OpenAI, Groq, Mistral, DeepSeek, Cerebras, Together, Fireworks, Ollama, and many other compatible endpoints. VernLLM has zero runtime dependencies and doesn't bundle provider SDKs or schema validators, so developers install only the clients they need. That keeps the library light, though it also means model credentials, provider billing, and provider-specific limitations remain the application's responsibility.

This is best suited to TypeScript teams that want a reusable call layer inside one service and prefer code-level control over a centrally managed proxy. Keeping execution in process removes an extra network hop and lets reliability decisions react to application state. A gateway can still be a better fit when many services in different languages need one shared policy and credential boundary. VernLLM is distributed under the MIT license, with source, documentation, and an npm package available publicly. The library itself is free and open source, but calls made through it still incur whatever charges the selected model provider applies.