Verbatimeter

Verbatimeter is an open-source Python tool that checks, deterministically, how grounded an AI-generated answer is in the source text it was supposed to draw from. The problem it targets is familiar to anyone shipping retrieval-augmented generation. A model reads some context, writes an answer, and along the way it can paraphrase loosely, drift away from the passage, or invent a quotation that never appeared. You want to know whether the output actually traces back to the retrieved material, and you want that answer without standing up a separate judge model or making a network call every time you check.

It works by aligning words rather than guessing at meaning. Using word-level dynamic-programming alignment, it detects three things. Verbatim reuse shows up as contiguous matching runs of at least three words. Verbatim paraphrasing is caught with longest-common-subsequence matching, which finds source words that survived even when they were reordered. Quotation verification checks that any span the answer presents as a direct quote genuinely exists in the source. The result is a matched ratio you can threshold on, plus highlighting that shows exactly which parts of an answer are supported and which are floating free. Because the checks are span-level, you're not just told that an answer is 80 percent grounded, you can see which sentences carried real support and which ones the model appears to have made up.

Because everything is a deterministic string operation, the same inputs always produce the same score, and there's no model to prompt or API to pay for. The README is blunt about one of its selling points, that it uses no network access, ever. That makes it cheap to run, easy to reason about, and safe to drop into environments where you can't send text to a third-party service. It also means results are reproducible in a way a judge model can't promise, which matters when you're trying to gate a release or debug a regression.

Getting started is a single pip install verbatimeter. From the command line you can run something like verbatimeter with a source file, an answer file, a quotes flag to check quotations, and a fail flag that exits non-zero when grounding falls short, which is what lets you wire it into continuous integration. As a library it's just as direct. A check function compares candidate text against source text and hands back a result with a matched ratio, and a verify decorator wraps your own generation function so grounding is checked automatically on the way out, scoped either to quotations or to the whole answer. That decorator is the piece that makes it easy to bolt onto a RAG agent, since you pass it the argument that carries your retrieved context and it verifies each response against that context without you rewriting the call site. Token counting is configurable and defaults to tiktoken, so word and token accounting line up with the model you're already using.

A few touches show it's meant for real pipelines rather than a demo. It has a streaming integration that verifies text as it arrives, so you can flag ungrounded output while a response is still being produced instead of only after the fact. The highlighting is configurable and ships with colorblind-accessible palettes. And it carries multilingual support across eleven languages, spanning cases from English and French through to Hindi, Urdu, and Arabic, so grounding checks aren't limited to English-only content.

The people it's for are developers and researchers building RAG systems, LLM applications, and content-verification workflows who want a fast, offline, reproducible grounding check they can actually put in tests. If you've tried to measure hallucination by asking another model to grade the first one, you know how slow, costly, and non-deterministic that loop gets. Verbatimeter is a different bet, that a large share of grounding questions are really questions about textual overlap, and those can be answered exactly with alignment rather than approximately with a second model.

On access, it's MIT-licensed and free, published on PyPI with source on GitHub, and it needs Python 3.10 or newer. It's an early-stage project sitting in the 0.1 range, so the surface area is still small and the community young, but the README is substantive, with usage examples, benchmarks, architecture decision records, and integration notes for RAG agents. For a team that wants a lightweight, auditable guardrail on how faithful its generated text is to its sources, it's a focused tool that does one job and does it without a model in the loop.