> ## Documentation Index > Fetch the complete documentation index at: https://vitruvio.sh/docs/llms.txt > Use this file to discover all available pages before exploring further. # Benchmarking and calibration > Measuring recall, and calibrating the cost model. ```bash theme={null} vitruvio bench --tier 1000 vitruvio bench --tier 1000 --gate # exits non-zero if the planner loses either gate vitruvio bench corpus --into ./corpus # keep the generated brain to look at vitruvio inspect doctor ``` ## The corpus is synthetic and has ground truth `vitruvio.bench` builds deterministic brains at N ∈ \{10², 10³, 10⁴, 10⁵} across six subjects with distinct vocabularies, and it knows which blocks answer which query. That is what makes recall a measurement rather than an impression. The minimum size matters: **below roughly 512 blocks the exhaustive plan wins legitimately**, so a small corpus never exercises an index at all. A benchmark that never touches the thing it is benchmarking is the failure this size floor exists to prevent. ## What it reports recall@\{1,5,10}, nDCG\@10, and p50/p95/p99 for six configurations: `scan`, lexical-only, vector-only, `planner(λ=0)`, `planner(default)` and `planner(floor=1.0)`. The `λ=0` column exists to demonstrate the recall/cost trade-off empirically rather than assert it rhetorically: with recall weighted at zero the planner picks the cheapest admissible plan, and the recall column shows what that costs. CI gates the N=10³ tier on two conditions: * recall\@10 at least as good as `scan`; * p95 within 3× of `scan`. The second is deliberately loose. A real planner **is allowed** to be slower on a tiny brain, and pretending otherwise is what pushes it toward bad plans at the sizes that matter. ## Calibration, and what is not built The cost model's constants are measured defaults, overridable through `[planner]` in `vitruvio.toml`. `query explain --analyze` reports the estimate beside the actual per node, and `estimation_error` summarises the gap — which is the honest way to find out the model is wrong about *this* brain. **Refitting the constants automatically is not implemented**: nothing persists the samples and there is no `calibrate` command. Correcting the model on a given machine is a hand edit for now. ## `inspect doctor` One command that answers "is anything about this brain going to disappoint me". Run it after a pull and before a publish. It reports rather than fixes, exits 0 whatever it finds, and every row carries a stable `code`, a `severity` of `ok`, `warn`, `fail` or `skip`, a `detail` for a person and a `remedy` when there is something to do. It never loads a model — index sidecars are read by their headers — and it never touches the network unless you pass `--registry`, which makes one read-only request; `--local DIR` points that request at a filesystem registry instead. The checks are declared once, in the runtime, and this table is tested against that declaration: | code | trips as | means | remedy | | - | - | - | - | | `env.extra.oras`, `env.extra.sentence_transformers`, `env.extra.pypdfium2`, `env.extra.keyring` | warn | an optional extra is not installed | install `pyboltzmann[oci]`, `vitruvio[local]`, `vitruvio[vision]`, `vitruvio[keyring]` | | `env.extra.usearch`, `env.extra.pyroaring` | fail | a dependency of `vitruvio-indices` is missing: the install is broken | reinstall `vitruvio-indices` | | `config.invalid` | fail | `vitruvio.toml` could not be loaded, so nothing below could be asked | `vitruvio config validate` names the field | | `config.actor` | warn | no actor is configured; every write is attributed, so writes will be refused | `vitruvio config set actor.id …`, or `--actor` | | `config.collaborators` | ok | which declared parties a write into the selected brain records as assisting, and where the declaration came from; `--assisted-by` may only select among them | — | | `brain.layout` | fail | no brain is selected, or the selected path is not an OCI layout | `--brain PATH`, `vitruvio brain use PATH`, or `vitruvio brain init PATH` | | `brain.integrity` | fail | the module roots do not match the blocks | `vitruvio brain verify`; pull again from the origin | | `modules.installed` | warn | nothing is installed yet | `vitruvio source register FILE`, or `vitruvio dist pull` | | `modules.partial` | warn | a selective pull left the published modules it did not ask for behind | `vitruvio dist pull` without `--module` | | `blocks.missing` | fail | named blocks are absent from the store: a selective install, or corruption | `vitruvio dist pull` | | `blocks.tombstoned` | warn | blocks were redacted under policy; they are still verifiable members, and this is lawful | nothing; `vitruvio inspect resolvability` lists them | | `blocks.unattributed` | warn | blocks with no registration or derivation record naming them, so nobody is on record as their creator; catalog structure written before pyboltzmann 0.9.1 is the known case | re-apply the catalog manifest with `vitruvio catalog apply`; `vitruvio inspect blocks MODULE` names them | | `indices.unbuilt` | warn | a declared structural index has never been built | `vitruvio index build` | | `indices.stale` | warn | a sidecar describes a different composition than the one installed | `vitruvio index build` | | `indices.model_mismatch` | fail | a vector sidecar was built with an embedder other than the configured one, read from its header | configure that embedder, or `vitruvio index build --force` | | `embedder.unavailable` | fail | the configured embedder cannot be constructed on this machine | install its extra; `vitruvio config embedder list` names it | | `embedder.semantic` | warn | the configured embedder is `hashing` or `fake`: it ranks, and carries no meaning | set a real model under `[embedding.text]` | | `registry.reference` | warn | no registry is configured | set `[registry].reference` or `namespace` | | `registry.credentials` | warn | no credentials for the registry host; resolved offline, as `registry whoami` does | `vitruvio registry login HOST` | | `registry.reachable` | skip, or ok / fail with `--registry` | whether one manifest resolves | check host and network; `--insecure` for plain HTTP | | `cache.models` | ok | where the model cache is and what it costs | — | "Missing modules" is exactly two signals: `modules.installed` when the brain holds nothing, and `modules.partial` when it was pulled selectively. A vector sidecar that is absent is not reported: a module whose blocks project to nothing embeddable has none, and that is not a fault. ## Determinism, because retrieval bugs hide in iteration order Three habits carry most of the weight in the test suite: * `fake:deterministic` derives vectors from sha256 — bit-exact, no dependencies — so "the vector index must find this synonym" is an assertion rather than a hope. * The suite runs under `PYTHONHASHSEED ∈ {0, 1, 42}` and compares the explanation JSON byte for byte. That catches the most insidious class of bug here: set iteration order leaking into a plan. * The five structural indices have byte-exact golden dumps. The vector index does not, and cannot — it asserts round-trip equality of *results*. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.