Skip to main content

​
The corpus is synthetic and has ground truth

vitruvio.bench builds deterministic brains at N ∈ {10², 10³, 10⁴, 10⁵} across six subjects with distinct vocabularies, and it knows which blocks answer which query. That is what makes recall a measurement rather than an impression. The minimum size matters: below roughly 512 blocks the exhaustive plan wins legitimately, so a small corpus never exercises an index at all. A benchmark that never touches the thing it is benchmarking is the failure this size floor exists to prevent.

​
What it reports

recall@{1,5,10}, nDCG@10, and p50/p95/p99 for six configurations: scan, lexical-only, vector-only, planner(λ=0), planner(default) and planner(floor=1.0). The λ=0 column exists to demonstrate the recall/cost trade-off empirically rather than assert it rhetorically: with recall weighted at zero the planner picks the cheapest admissible plan, and the recall column shows what that costs. CI gates the N=10³ tier on two conditions:
  • recall@10 at least as good as scan;
  • p95 within 3× of scan.
The second is deliberately loose. A real planner is allowed to be slower on a tiny brain, and pretending otherwise is what pushes it toward bad plans at the sizes that matter.

​
Calibration, and what is not built

The cost model’s constants are measured defaults, overridable through [planner] in vitruvio.toml. query explain --analyze reports the estimate beside the actual per node, and estimation_error summarises the gap — which is the honest way to find out the model is wrong about this brain. Refitting the constants automatically is not implemented: nothing persists the samples and there is no calibrate command. Correcting the model on a given machine is a hand edit for now.

​
inspect doctor

One command that answers “is anything about this brain going to disappoint me”. Run it after a pull and before a publish. It reports rather than fixes, exits 0 whatever it finds, and every row carries a stable code, a severity of ok, warn, fail or skip, a detail for a person and a remedy when there is something to do. It never loads a model — index sidecars are read by their headers — and it never touches the network unless you pass --registry, which makes one read-only request; --local DIR points that request at a filesystem registry instead. The checks are declared once, in the runtime, and this table is tested against that declaration: “Missing modules” is exactly two signals: modules.installed when the brain holds nothing, and modules.partial when it was pulled selectively. A vector sidecar that is absent is not reported: a module whose blocks project to nothing embeddable has none, and that is not a fault.

​
Determinism, because retrieval bugs hide in iteration order

Three habits carry most of the weight in the test suite:
  • fake:deterministic derives vectors from sha256 — bit-exact, no dependencies — so “the vector index must find this synonym” is an assertion rather than a hope.
  • The suite runs under PYTHONHASHSEED ∈ {0, 1, 42} and compares the explanation JSON byte for byte. That catches the most insidious class of bug here: set iteration order leaking into a plan.
  • The five structural indices have byte-exact golden dumps. The vector index does not, and cannot — it asserts round-trip equality of results.