RECORDED EVALUATION / NO MODEL RUNS ON THIS PAGE
Does BenchSloth
Actually Work?
Inspect the evidence, including failures. These synthetic references were frozen before the final run, with development examples kept separate. Assistant-authored, implementer-visible labels; human validation remains n=0. Exact citations prove where text came from, not that the model understood the question. The combined candidate gets15/24 exact evidence answers versus14/24 rules, but13/24 exact workflows versus14/24 and8 unsupported answers versus7. No reliable model advantage is established. Gemma relationships ship as experimental proposals requiring review.
What the comparison measures
The primary score requires exact answer status and exact evidence IDs. Extra evidence counts against every method. The baseline is BM25 retrieval with section context, a generic attention vocabulary, factual-value checks and v0.5 structural candidate rules. All original source remains visible. The top3 retrieval shortlist contains all required passages for16/16 answerable questions. The v0.5 source-view adapter is explicitly the same retrieval baseline: v0.5 had no native QA interface. This is not a new v0.5 model inference run.
Gemma selects evidence candidates, including no-answer and ambiguity options. Verification checks IDs, verbatim excerpts, schema, earlier targets and explicitly unresolved language. It cannot certify semantic truth. QA is read only and makes no timer offers. Intervals are descriptive; related wording and implementer-visible labels limit generalization. Complete QA workflow means exact status, IDs and question-specific relationship, not physical task completion.
Reproduce
python -m venv .venv .venv/bin/pip install -r requirements-dev.txt ollama pull gemma3:4b # separate Google terms ollama serve .venv/bin/python scripts/v6/evaluate.py final .venv/bin/python scripts/v6/summarize.py .venv/bin/python -m unittest discover -s tests -v .venv/bin/uvicorn server_v6:app --host 127.0.0.1 --port 8772
Gemma 3 4B Q4_K_M, local CPU, temperature 0, seed 42, 4096-token context, 24 output-token cap, 90-second timeout, no hidden retry. Data and engine hashes are preserved in evaluation/v6. The evaluator writes missing runs only: use a copied checkout and move final-run aside for fresh inference. Latency varies by hardware.
The earlier failures stay on the record.
The separate v0.5 planner set has 20 cases and 1–15 actual lines, not 25. Strong rules and v0.5 tie. The separate 18/21/25-line supplement is not pooled: rules and v0.5 recover 6/6 links and 3/3 waits; v0.4 recovers 1/6 and 2/3. Two main-set waits remain conservatively blocked. Original v0.4 six-line demo failures remain in the repository.
| Planner | Links | Extra | Waits | Unsupported | Exact plans | Mean |
|---|
Architecture, boundaries & ownership
Read → retrieve → local Gemma interpretation → evidence checks → human review → source-restricted plan → manually confirmed interval → original-source handoff. QA cannot relax the unchanged v0.5 planner's restrictions. Session facts, timers and highlighting run in JavaScript. Prepared inference is labeled recorded.
Original code and SVG artwork: MIT. Gemma weights are not distributed and have separate Google terms. No BioRender figure, Canva template, Base44 backend or Lovable app is claimed as integrated. Only synthetic notes are shipped. No credentials, participant responses, model weights or runtime binaries are included.
Intended source repository — publication pending · Optional raw evidence JSON · Authentic captioned demo