Run and assess answers
Keep the question, first answer and judgment traceable.
Offline human-attested prose scoring is available. No model performance has been measured; no AI API or untrusted-Lean executor is connected.
- Choose first. Select core IDs, model/version, retrieval policy and tools before collecting answers. Give the model only the prompt-only records.
- Freeze the run. Save raw first responses and metadata. The recorder binds prompts, references, rubric and answers by SHA-256; missing answers stay in the denominator.
- Have a human assess it. Fill each frozen rubric criterion with 0/1 and a reason. The report records those judgments, not automatically verified mathematics or authenticated grader identity.
Commands and input formats
From the repository root, Python 3.11+:
python tools/sog_bench_prose.py prepare --submissions responses.jsonl --run-metadata run.json --output .astis/sog-bench/runs/my-runCopy the generated adjudication-template.json to completed.json. An actual human supplies grader identity, timestamp and all four criterion judgments per nonempty answer.
python tools/sog_bench_prose.py score --packet .astis/sog-bench/runs/my-run --adjudication completed.json --output .astis/sog-bench/runs/my-run-score.jsonRun-metadata template · Complete recording and scoring instructions
Other tracks and scientific limits
Graph JSON has deterministic exact-snapshot scoring; it certifies no mathematical necessity. Lean reference compilation checks trusted references only, not arbitrary submissions.
Human assessments require actual mathematical checking. No keyword grader or unvalidated AI grader is supplied. Run execution, retrieval compliance and human identity are not authenticated by this recorder.
All current tasks are public-dev. These records cannot demonstrate unseen generalization, difficulty calibration or private-family evaluation. A paired core is not two independent samples.
Reviewed prompt-only tasks · Full protocol · Structural graph QA