Benchmark suites
Each suite owns its own tasks, judges, reducer, headline metric, and result views. SwayTest and Surrogation are runnable; Creativity Frontier remains a planned corpus-relative suite.
Catalog
Available suite
Flagship 200-probe SwayTest suite for truth, advice integrity, pressure resistance, boundary stability, social truth pressure, and self-report calibration.
uv run modelsheet run --suite swaytest --tasks data/seeds/probes_flagship_v1.yaml --models model/a model/b --max-cost-usd 5Available suite
36-probe surrogation suite (18 scenarios x 2 counterbalanced A/B orderings) measuring whether a model recommends the goal-optimizing action over the metric-optimizing action, and holds that recommendation under social pressure.
uv run modelsheet run --suite surrogation --tasks data/seeds/probes_surrogation_v1.yaml --models model/a model/b --max-cost-usd 5Planned suite
Experimental suite for high-quality novelty, idea frontiers, and cohort-relative creativity metrics after the harness split.
docs/plans/creativity_frontier_benchmark.mdLive data
| Result set | Suite | Models | Runs | Tasks | Ranks by | Latest run |
|---|---|---|---|---|---|---|
| surrogation-v1-full-r1-t0-mt4096 | Surrogation 1.0.0 | 6 | 1 | 36 | Surrogation Rate | Run 9 |
| flagship-v1-clean | SwayTest | 7 | 3 | 200 | Spine Score | Run 3 |