zperf evidence policy
Gate
A zperf rule may be documented / treated as a speed error (cperf/jsperf-style escalate) only if both exist under packages/zperf/evidence/:
<RULE>.md— claim, harness, Zig version / optimize mode, limits<RULE>.jsonl— ≥1 before/after line with fieldsrule,case,role,zig,optimize,median_ms,ops,ratio,ts
Otherwise keep the rules.zig severity (warning / info / hint / off) and treat as review prompt, not a “must fix for speed” CI gate.
Having JSONL is necessary, not sufficient. Escalate only when wall-clock clearly agrees and the failure mode matches the rule. Modest ratios, env-dependent I/O, or optimizer-elided benches stay non-error.
CLI today may still fail the process on any finding; this policy is the keep / escalate contract until presets land.
Current map
As of 2026-10-22: measured with Zig 0.17.0 -Ofast, median of 9, harness examples/bench/evidence_fill.zig.
G3/r4 (2026-10-22): several rules demoted to default off after OK-idiom mass on mature Zig corpora (see corpus.md; gate definitions in AGENT.md). The evidence JSONL for these rules stays valid; do not escalate to error without re-validating G4 on the default-on set.
| Rule | rules.zig sev | Evidence | Measured ratio | Escalate? |
|---|---|---|---|---|
| zp-alloc-hot | off | zp-alloc-hot | ~1.4× GPA vs arena retain | no (modest; prefer zp-create-destroy-hot / zp-fixed-size-alloc-hot for fixed sizes) |
| zp-append-without-reserve | off | zp-append-without-reserve | ~2.1× | no (info) |
| zp-page-allocator-small | warning | zp-page-allocator-small | ~341× page vs GPA | candidate error |
| zp-print-hot | off | zp-print-hot | ~6e4× print vs quiet | no — env I/O (like jsperf's jp-console-hot) |
| zp-unbuffered-io | off | zp-unbuffered-io | ~49× empty vs 4KiB | candidate error |
| zp-aos-single-field | off | zp-aos-single-field | ~6.6× AoS vs SoA | no (review prompt) |
| zp-bool-array | off | zp-bool-array | ~30× bool[] vs bitset | no (info) |
| zp-static-string-map | info | zp-static-string-map | ~2.5× | no (info) |
| zp-runtime-div-hot | off | zp-runtime-div-hot | ~12× mod vs mask | no (review prompt) |
| zp-queue-ordered-remove | warning | zp-queue-ordered-remove | ~466× orderedRemove vs ring | candidate error |
| zp-shift-on-read | warning | zp-shift-on-read | ~17× | candidate error |
| zp-per-packet-alloc | off | zp-per-packet-alloc | shared zp-alloc-hot ~1.4× | no (same as zp-alloc-hot) |
| zp-invariant-float-div | hint | zp-invariant-float-div | ~2.0× | stays hint |
| zp-map-put-without-ensure | off | zp-map-put-without-ensure | ~2.8× | no (info) |
| zp-arena-reset-free-all | warning | zp-arena-reset-free-all | ~3.2× free_all vs retain | borderline warning |
| zp-create-destroy-hot | info | zp-create-destroy-hot | ~11× | candidate error |
| zp-fixed-size-alloc-hot | info | zp-fixed-size-alloc-hot | ~8.5× | candidate error |
| zp-open-file-hot | hint | zp-open-file-hot | ~2134× open once | no (hint; path may change) |
| zp-clock-hot | hint | zp-clock-hot | ~42× sample once | no (hint; a fresh timestamp has nothing to hoist) |
| zp-json-hot | hint | zp-json-hot | ~2683× parse once | no (hint; a changing payload cannot be hoisted) |
| zp-alloc-print-hot | hint | zp-alloc-print-hot | ~50× GPA print once; arena ~12× | no (hint; a changing value cannot be hoisted) |
Not measured / N/A this pass
| Rule | Status | Why |
|---|---|---|
| zp-debug-allocator | N/A (static-only) | Build-mode allocator selection; not one-binary before/after |
| zp-large-struct-by-value | inconclusive | ReleaseFast elided by-value vs ptr microbench (~50 ns total) — do not cite |
| zp-missing-free | N/A (correctness) | Missing free/deinit — leak/UB, not speed |
| zp-broad-runtime-safety | pending measure | Broad @setRuntimeSafety(false) — only meaningful under ReleaseSafe |
| zp-packable-bools | pending measure | Packable bools — density claim; no wall JSONL yet |
| zp-usize-index-field | pending measure | usize index fields — density; no wall JSONL yet |
| zp-catch-unreachable-alloc | N/A (correctness) | catch unreachable on alloc — UB on OOM |
| zp-oversized-enum-tag | N/A (static size) | Oversized enum tag — ABI/size; no wall JSONL |
| zp-optional-scalar-field | pending measure | Optional scalar padding — density; no wall JSONL yet |
| zp-catch-unreachable-io | N/A (correctness) | catch unreachable on I/O |
Regenerating
bash packages/zperf/scripts/regen-evidence.shRe-run after Zig minor upgrades; update matching .md medians if ratios move materially.
Candidate errors (policy, not CLI presets yet)
Strong agreement in these measurements: zp-page-allocator-small, zp-queue-ordered-remove, zp-shift-on-read, zp-create-destroy-hot, zp-fixed-size-alloc-hot (zp-unbuffered-io demoted off after G3 OK-idiom).
Keep as warning/info until --preset lands; do not invent error in CI without G4 + preset. Default-on after r4: zp-page-allocator-small, zp-broad-runtime-safety, zp-static-string-map, zp-queue-ordered-remove, zp-shift-on-read, zp-invariant-float-div, zp-arena-reset-free-all, zp-create-destroy-hot, zp-fixed-size-alloc-hot.