Methodology
Five gates stand between a phone run and a published number. Each one exists because skipping it corrupts the statistics.
1Physical measurement
Every candidate starts as a real run on a real phone: tokens, elapsed time, device class, runtime, and workload recorded together. Self-reported files that cannot prove physical execution never enter the qualified population.
2Validation and deduplication
Submissions are schema-checked, hash-verified, and deduplicated by content: re-sending identical bytes returns the original receipt instead of a second row. Malformed or oversized envelopes are rejected before they can consume quota.
3Consent and withdrawal
Contribution needs explicit per-submission consent after previewing the exact shared fields. Withdrawal is terminal: revoked rows are tombstoned and excluded from every surface. Consent policy content is still under review — no real consent version is approved yet.
4Exact-cohort grouping
Runs are grouped only when they match on every comparability field: model hash, device class, runtime, backend, quantization, workload, prompt hash, context, threads, tokens, sampler, optimization, and build provenance. Differ in one field, and the runs are never averaged together.
5Quorum-gated publication
A cohort publishes only with 5+ eligible runs from 3+ independent installations. Below that: insufficient evidence — counts visible, statistics withheld. Never zeros, never partial means, never a pooled average across incompatible runs.
Synthetic data stays synthetic
Staging runs on synthetic fixtures marked test-only. They can never satisfy genuine quorum and are filtered from every public aggregate. No synthetic measurement is ever presented as device evidence.