Equivalence testing and run-count design for stochastic machine-learning experiments
Claims that two training methods perform equally well often rest on a non-significant difference between two or three runs per method. We develop a protocol, with its power theory, for certifying practical equivalence in per-run metrics such as the AUROC (area under the receiver operating characteristic curve). It rests on a two-level (seed x item) variance model separating between-run dispersion from item-level noise. Equivalence within a pre-specified margin is tested by two one-sided tests (TOST), equivalently by a confidence interval inside the margin; a two-level bootstrap covers composite statistics. A closed-form formula converts a margin and a target power into runs per method; it...
| Authors | Dmytro Knopov, Nadiya Shvai |
|---|---|
| Primary category | cs.LG |
| Published | 16 October 2026 |
| Last updated at source | 16 October 2026 |
Popularity, last 6 days
0 now. Counts as reported by the source, snapshotted daily by Silytics.
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 24 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.