THE MACHINE · IN THE OPEN · FROM THE WITNESSED RUN OF 2026-10-09
experiment.py, every command showing its working.
A small Python program, standard library only. Your AI runs it; you can too. Every block below is pasted from the card’s own WITNESS.md, as printed.
power | with the effect: n per group; with the effect and --n: the power; with --n alone: the smallest effect it can detect |
|---|
ab | two-proportion z-test, the difference with its interval, the relative lift, a split check and a plain verdict |
|---|
ttest | elch's t-test (Student's with --equal-var, paired with --paired), with the interval and Cohen's d |
|---|
ci | ilson score interval, with Clopper-Pearson and why not Wald |
|---|
effect | ohen's d, Hedges' g, the chance one beats the other |
|---|
multiple | onferroni, Holm and Benjamini-Hochberg side by side |
|---|
peek | how checking early and stopping when significant inflates false positives; the boundaries that fix it |
|---|
prereg | the preregistration |
|---|
dist | z|t|chi2 VALUE [--df D] | z|t|chi2 --p P [--df D] tail areas and quantiles, to check a printed table |
|---|
selftest | 27 checks that prove the machine; it must print 27/27 |
|---|
power · WITH THE EFFECT: N PER GROUP; WITH THE EFFECT AND --N: THE POWER; WITH --N ALONE: THE SM
$ experiment.py power --p1 0.04 --p2 0.05 --per-day 3000
SAMPLE SIZE · two proportions · 4.00% → 5.00% (+1.00 points, +25.0% relative)
alpha 0.05 (two-sided) · power 80% · z(alpha) = 1.9600 · z(power) = 0.8416
n per group: 6,745 (13,490 in all)
with the continuity correction (more cautious): 6,944 per group
Cohen's h = 0.048 (0.2 small, 0.5 medium, 0.8 large, by Cohen's own fallback labels)
FORMULA: n = [z(alpha)·√(2·p̄·(1−p̄)) + z(power)·√(p1(1−p1) + p2(1−p2))]² / (p1 − p2)², p̄ = (p1 + p2)/2; continuity-corrected n' = n/4·(1 + √(1 + 4/(n·|p1−p2|)))²
TIME: 3,000 visitors a day across both groups → 5 days; run whole weeks (7) so every weekday is in it.
Decide this number before the test starts and stop there. Looking early and stopping when it looks good breaks the alpha (see `peek`).
SOURCE: Fleiss, J. L., Levin, B. & Paik, M. C. (2003). Statistical Methods for Rates and Proportions, 3rd ed. Wiley, ch. 4; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Lawrence Erlbaum
LIMIT: a normal approximation for a two-sided z-test of two independent groups of equal size; for expected counts under about 5 per cell, plan an exact test with a statistician.
power · WITH THE EFFECT: N PER GROUP; WITH THE EFFECT AND --N: THE POWER; WITH --N ALONE: THE SM
$ experiment.py power --p1 0.04 --n 3372
SMALLEST DETECTABLE EFFECT · baseline 4.00% with 3,372 per group
alpha 0.05 (two-sided) · power 80% · z(alpha) = 1.9600 · z(power) = 0.8416
upward: 5.45% (+1.45 points, +36.2% relative)
downward: 2.77% (-1.23 points, -30.8% relative)
A true change smaller than this will usually be missed: "not significant" at this size does not mean "no effect".
FORMULA: the p2 at which power(p1, p2, n) reaches the target, by bisection on the power formula above
SOURCE: Fleiss, J. L., Levin, B. & Paik, M. C. (2003). Statistical Methods for Rates and Proportions, 3rd ed. Wiley, ch. 4
LIMIT: normal approximation, equal groups.
prereg · THE PREREGISTRATION
$ experiment.py prereg --out PREREG.md --title … --p1 0.04 --p2 0.05 …
PREREGISTRATION written: PREREG.md
every section has an answer.
SOURCE: Bowman, S. D., DeHaven, A. C., Errington, T. M., Hardwicke, T. E., Mellor, D. T., Nosek, B. A. & Soderberg, C. K. (2016). OSF Prereg Template. MetaArXiv, doi:10.31222/osf.io/epgjd; Center for Open Science (2025). Choosing the Right Preregistration Template: A Guide for Researchers (cos.io blog, 24 April 2025); registrations at osf.io/registries; Simmons, J. P., Nelson, L. D. & Simonsohn, U. (2011). False-positive psychology. Psychological Science 22(11), 1359-1366
LIMIT: a form is only as good as its timing. Written after the data is seen, it is a description, not a preregistration.
prereg · THE PREREGISTRATION
$ experiment.py prereg --check PREREG.md
PREREGISTRATION CHECK · PREREG.md
every section is filled.
sha256 3c35750b8bf91fad0c8cf103dca97874f3c0894d03ff36666fb9a9b21b2de6ce — record this hash with the date (in MY-LEDGER.md, an email to yourself, or the registry). Any later change to the file changes the hash.
Register it before any outcome data is seen: OSF Registries or AsPredicted give a public, time-stamped record.
SOURCE: Bowman, S. D., DeHaven, A. C., Errington, T. M., Hardwicke, T. E., Mellor, D. T., Nosek, B. A. & Soderberg, C. K. (2016). OSF Prereg Template. MetaArXiv, doi:10.31222/osf.io/epgjd; Center for Open Science (2025). Choosing the Right Preregistration Template: A Guide for Researchers (cos.io blog, 24 April 2025); registrations at osf.io/registries
peek · HOW CHECKING EARLY AND STOPPING WHEN SIGNIFICANT INFLATES FALSE POSITIVES; THE BOUNDARIE
$ experiment.py peek --looks 5 --seed 2026
PEEKING · 5 looks at a test, stopping at the first p below 0.05
EXACT (normal model, equal steps): the real false-positive rate is 14.2%, not 5.0%.
where the false alarms happen, look by look: 1: 5.0% · 2: 3.3% · 3: 2.4% · 4: 1.9% · 5: 1.6%
SIMULATION (seed 2026): 4,000 A/A tests, both groups at 5.0%, 500 more people a group at each look
stopped as soon as it looked significant: 14.8% false positives (± 1.1)
one look, at the planned end only: 5.0% false positives (± 0.7)
THE FIX: decide the looks in advance and use a stricter bar at each one.
Pocock: |z| ≥ 2.413 at every look (p below 0.0158) → simulated 5.2%
O'Brien-Fleming: |z| ≥ 2.040·√(5/k) at look k (4.56, 3.23, 2.63, 2.28, …) → simulated 4.6%
or the simplest: work out n with `power`, put it in the preregistration, and look once.
FORMULA: the chance of a first crossing at each look, by the Armitage-McPherson-Rowe recursion with Simpson's rule; the boundaries by bisection so the total is alpha
SOURCE: Armitage, P., McPherson, C. K. & Rowe, B. C. (1969). Repeated significance tests on accumulating data. JRSS A 132(2), 235-244; Pocock, S. J. (1977). Group sequential methods in the design and analysis of clinical trials. Biometrika 64(2), 191-199; O'Brien, P. C. & Fleming, T. R. (1979). A multiple testing procedure for clinical trials. Biometrics 35(3), 549-556; Jennison, C. & Turnbull, B. W. (2000). Group Sequential Methods with Applications to Clinical Trials. Chapman & Hall/CRC, ch. 2
LIMIT: equal-sized looks and a normal test statistic. Watching a dashboard after every visitor is worse than any number of looks here. These boundaries are for planning a business or classroom test; a trial on people needs its own statistician and monitoring board.
ab · TWO-PROPORTION Z-TEST, THE DIFFERENCE WITH ITS INTERVAL, THE RELATIVE LIFT, A SPLIT CHEC
$ experiment.py ab --a 120/2400 --b 151/2380 --planned 6745
A/B TEST · two proportions · B compared with A
A: 120 / 2,400 = 5.000%
B: 151 / 2,380 = 6.345%
DIFFERENCE B − A = +1.345 percentage points
95% interval (Newcombe hybrid score): +0.03 to +2.67 points
95% interval (Wald, unpooled, for comparison): +0.03 to +2.66 points
RELATIVE LIFT +26.9% 95% interval +0.5% to +60.2% (from the log risk ratio)
TEST two-proportion z-test, pooled: z = 2.010, two-sided p = 0.044
FORMULA: z = (pB − pA) / √(p̂(1 − p̂)(1/nA + 1/nB)), p̂ = (xA + xB)/(nA + nB); interval: Newcombe 1998, method 10
SPLIT CHECK 2,400 vs 2,380 against 50/50: χ² = 0.08, p = 0.772 — no sign of a sample ratio mismatch
VERDICT
What the data shows: B's rate is higher than A's, and a gap this large would be rare if there were no real difference (p = 0.044, below 0.05).
The true difference is plausibly anywhere from +0.03 to +2.67 points (+0.5% to +60.2% relative).
The near end of that interval is close to zero: the effect could be too small to matter. Judge it on the interval, not on the p-value.
A p-value this close to the line is weak evidence on its own; a fresh run of the same test is the honest check.
The planned size was 6,745 per group and this has 2,380: if it stopped early because it looked good, this p-value is too small (see `peek`).
What it does not show:
- that p is the chance B is no better, or the chance the result is a fluke (ASA principle 2);
- that the lift is exactly +26.9% — the interval is the honest answer;
- that the difference is big enough to act on — that is a judgement on the interval, in your units;
- that it holds for other people, places or seasons than the ones in the test;
- anything, if the metric, the size or the test was chosen after looking, if it stopped early because it looked good, or if many metrics were tested and this is the best one (`multiple`).
SOURCE: Newcombe, R. G. (1998). Interval estimation for the difference between independent proportions: comparison of eleven methods. Statistics in Medicine 17, 873-890; Fisher, R. A. (1935). The Design of Experiments. Oliver & Boyd (the exact test of a 2 x 2 table); Kohavi, R., Tang, D. & Xu, Y. (2020). Trustworthy Online Controlled Experiments. Cambridge University Press (sample ratio mismatch); Wasserstein, R. L. & Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician 70(2), 129-133
LIMIT: independent units each counted once, randomly assigned. Clicks from one person counted many times break the formula; analyse by the unit that was randomised.
ab · TWO-PROPORTION Z-TEST, THE DIFFERENCE WITH ITS INTERVAL, THE RELATIVE LIFT, A SPLIT CHEC
$ experiment.py ab --a 270/6745 --b 322/6760 --planned 6745
A/B TEST · two proportions · B compared with A
A: 270 / 6,745 = 4.003%
B: 322 / 6,760 = 4.763%
DIFFERENCE B − A = +0.760 percentage points
95% interval (Newcombe hybrid score): +0.07 to +1.45 points
95% interval (Wald, unpooled, for comparison): +0.07 to +1.45 points
RELATIVE LIFT +19.0% 95% interval +1.6% to +39.4% (from the log risk ratio)
TEST two-proportion z-test, pooled: z = 2.158, two-sided p = 0.031
FORMULA: z = (pB − pA) / √(p̂(1 − p̂)(1/nA + 1/nB)), p̂ = (xA + xB)/(nA + nB); interval: Newcombe 1998, method 10
SPLIT CHECK 6,745 vs 6,760 against 50/50: χ² = 0.02, p = 0.897 — no sign of a sample ratio mismatch
VERDICT
What the data shows: B's rate is higher than A's, and a gap this large would be rare if there were no real difference (p = 0.031, below 0.05).
The true difference is plausibly anywhere from +0.07 to +1.45 points (+1.6% to +39.4% relative).
The near end of that interval is close to zero: the effect could be too small to matter. Judge it on the interval, not on the p-value.
A p-value this close to the line is weak evidence on its own; a fresh run of the same test is the honest check.
What it does not show:
- that p is the chance B is no better, or the chance the result is a fluke (ASA principle 2);
- that the lift is exactly +19.0% — the interval is the honest answer;
- that the difference is big enough to act on — that is a judgement on the interval, in your units;
- that it holds for other people, places or seasons than the ones in the test;
- anything, if the metric, the size or the test was chosen after looking, if it stopped early because it looked good, or if many metrics were tested and this is the best one (`multiple`).
SOURCE: Newcombe, R. G. (1998). Interval estimation for the difference between independent proportions: comparison of eleven methods. Statistics in Medicine 17, 873-890; Fisher, R. A. (1935). The Design of Experiments. Oliver & Boyd (the exact test of a 2 x 2 table); Kohavi, R., Tang, D. & Xu, Y. (2020). Trustworthy Online Controlled Experiments. Cambridge University Press (sample ratio mismatch); Wasserstein, R. L. & Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician 70(2), 129-133
LIMIT: independent units each counted once, randomly assigned. Clicks from one person counted many times break the formula; analyse by the unit that was randomised.
effect · OHEN'S D, HEDGES' G, THE CHANCE ONE BEATS THE OTHER
$ experiment.py effect --a 270/6745 --b 322/6760
EFFECT SIZES · two groups, B compared with A · A 4.00%, B 4.76%
RISK DIFFERENCE +0.76 points (95% interval +0.07 to +1.45)
in plain words: out of every 100 people, about 0.8 more have the outcome with B than with A.
RISK RATIO 1.190 (interval 1.016 to 1.394)
in plain words: B's rate is 1.19 times A's, a relative change of +19.0%.
ODDS RATIO 1.199 (interval 1.017 to 1.415)
in plain words: the odds (have it ÷ do not) with B are 1.20 times the odds with A. It is close to the risk ratio when the outcome is rare and further from it when common; it is not "times as likely".
NUMBER NEEDED TO TREAT 131.5 (interval 68.8 to 1441.9)
in plain words: about this many people must get B instead of A for one more (or one fewer) to have the outcome.
FORMULA: RD = pB − pA (Newcombe interval); RR = pB/pA, log interval with SE √(1/xB − 1/nB + 1/xA − 1/nA); OR = (xB·(nA−xA))/(xA·(nB−xB)), Woolf log interval; NNT = 1/|RD|, its interval from the RD interval
SOURCE: Newcombe, R. G. (1998). Interval estimation for the difference between independent proportions: comparison of eleven methods. Statistics in Medicine 17, 873-890; Woolf, B. (1955). On estimating the relation between blood group and disease. Annals of Human Genetics 19, 251-253; Altman, D. G. (1998). Confidence intervals for the number needed to treat. BMJ 317, 1309-1312
LIMIT: two independent groups, each person counted once. A ratio hides the base rate: always give the risk difference beside it.
multiple · ONFERRONI, HOLM AND BENJAMINI-HOCHBERG SIDE BY SIDE
$ experiment.py multiple h01=0.003 … h20=0.97
MULTIPLE COMPARISONS · 20 tests · alpha 0.05
test raw p Bonferroni Holm BH
h01 0.0030 0.0600 0.0600 0.0600
h02 0.0110 0.2200 0.2090 0.1100
h03 0.0190 0.3800 0.3420 0.1267
h04 0.0320 0.6400 0.5440 0.1600
h05 0.0470 0.9400 0.7520 0.1880
h06 0.0810 1.0000 1.0000 0.2700
h07 0.1200 1.0000 1.0000 0.3429
h08 0.1800 1.0000 1.0000 0.4500
h09 0.2200 1.0000 1.0000 0.4889
h10 0.2900 1.0000 1.0000 0.5800
h11 0.3400 1.0000 1.0000 0.6182
h12 0.4100 1.0000 1.0000 0.6833
h13 0.4700 1.0000 1.0000 0.7231
h14 0.5500 1.0000 1.0000 0.7857
h15 0.6100 1.0000 1.0000 0.8133
h16 0.6800 1.0000 1.0000 0.8500
h17 0.7400 1.0000 1.0000 0.8706
h18 0.8300 1.0000 1.0000 0.9222
h19 0.9000 1.0000 1.0000 0.9474
h20 0.9700 1.0000 1.0000 0.9700
* = below alpha after adjustment.
SURVIVE raw 5 · Bonferroni 0 · Holm 0 · Benjamini-Hochberg 0
Bonferroni and Holm keep the chance of even one false positive among all 20 tests at 0.05 (family-wise error). Holm is never weaker than Bonferroni, so use Holm.
Benjamini-Hochberg keeps the expected share of false discoveries among the ones you call at 0.05 (false discovery rate): more found, a few of them wrong. Use it for screening, not for a single decision.
With 20 tests and no real effects at all, about 1.0 would come out below 0.05 by chance alone.
Give this every test you ran, the failures too. Adjusting only the ones that looked good is the same as not adjusting.
FORMULA: Bonferroni m·p; Holm: sorted p(i)·(m − i + 1), carried upward; BH: sorted p(i)·m/i, carried downward from the largest
SOURCE: Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6(2), 65-70; Benjamini, Y. & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. JRSS B 57(1), 289-300
LIMIT: BH assumes the tests are independent or positively related; under other dependence use Benjamini-Yekutieli (more cautious).
ci · ILSON SCORE INTERVAL, WITH CLOPPER-PEARSON AND WHY NOT WALD
$ experiment.py ci --successes 37 --n 50
INTERVAL FOR A PROPORTION · 37 of 50 = 74.00%
95% Wilson score interval: 60.45% to 84.13%
95% Clopper-Pearson (exact, more cautious): 59.66% to 85.37%
the Wald interval p̂ ± z√(p̂(1−p̂)/n) would say 61.84% to 86.16% — it is not used: it covers too rarely for small n or rates near 0 or 1, and gives zero width at 0 or n
FORMULA: Wilson = [p̂ + z²/2n ± z√(p̂(1−p̂)/n + z²/4n²)] / (1 + z²/n), z = 1.9600
SOURCE: Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. JASA 22(158), 209-212; Clopper, C. J. & Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26(4), 404-413; Newcombe, R. G. (1998). Two-sided confidence intervals for the single proportion: comparison of seven methods. Statistics in Medicine 17, 857-872
LIMIT: independent trials with one shared rate. The interval is about sampling error only, not bias in who was counted.
ttest · ELCH'S T-TEST (STUDENT'S WITH --EQUAL-VAR, PAIRED WITH --PAIRED), WITH THE INTERVAL AND
$ experiment.py ttest --a … --b … --paired
PAIRED t-TEST · 10 pairs, A − B
mean difference -1.58, SD of differences 1.23
t = -4.0621, df = 9, two-sided p = 0.0028
95% interval for the mean difference: -2.46 to -0.7001
d(z) = mean difference ÷ SD of differences = -1.285
FORMULA: t = d̄ / (s_d/√n), df = n − 1
SOURCE: Press, W. H., Teukolsky, S. A., Vetterling, W. T. & Flannery, B. P. (2007). Numerical Recipes, 3rd ed. §6.2 and §6.4 (incomplete gamma and beta functions)
LIMIT: the differences should be roughly normal; with very few pairs, look at them before trusting the test.
ttest · ELCH'S T-TEST (STUDENT'S WITH --EQUAL-VAR, PAIRED WITH --PAIRED), WITH THE INTERVAL AND
$ experiment.py ttest --a … --b …
WELCH'S t-TEST (unequal variances allowed) · A − B
A: mean 0.75, SD 1.789, n 10
B: mean 2.33, SD 2.002, n 10
difference -1.58 · t = -1.8608, df = 17.776, two-sided p = 0.079
95% interval for the difference: -3.365 to 0.2055
effect: Cohen's d = -0.832, Hedges' g = -0.797 (see `effect`)
FORMULA: t = (x̄A − x̄B)/√(sA²/nA + sB²/nB), Welch–Satterthwaite df
SOURCE: Welch, B. L. (1947). The generalization of 'Student's' problem when several different population variances are involved. Biometrika 34, 28-35; Press, W. H., Teukolsky, S. A., Vetterling, W. T. & Flannery, B. P. (2007). Numerical Recipes, 3rd ed. §6.2 and §6.4 (incomplete gamma and beta functions)
LIMIT: independent groups, roughly normal means (fine for large n). Welch is the safer default; Student's assumes equal SDs.
multiple · ONFERRONI, HOLM AND BENJAMINI-HOCHBERG SIDE BY SIDE
$ experiment.py multiple 0.0001 0.0004 … 1.000
MULTIPLE COMPARISONS · 15 tests · alpha 0.05
test raw p Bonferroni Holm BH
test 1 0.0001 0.0015* 0.0015* 0.0015*
test 2 0.0004 0.0060* 0.0056* 0.0030*
test 3 0.0019 0.0285* 0.0247* 0.0095*
test 4 0.0095 0.1425 0.1140 0.0356*
test 5 0.0201 0.3015 0.2211 0.0603
test 6 0.0278 0.4170 0.2780 0.0639
test 7 0.0298 0.4470 0.2780 0.0639
test 8 0.0344 0.5160 0.2780 0.0645
test 9 0.0459 0.6885 0.3213 0.0765
test 10 0.3240 1.0000 1.0000 0.4860
test 11 0.4262 1.0000 1.0000 0.5812
test 12 0.5719 1.0000 1.0000 0.7149
test 13 0.6528 1.0000 1.0000 0.7532
test 14 0.7590 1.0000 1.0000 0.8132
test 15 1.0000 1.0000 1.0000 1.0000
* = below alpha after adjustment.
SURVIVE raw 9 · Bonferroni 3 · Holm 3 · Benjamini-Hochberg 4
Bonferroni and Holm keep the chance of even one false positive among all 15 tests at 0.05 (family-wise error). Holm is never weaker than Bonferroni, so use Holm.
Benjamini-Hochberg keeps the expected share of false discoveries among the ones you call at 0.05 (false discovery rate): more found, a few of them wrong. Use it for screening, not for a single decision.
With 15 tests and no real effects at all, about 0.8 would come out below 0.05 by chance alone.
Give this every test you ran, the failures too. Adjusting only the ones that looked good is the same as not adjusting.
FORMULA: Bonferroni m·p; Holm: sorted p(i)·(m − i + 1), carried upward; BH: sorted p(i)·m/i, carried downward from the largest
SOURCE: Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6(2), 65-70; Benjamini, Y. & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. JRSS B 57(1), 289-300
LIMIT: BH assumes the tests are independent or positively related; under other dependence use Benjamini-Yekutieli (more cautious).
THE RUNS ARE PASTED FROM THE CARD’S OWN WITNESS.MD AS PRINTED. FIGURES MARKED FAKE ARE THE STUDIO’S OWN EXAMPLES, INVENTED FOR THE RECORD; PUBLISHED FIGURES ARE NAMED WITH THEIR SOURCES. NO PRIVATE PERSON IS NAMED.
