The power behind the ship floor

A step-by-step calculation for Lost Logs, Found Lift, part I

Part I states that, with about 140,000 products per arm and a 3.2% baseline yield, a true 6% relative ITT lift gives 81% power to reject a lift of zero and 65.5% power to show that the lift exceeds the +1.0% ship floor. This page derives both numbers, the sample size the floor would need, and the 18% behind the breach-only return guardrail.

Notation and method

The page is a normal-approximation (Wald) power calculation for a log risk ratio, with the standard error from the delta method. Each section states the formula, then puts in the case values.

Case values: p_C = 0.032, L = 0.06, m = 0.01, n = 140,000, α = 0.05, target power 0.80. The page writes the quantiles as 1.96 and 0.8416 so that every product can be checked by hand. With full-precision quantiles the one figure that moves is the sample size in section 7, 197,446 instead of 197,448 per arm. Outcomes are treated as independent Bernoulli draws with a common probability within each arm. The +1.0% floor is a relative improvement, 0.032 percentage points on the 3.2% baseline.

1. Translate the assumed lift into yields

Relative lift is the proportional change in yield, and the expected number of finished products in an arm is the yield times the products in it:

Case values: p_T = 0.032 × 1.06 = 0.03392.

The new-station count is an expectation, which is why it is not a whole number. Real counts vary from one experiment to the next.

2. Work on the log scale

The risk ratio, its log, and the conversion back to a lift:

Source: the log of the risk ratio as the scale for its interval is the log method of Katz, Baptista, Azen and Pike (1978) [1].

The case builds its interval on the log of the ratio of the two yields and converts the bounds back at the end. The ratio itself has a skewed sampling distribution. Its logarithm is close to normal, which is what a normal-approximation interval needs.

Case values: at the assumed rates, θ = ln(1.06) = 0.058269.

3. The standard error of the estimate

A yield estimate from n independent products has variance

and the delta method carries that variance through a smooth transformation g:

With g(p) = ln p and g'(p) = 1/p, the variance of ln p̂ is approximately (1 − p)/(n p). The two arms are independent, so the variances of the two logs add, and the standard error of the log risk ratio is

For planning, the true yields go into s. The interval computed on observed data uses the observed yields instead.

Source: the delta method is textbook asymptotics (Agresti 2013, section 16.1 [2]). Applied to the log ratio of two binomial proportions it gives the Katz variance (1 − p_T) / (n p_T) + (1 − p_C) / (n p_C) [1].

Case values:

For planning, the estimate is treated as normal around the true value with this fixed spread: θ ∼ N(θ, s²).

4. Power to reject a lift of zero: 81.2%

For a point null θ = θ₀ the Wald statistic is Z = (θ − θ₀) / ŝ with ŝ computed from the observed yields, and a two-sided test at level α rejects when |Z| > z_{1 − α/2}. Under the planning approximation Z ∼ N(d, 1) with d = (θ − θ₀) / s, so

Source: the normal-approximation framework for power is Lachin (1981) [3], and its application to the log risk ratio is the log method of Blackwelder (1993) [4]. The expression as written here, with the planning s in place of ŝ, is this page's derivation from them.

Case values. A lift of zero is a ratio of 1, so θ₀ = 0. A two-sided test at 5% rejects when the estimate is more than 1.96 standard errors from zero:

Under the assumed 6% lift, θ/s is centred on θ / s = 0.058269 / 0.020482 = 2.8449, so

The second term, the chance of a significant result in the wrong direction, is below 0.0001%, so the power is 81.2%, reported as 81%.

So the estimate must exceed 1.96 × 0.020482 = 0.040145 on the log scale, a measured lift of about +4.1%, and a true 6% lift produces such an estimate 81% of the time.

5. Power to clear the +1.0% floor: 65.5%

The two-sided 100(1 − α)% interval for the log risk ratio is θ ± z_{1 − α/2} s. Exponentiating gives the interval for the ratio, and subtracting 1 gives the interval for the lift. The rule passes when the lower bound clears the floor θ₀ = ln(1 + m):

Source: testing against a margin instead of zero is the design of Blackwelder (1982) [5]. The asymptotic power of the log method with a null ratio other than 1 is in Blackwelder (1993) [4].

This is a one-sided superiority test against a margin at level α / 2, because the cutoff comes from a two-sided interval.

Case values. The ship rule asks for more: the lower bound of the 95% interval must exceed a ratio of 1.01.

So the estimate must now exceed a measured lift of about +5.1%. The true effect sits

standard errors above the floor, and

The critical value stays at 1.96, because the rule reads the lower end of a two-sided 95% interval, a one-sided 2.5% test. A one-sided 5% test would use 1.645 and report higher power than the rule delivers.

6. Why the same estimate gives less power

The spread of the estimate is the same in both calculations. Only the bar moves, from zero to the +1.0% floor, which takes 17% off the distance between the true effect and the bar (0.058269 down to 0.048319). Because the standardised distance falls from 2.84 to 2.36, power falls from 81% to 65.5%.

7. Sample size for 80% power against the floor

Power 1 − β needs the standardised distance to reach z_{1 − α/2} + z_{1 − β}. Substituting s = √V / n and solving for n gives

Source: the general form n = V (z_{1 − α/2} + z_{1 − β})² / (θ − θ₀)² follows Lachin (1981) [3]. With the Katz variance and a null ratio other than 1 it is the log-method sample size of Blackwelder (1993) [4]. The single-variance form here, evaluated at the alternative, is this page's derivation from those sources.

In words: the noise, times the squared sum of the evidence and power quantiles, divided by the squared gap between the true effect and the floor, per arm and rounded up. It needs L > m.

Case values. Setting the standardised distance equal to 1.96 + 0.8416 = 2.8016 (the 97.5% and 80% normal quantiles) and solving for n:

That is the "about 197,000 products per arm" in Part I, about 41% more than the planned 140,000. The same formula against zero gives 135,771 per arm for 80% power. Section 9 uses that figure.

8. The return guardrail: why a pass is not required

Part I says the return guardrail is breach-only because, at this volume, a harmless change would pass its +10% margin only about 18% of the time. The same machinery gives that number. The guardrail is read on finished products, so its sample is the yield's numerator, a few thousand per arm, and not the 140,000 products.

A guardrail passes when the upper bound of its 95% interval is at or below the margin m_g. At a true change of zero the estimate is centred on zero with standard error s_g, so

with r the return rate and n_f the finished products per arm. Solving the same equation for a target pass rate 1 − β gives n_f = 2(1 − r)r(z_{1 − α/2} + z_{1 − β}ln(1 + m_g))².

Source: the same Wald machinery as sections 5 and 7 [3, 4], with the pass condition of a non-inferiority test [5] evaluated at a true change of zero.

Case values. A +10% relative margin is a ratio of 1.10: a 5% return rate may rise to 5.5% before it counts as harm. The interim cohort finishes 140,000 × 0.032 = 4,480 products per arm, the case's "about 4,500", with the design return rate r = 0.05 and margin m_g = 0.10:

That 17.7% is the "about 18%" in Part I.

An 80% pass rate at no true change would need n_f = 38 × (2.8016 / 0.095310)² = 32,834 finished products per arm, about 1.0 million products per arm at the 3.2% yield. At this sample the pass test is too insensitive to certify anything, so the design uses the guardrail for what it can show. A breach, the lower bound above +10%, stops the launch. A pass is not a condition for shipping, and an inconclusive result does not show that returns are within the margin. Two simplifications: both arms are given the old-station count of finished products (under the planning lift the new station finishes about 4,749), and returns are compared among finished products, a population the new station changes if it changes completion.

9. Where the planning lift of 6% comes from, and why the power is 81%

Three quantities on this page are chosen. The rest follow from them.

Section 4 gives power as 1 − Φ(z_{1 − α/2} − θ / s). Setting it to the target 1 − β and solving for the effect gives the smallest log ratio the sample detects with that power against zero (the opposite tail, below 10⁻⁶ here, is dropped),

where s depends on L through p_T = p_C (1 + L), so the equation is solved by iteration: evaluate s at a guess, update L, repeat until it settles. Against the floor the same step starts from θ₀: θ_min = θ₀ + (z_{1 − α/2} + z_{1 − β})  s.

Source: the minimum detectable effect as the inversion of the power formula is Kohavi, Tang and Xu (2020), chapter 17 [6], whose equation 17.8 is the mean-difference shortcut n ≈ 16σ² / δ² (after van Belle 2008). The log-ratio inversion here, where s depends on the effect, is this page's application of it, and n on this page is per arm. The 80% convention is Cohen (1992) [7].

Case values. At n = 140,000 and the 6% guess, s = 0.020482, so θ_min = 2.8016 × 0.020482 = 0.057382, a lift of 5.906%. Re-evaluating s at that lift gives s = 0.020486 and

At 136,000 per arm the same calculation gives 5.995%, which is the design note's "80% power at about 136,000". The planning lift of 6% is that minimum detectable effect rounded to a whole number. The simulated effect is a separate input and plays no part in it.

The 81% follows from those choices. At 6% the sample of 140,000 is 4,229 products per arm above the 135,771 that 80% needs (section 7), and the surplus lifts power from 80.0% to 81.2%.

Against the +1.0% floor the inversion gives θ_min = 0.009950 + 2.8016 × 0.020436 = 0.067204, a lift of 6.951%: the sample would need a true lift of about 7% for 80% power against the floor, which is the sample-size shortfall of section 7 read the other way round.

10. What the calculation leaves out

• It covers the primary metric at the interim look only. Guardrail failures, decision blockers and the combined interim and final looks are outside it, so it is not the overall probability of shipping.
• It assumes the planning lift of 6%. At the simulated +2.55% ITT lift, the case reports that the ship rule clears the floor in about 11% of runs at the interim look.
• It is a normal approximation with the variance evaluated at the assumed rates, and it treats products as independent. Clustered outcomes would widen s and lower both powers. Blackwelder (1993) [4] reports that this asymptotic log-method power can be off at small event counts. The interim cohort's 4,480 expected completions per arm are far from that regime. The return guardrail's 224 expected returns per arm are closer to it.
• The return calculation in section 8 fixes both finished-product counts at the old-station figure and compares returns among finished products only, a population that changes if completion changes.

Reproduce it

Python 3.14, standard library only. The comments after each print show the output.

# Python 3.14, standard library only
from math import log, sqrt
from statistics import NormalDist

nd = NormalDist()
z, z80 = 1.96, 0.8416  # nd.inv_cdf(0.975) and nd.inv_cdf(0.80), rounded as on this page
n, p_c, lift, floor = 140_000, 0.032, 0.06, 0.01
p_t = p_c * (1 + lift)
s = sqrt((1 - p_t) / (n * p_t) + (1 - p_c) / (n * p_c))
theta = log(1 + lift)
power_zero = 1 - nd.cdf(z - theta / s) + nd.cdf(-z - theta / s)
power_floor = 1 - nd.cdf(z - (theta - log(1 + floor)) / s)
v = (1 - p_t) / p_t + (1 - p_c) / p_c  # V on this page: n * s ** 2
n_80 = v * ((z + z80) / (theta - log(1 + floor))) ** 2
print(f"s={s:.6f} zero={power_zero:.4f} floor={power_floor:.4f} n80={n_80:,.0f}")
# s=0.020482 zero=0.8119 floor=0.6551 n80=197,448

def mde(nn, fl=0.0, target=0.80):
    # smallest relative lift with the target power against the floor fl, by bisection on [0, 0.5]
    lo, hi = 0.0, 0.5
    for _ in range(60):  # 60 halvings take the bracket far below float precision
        m = (lo + hi) / 2
        pm = p_c * (1 + m)
        sm = sqrt((1 - pm) / (nn * pm) + (1 - p_c) / (nn * p_c))
        power = 1 - nd.cdf(z - (log(1 + m) - log(1 + fl)) / sm)
        if power < target:
            lo = m
        else:
            hi = m
    return hi
print(f"mde zero={100 * mde(n):.3f}% at 140,000, {100 * mde(136_000):.3f}% at 136,000, mde floor={100 * mde(n, floor):.3f}%")
# mde zero=5.907% at 140,000, 5.995% at 136,000, mde floor=6.951%

r, m_g, n_f = 0.05, 0.10, n * p_c  # return guardrail on finished products
s_g = sqrt(2 * (1 - r) / (n_f * r))
print(f"return pass at no change={nd.cdf(log(1 + m_g) / s_g - z):.3f}, 80% needs n_f={2 * (1 - r) / r * ((z + z80) / log(1 + m_g)) ** 2:,.0f}")
# return pass at no change=0.177, 80% needs n_f=32,834

The inputs are the case's design settings (repository README, Power paragraph). The estimator is the delta-method interval on the log ratio that the case uses throughout.

References

1. Katz, D., Baptista, J., Azen, S. P. and Pike, M. C. (1978), "Obtaining confidence intervals for the risk ratio in cohort studies", Biometrics 34(3), 469 to 474. The log-ratio interval with variance (1 − p_T)/(n p_T) + (1 − p_C)/(n p_C). Sections 2, 3 and 5.
2. Agresti, A. (2013), Categorical Data Analysis, 3rd edition, Wiley, section 16.1, "Delta method". Section 3.
3. Lachin, J. M. (1981), "Introduction to sample size determination and power analysis for clinical trials", Controlled Clinical Trials 2(2), 93 to 113. The normal-approximation power and sample-size formulas. Sections 4, 7 and 8.
4. Blackwelder, W. C. (1993), "Sample size and power for prospective analysis of relative risk", Statistics in Medicine 12(7), 691 to 698. Power and sample size for the log risk ratio, including a null ratio other than 1. Sections 4, 5, 7 and 8. It also reports that the asymptotic log-method formula can understate power when both ratios are below 1, one reason section 10 calls these numbers approximations.
5. Blackwelder, W. C. (1982), "'Proving the null hypothesis' in clinical trials", Controlled Clinical Trials 3(4), 345 to 353. Testing against a margin instead of zero. Sections 5 and 8.
6. Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, chapter 17, "The Statistics behind Online Controlled Experiments". Power, the minimum detectable effect and equation 17.8, n ≈ 16σ²/δ², which the chapter attributes to van Belle (2008), Statistical Rules of Thumb, 2nd edition, Wiley. Section 9.
7. Cohen, J. (1992), "A power primer", Psychological Bulletin 112(1), 155 to 159. Power of 0.80 "proposed for general use". Section 9.

Frank Harrell's Biostatistics for Biomedical Research (online) covers the comparison of two proportions and its intervals in the same terms and is a readable second route into sections 2 to 5.