Lost Logs, Found Lift, part I

Designing an A/B test before reading its dashboard

12-minute read

TL;DR

• In a simulated A/B test, 280,000 products are assigned at random to either an old gate or a new gate with a bug mid-process.
• The dashboard reports a +6.4% lift with p = 0.009, apparently enough to ship.
• Part I explains the design and initial results.
• Part II investigates the missing records and determines which evidence is trustworthy; Part III tackles a staged-rollout decision.
• An apparent win is not enough without verifying the supporting data.
• A walk-through of every calculation is on a companion page, the power calculation.

Contents

1. An assembly line with a faulty scanner
2. About this simulation
3. The A/B test, step by step
   1. Decide what is worth shipping
   2. Design the comparison
   3. Set the evidence the rule needs
   4. Collect complete, trustworthy records
   5. Read the dashboard against the design
4. Check your own design
5. Remaining questions
6. Next
7. Limits of the case

An assembly line with a faulty scanner

A factory compares its current assembly station with a redesigned one, meant to increase effective production per week. Products are randomised between the two stations, some take a legacy manual route instead, yet all of them end at the final inspection station where completion is recorded whether or not the product was ever scanned.

The new station's scanner has a bug causing it to miss products from an older component batch which, coincidentally, is also a harder batch to assemble. Those products stay on the line, and their completion is recorded at inspection like everyone else's.

All the while, the dashboard continues to display the computed yield from the scanned products only, meaning the intact outcome records have a faulty filter deciding which products the dashboard sees.

Does routing products to the redesigned station raise 7-day yield enough to cover its cost, accounting for possible rise in packing-step errors, cycle time or returns?

An A/B test turns that question into a decision in four steps, and a fifth step checks the result against them:

1. Decide what is worth shipping.
2. Design the comparison.
3. Set the evidence the rule needs.
4. Collect complete, trustworthy records.
5. Read the dashboard against the design.

About this simulation

The records are simulated with a known truth and pre-selected starting parameters, so every claim about bias or error can be measured. The simulator follows each product from arrival to its final outcome:

• Generate 20,000 eligible products per day. Plan 21 days of intake, but generate only the first 14 days, the interim cohort of 280,000 products.
• Use each product's id to assign it at random to one of the stations.
  • Route 15% of products, in either arm, through the legacy manual route. They bypass both stations and generate no assembly scan, giving an ordinary reason for a missing scan.
• Set stations' completion probabilities:
  • The old-station completion average is 3.2%.
  • The new station's average is increased by 3% relative, e.g. a product with a 2% baseline probability would have a 2.06%.
  • Manual-route products receive no benefit.
• Give each product a probability of finishing within seven days:
  • Vary it with supply line and component batch.
  • Older-batch products have lower baselines.
  • Draw whether each product finishes: the probability is an input and the finish is a random draw, so observed completion rates vary from run to run.
• Simulate the logging fault by randomly dropping 35% of scans for older-batch products handled by the new station, keeping their assignment and completion records intact.
• Simulate a reporting delay: after a product finishes, its record takes a random 0-36 hours to appear in the central system.

Using a simulation has a few key advantages. I can keep a truth table of potential outcomes for auditing; the truth is known, so I can switch the fault off to isolate its consequences; I don't have the actual resource constraints a real company has, allowing me to run 5,000 simulated experiments per scenario. With that many runs the simulation's own sampling error (the Monte Carlo standard error) on a 5% rate is about 0.3 percentage points. The limits section lists what this buys and what it costs.

1. Decide what is worth shipping

The redesigned station costs more to maintain, so the team has to say beforehand what gain pays for it and what harm it will not accept, or the result will be judged after the fact by whoever reads it first.

For this case, the team defined a business requirement of at least a 1.0% relative ITT lift to cover the new station's maintenance cost.

Three harms would block the change whatever the yield does: more packing-step errors, slower processing and more returns. Step 3 turns the floor and the harms into a rule.

2. Design the comparison

Since "the team" expresses the maintenance-cost requirement as a total 7-day yield improvement, I define the primary effect on the same population, which is the whole eligible production across all routes, including the manual one.

Random assignment of products to stations prevents favouring any specific product by either station, allowing a fair comparison of the two arms' yields and an estimate of the effect of the assignment. Each product is handled by one station, and products do not affect each other. The 15% that take the manual route keep their assigned arm and receive no benefit.

That effect of assignment over the whole population is the intention-to-treat (ITT) effect: the test's estimand, written down before any data arrive.

Each product gets a score:

An arm's yield is the average score over the products assigned to it:

where n_A is the number of products assigned to arm A.

Absolute difference and relative lift

I report the difference between the arms as both an absolute change and a relative lift. The minimum improvement required for shipping, the "ship floor", is a relative ITT lift, so results must be compared with it on that same scale and population.

These two scales, and the log scale the interval is built on, are sections 1 and 2 of the power calculation.

Why the overall benefit is smaller

Manual-route products remain in the denominator because the launch decision covers all eligible products. They receive no benefit, so the overall relative lift is 85% of the lift among station-handled products.

This shortcut works because manual routing is independent of baseline completion probability, so both route groups have the same expected baseline yield. With different baselines, the calculation would have to account for the difference.

The design in one table

3. Set the evidence the rule needs

A favourable number on its own is not a decision, it needs the right context. The confidence interval (CI) puts a range around the estimated gain to show its uncertainty. The guardrails set limits on costs and risks for an acceptable trade-off. The plan checks that the experiment can meet that bar at all. Every figure in this section has a worked derivation on the power calculation page.

At the interim look (day 14 of a 21-day intake; a final look follows with a stricter 97.5% interval so that the two reads together carry close to one look's error), adoption requires:

1. Lower bound of the 95% CI on relative ITT lift > +1.0%.
2. The required guardrails to pass.
3. No guardrail breach.
4. No unresolved decision blocker.

Similarly, a favourable CI is not enough to justify shipping if unresolved problems with which products were counted or how outcomes were measured make the evidence unreliable.

Protect against harm elsewhere

The case's three guardrails:

The p95 cycle time is the processing time below which 95% of measured runs fall. It checks the slower end of processing rather than only the average.

Write a guardrail's 95% interval as [LB, UB] and its margin as m:

Returns can block shipping, but they do not have to pass. Shipping is blocked only if the evidence shows that the return rate increased by more than 10% relative to the old station.

This weaker requirement was agreed before the test because only about 4,500 products finish in each arm, and with an assumed 5% return rate, 225 returned products are too few for a reliable test. With that sample, even if returns are unchanged, the test would confirm that they stayed within the limit only about 18% of the time. Requiring that confirmation would block most harmless changes.

An inconclusive result therefore does not block shipping if the other requirements are met. It leaves the risk unresolved; it does not establish that returns stayed within the limit. Section 8 of the power calculation derives the 18%.

Check that the experiment can meet the bar

Before any data arrive, the team should know how likely the rule is to pass if the new station works. That probability is power: the chance that the planned analysis clears its bar when the true effect has a stated size. Sections 1 to 7 and 9 of the power calculation walk through every figure below.

Three things are fixed first, and everything else follows from them:

• The sample. The interim look reads the products taken in during its intake window, about 140,000 per arm.
• The target. The plan aims for 80% power, the usual planning convention: a one-in-five chance of missing a real effect of the planning size.
• The bar. The ship rule asks for more than a lift above zero, but the business requirement sets the floor, so the lower bound of the interval must clear +1.0%.

From the sample and the target comes the planning lift: with 140,000 products per arm and the 3.2% baseline, the smallest relative lift that the test detects with 80% power against zero is 5.9%, rounded to 6%.

At a true 6% lift the same data answer two questions with different confidence: the test shows some lift at all 81% of the time, and clears the +1% ship floor only 65.5% of the time. Reaching 80% power against the floor at a true 6% lift would need about 197,000 products per arm, or a true lift of about 7.0% at 140,000. The case keeps 140,000 per arm and states the shortfall. That is a trade-off to agree before collecting data; lowering the requirement afterwards would defeat its purpose.

Four lifts appear in this post, and they are easy to mix up. The last column says how often the ship rule clears its floor at the interim look if that lift were the truth:

To see the full calculation, step by step, open the power calculation in a new tab.

4. Collect complete, trustworthy records

The analysis can only use records that exist, and a seven-day outcome is not complete on the day the product arrives. So before anyone reads a number, the design has to answer three questions: which systems record each product, when their records count as complete, and which table feeds which analysis.

Who records what

Three systems record each product, and the analysis joins them by product id. The routing system records the assignment, the scanner at the station logs the scan, and the final inspection station reports the outcome through the central records.

Each record answers a different question, and the gaps between them matter: a product can be assigned without ever being scanned, and it can finish without its scan reaching the log. The design has to say which of these records is allowed to decide who gets counted.

When a record is complete

A record is complete when the result it reports can no longer change and the record itself has arrived. Two clocks decide that:

1. The intake clock marks when a product arrives. The interim analysis covers every product that arrived in the first 14 days.
2. The outcome clock starts separately for each product on arrival. It runs for the 7-day window, and the completion record then takes another 0 to 36 hours to reach the central system.

So a product arriving on day 0 is settled a little over a week later, and a product arriving at the end of day 14 is settled by day 22.5. The cohort is complete on day 22.5, and only then can every product in it be counted as finished or not.

Returns have a third, slower clock. A return can land up to 14 days after completion, so the cohort's return records are complete by design on day 35.

One consequence the design has to accept: the cohort is complete only on day 22.5, a day and a half after intake closes, so the interim read cannot stop the last week of intake. Catching a scanner problem sooner needs separate monitoring of scan health.

Which table feeds the analysis

Five tables, one row per thing counted; the last column says what each may do in the primary analysis. The dashboard breaks that rule: it counts only products with a logged scan, so its population depends on what reached the log, and section 5 reads it against this design.

5. Read the dashboard against the design

The dashboard is a calculation, and the question is whether it is the calculation the design asked for.

On day 14 the dashboard looked like this:

The new station's yield was 3.036% against 2.853% at the old station, a difference of 0.18 percentage points and a relative lift of +6.4% (95% interval +1.6% to +11.5%, p = 0.009). The whole interval sat above the +1.0% ship floor, and both guardrails passed. The interval is the log-ratio interval of sections 2 and 3 of the power calculation.

The p-value tests a lift of zero under the dashboard's own assumptions; it says nothing about the +1.0% floor or the logging, and statistical significance is not practical importance.

Set the dashboard beside the design, stage by stage:

Compare the counts before and after filtering to logged scans:

Assigned products split close to 50/50; products with a logged scan do not, a sample ratio mismatch (Kohavi, Tang and Xu, 2020, chapter 21; Fabijan et al., 2019). The dashboard appears to clear the floor, and its population and its clock both differ from the design.

Check your own design

• ☐ Define the outcome and its time window.
• ☐ State the population the decision covers, and write down the estimand and the ship floor before the test starts.
• ☐ Name what receives an assignment, what contributes one outcome and the grain of every table.
• ☐ Separate chosen settings and declared requirements from measured evidence.
• ☐ Compute power against the ship floor as well as against zero.
• ☐ Date the complete read: outcome window plus records delay, and the slowest guardrail.
• ☐ Check the records and filters that decide who enters each analysis, and run the sample-ratio check on every population against the ratio it should preserve.

Remaining questions

1. Where did the scans go? 280,000 products were assigned and 233,889 scans were logged. The manual route accounts for about 15% in both arms, but the arms did not lose equally: 119,182 logged scans at the old station against 114,707 at the new one. Which products are missing, and what do they have in common?
2. Does a missing scan change the estimate, or only the count? A filter that drops products at random costs precision and nothing else. A filter that drops the harder-to-finish products from one arm moves the lift. Which kind is this, and by how much?
3. How much of the +6.4% is the early read? The dashboard read the cohort on day 14 with about half of its outcome windows still open. What does the same calculation give on day 22.5, when the cohort is complete?
4. What does the design's own estimate say? Intention-to-treat, on every assigned product, read when the records are complete: does its lower bound clear the +1.0% floor?
5. Could the team have seen this on day 14 without the ground truth? The sample-ratio check already flags the scan population. Which other checks, run on the records alone, would have caught the fault, and how early?
6. What is the decision? Ship, continue to the final look, or stop and fix the scanner first, and what has to be true before the experiment is rerun.

Next

Part II works through those questions across 5,000 simulated experiments and sets out the decision. Part III extends the case to a staged rollout.

The repository contains the design, diagnostic notebook, decision memo, simulation tables and reproduction commands.

Limits of the case

The baseline yield, station benefit, manual-route share and scan-loss probability are assumptions chosen for the simulation, none taken from a real production line, and the factory illustrates a measurement problem rather than realistic manufacturing throughput. The run shown was selected by a stated rule to demonstrate the pattern; the claims about bias and error rates in Part II come from repeated simulations of the specified scenarios and make no estimate of how often real companies meet this problem.

Part I has not established how much of the apparent improvement comes from the station, the missing scans or the early readout. What the power figures themselves leave out is listed in section 10 of the power calculation.

References

• Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press: chapter 21, "Sample Ratio Mismatch and Other Trust-Related Guardrail Metrics", pp. 219-225.
• Fabijan, A. et al. (2019), "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners", KDD 2019, pp. 2156-2164, doi:10.1145/3292500.3330722.