18 January 2026 · Notes
When to stop an app experiment you already dislike
Dislike is not a stopping rule. Neither is a promising-looking dashboard on day four. The readout plan should name three stops before the first user is assigned: harm on a guardrail, futility on the primary, and a calendar stop even if the interval is still wide.
Harm is the only stop that should be allowed to fire without a full sample. If refund-adjusted revenue or crash-free sessions cross a pre-written line, you halt and you write the memo as a safety readout. That memo is still a readout. It is not an informal Slack screenshot.
Futility is harder in product culture because it feels like giving up. In the studio we treat futility as a scheduled look with an alpha spend you actually wrote down. If you did not write it down, you do not get a futility call; you get the calendar stop. Peeking dressed as “we can already see it” is how teams ship variants that later look noisy in a holdback.
The calendar stop exists because app traffic is seasonal and GB school holidays move onboarding mix. Running “until significance” through a mix shift is how a spring paywall becomes a story about April, not about the copy. Put the end date in the memo title when you archive it. Future you will thank present you during the next planning cycle.
This is not a design-of-experiments textbook. It is a filing rule. Experiment Readout Studio module two exists so the stop is chosen when people are still polite, not when they are tired of the flag.