12 March 2026 · Notes
Reading a null result without writing an apology
Teams treat a null as a social problem. The memo starts explaining why the test was still valuable, then offers three slices that “lean positive”, then recommends iterating the same idea with a larger sample. That sequence is how a null leaves the archive as a rumour of a win.
In an App Experiment Readout System the decision sentence is allowed to be boring: hold the current onboarding; do not ship the variant; do not cite this test as evidence for a related bet. The value of the work is recorded in a separate paragraph about learning cost, not inside the result.
Assignment health still comes first. A dirty log can manufacture a null. If the ratio is off, you do not get to call the experiment informative. You get to repair instrumentation and, if the product still cares, run it again under a new readout plan.
When the log is clean, put the confidence interval in the primary band and stop. Diagnostics may show that new users from paid search behaved differently. That belongs in an appendix labelled as not a claim. If a director wants to pursue that slice, they commission a new test with that slice pre-declared. They do not get to promote it from the appendix during the same meeting.
We practise this tone in Experiment Readout Studio because it is a writing problem as much as a statistics problem. The apology is optional. The recommended action is not.