2 February 2026 · Notes
Metric trees that survive a mobile staged rollout
A metric tree drawn for a website homepage test often assumes everyone who could see the change has already seen it. Staged mobile rollouts break that. Ten percent of devices receive a binary, store listing copy still sends traffic to both experiences, and a holdback sits in the platform for a week after “ship”.
The readout then needs a tree whose top node is the exposure definition, not “weekly active users”. Under that node you can hang primary conversion among the exposed, guardrails among the same population, and a separate branch for people who never received the binary. Mixing those populations in one conversion rate is how a staged rollout looks like a country-level win that later vanishes.
We ask Studio participants to print the tree and write the exposure rule on the trunk in a sentence a designer could repeat. If they cannot, the tree is decoration. Guardrail Metrics Clinic spends most of its afternoon on this trunk sentence, because health metrics are the first to get computed on the wrong denominator.
Store-listing tests deserve their own tree. The exposure happens off-app. Treating listing conversion as if it were an in-app paywall experiment will smuggle unmatched users into the primary band. Name that surface in the memo title. It is not pedantry. It is how later readers avoid citing the wrong test in a Q3 review.