21 November 2025 ยท Notes
Secondary metrics without turning them into a hunt
Secondary metrics are not the enemy. Silence is. When a team is forbidden from looking at anything but the primary, the interesting numbers migrate to private notebooks and reappear in the last five minutes of Friday review. An honest readout system gives those numbers a drawer with a lock.
Before assignment, list diagnostics you expect to move if the mechanism is real: time-to-first-success, a funnel step, a support tag. They cannot change the ship/hold call unless you promoted them to guardrails in the plan. After the test, you may add exploratory cuts in an appendix with a header that says they were not pre-registered. The header is the lock.
What you must not do is recompute significance on twenty extras and pick the two that cleared 0.05. That is the hunt. In clinic we ask authors to paste the query time from the warehouse next to each extra chart. Late queries are a smell. They do not prove misconduct, but they belong in the memo so a later reader can discount them.
Signal vs Noise for Product Teams spends a week on this filing habit because product people are rewarded for narrative completeness. Completeness and validity pull in opposite directions. The appendix is how you honour curiosity without letting it drive the company van.
If a diagnostic later becomes a primary in a new experiment, good. That is a new readout plan, with a new sample, not a sequel written in the same file.