Skip to content

Statistical Significance

Statistical significance is a measure of whether a difference between two versions of something is likely real, or just random variation. Most teams aim for 95% confidence, meaning the difference has less than a 5% chance of being noise.

The most common A/B testing mistake is calling a winner before the test reaches significance. A test that “looks” like a 20% lift after 200 conversions can flip to a loss with another 200. Patience beats optimism.

Related terms

See also on Miss Pepper AI