Experimentation Results — How to Interpret?

Online controlled experiments (or A/B testing, multivariate testing) are one of the most effective ways to make causal inferences, to scientifically validate most ideas. As they grew more and more popular, setting up and interpreting experimentation results have become must-have skills for analysts and popular interview questions.

Even though I am quite experienced with experimentations, I still sometimes get confused about the exact definitions of common terms like statistical significance. So I put together this cheat sheet to help you (and me) remember the right interpretation of those experimentation results.

P.S. If you want to learn online experimentation, I highly recommend the book Trustworthy Online Controlled Experiments (A Practical Guide to A/B Testing), written by industry practitioners, it has a great mix of theory and practical advice that most other books lack.

Viz made with Tableau & Background made with Canva

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store