Online controlled experiments (or A/B testing, multivariate testing) are one of the most effective ways to make causal inferences, to scientifically validate most ideas. As they grew more and more popular, setting up and interpreting experimentation results have become must-have skills for analysts and popular interview questions.
Even though I am quite experienced with experimentations, I still sometimes get confused about the exact definitions of common terms like statistical significance. So I put together this cheat sheet to help you (and me) remember the right interpretation of those experimentation results.
P.S. If you want to learn online experimentation, I highly recommend the book Trustworthy Online Controlled Experiments (A Practical Guide to A/B Testing), written by industry practitioners, it has a great mix of theory and practical advice that most other books lack.
