Sample Size Calculator for Unequal Sample Sizes — A Tableau Template

  • You don’t finish a test with a sample size that cannot detect the effect you want to detect to make a business decision
  • You have a stopping point -you don’t end the test through p-hacking
  • You can prioritize and plan tests with the limited amount of traffic you’ve got
  • Baseline rate: the higher the baseline rate, the less sample size is needed to detect a certain MDE.
  • Relative MDE: the minimal detectable effect relative to the baseline rate the given sample size can detect. MDE indicates practical significance, what’s the lift that’s needed to make a business decision. 1% increase in conversion rate for Amazon has much higher practical significance than a 1% increase in conversion rate for an early-stage start-up.
  • Ratio of arm 2 to arm 1: not every test is 50–50 split, if you need to create a global holdout group it usually is 1% — 10% of the total sample.
  • P-value: the probability of detecting the observed difference or more extreme under the assumption there is no difference.
  • Power: the probability of not committing a type-II error — the ability to detect a difference if there’s one.
  • Runtime vs. MDE: when making sample size recommendations, it’s always helpful to think as your stakeholder. What do they care about? How long do we need to run the test to detect a difference that we want to detect?




Books | Marketing | Data Viz | Analytics & Experimentation | Entrepreneurship 💡Founder of | Blog:

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

An Address is Worth So Much More Than a Piece of Mail

Sketches, Data Visualization and more

Random Forest: The Math of Intelligence

Revolutionising the Public Service

Building a Web Application to Deploy Machine Learning Models

3 things to know to get started with time series

Big Data or Smart Data?

Improving Olist’s delivery performance

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store


Books | Marketing | Data Viz | Analytics & Experimentation | Entrepreneurship 💡Founder of | Blog:

More from Medium

What is Analytics?

Five real-world applications of data analytics in healthcare

Do storks bring babies? Here is why every data person should be aware of spurious correlation.

Part 1 : Austin Bikes EDA and Geo Anal