Also just referred to as MDE.
Stands for:
To run a properly powered experiment, the MDE must be calculated ahead of running the test through power or sample size calculations. (Here's two resources on how to do so: 1. power 2. sample size).
The larger the MDE, the smaller the sample size needed to run your experiment.
And vice versa. The smaller the MDE, the bigger the sample required for your experiment to be adequately powered.
As a very basic guideline, you should consider setting the MDE to a relative percentage uplift of up to 5%.
If the experiment isn't powered enough to detect a 5% effect, the test results can't considered trustworthy.
However, it's also dangerous to go much beyond 5% because, typically, most trustworthy tests don't yield more than a 5% relative conversion lift.
Note, the MDE is different from what some experimenters call the Minimum Effect of Interest (MEI) which is viewed by some an an "input parameter" where as the MDE is seen as an "output parameter".
However, typically, the terms MEI and MDE are used interchangeably, and MDE is what's displayed on most sample size calculators.
An additional note, the observed effect, or treatment effect, is altogether different. The observed/treatment effect is the conversion difference detected.
Test results are only trustworthy when the treatment effect is larger than the MDE.
« Back to Glossary Index
