Great write up Alan, 👏👏 It’s so important to understand the nuances of a/b testing. It’s very rare that it gives a straight answer in my experience, especially as the distance between the a/b variants and the measured metric (purchase, re-purchase, etc.) increases. But testing in a systematic way like this has many other benefits, like doing many a/b tests around a set of landing pages for e.g. usually gives many higher level customer insights, or they are excellent to introduce large scale changes on critical pages.
The book "How not to be wrong - the power of mathematical thinking" is also a must have for statistical contextualization and the shenanigans in studies.
Great write up Alan, 👏👏 It’s so important to understand the nuances of a/b testing. It’s very rare that it gives a straight answer in my experience, especially as the distance between the a/b variants and the measured metric (purchase, re-purchase, etc.) increases. But testing in a systematic way like this has many other benefits, like doing many a/b tests around a set of landing pages for e.g. usually gives many higher level customer insights, or they are excellent to introduce large scale changes on critical pages.
Great insights
The book "How not to be wrong - the power of mathematical thinking" is also a must have for statistical contextualization and the shenanigans in studies.
Thanks for continuing to get the word out.