Why Don’t You Support Multivariate Tests?

Knowledge Base > Pre-Sales Questions > Running Tests > Why Don’t You Support Multivariate Tests?

in Pre-Sales QuestionsRunning Tests

Simple Page Tester only tests a Master and Variation, a simple A/B test.

The reason we don’t support Multivariate tests is because they take an order of magnitude more traffic (like 10x or more) to get to statistical significance.

Without getting to statistical significance you’re basically making an assumption on not enough data. This is bad and can lead to false conclusions about your test result.

If you’re into math, here’s a great summary of how statistic significance is calculated and you’ll see why multivariate tests take so much more data:

https://developer.amazon.com/public/apis/manage/ab-testing/doc/math-behind-ab-testing

Summary:

Generally speaking (even when people aren’t using our product) we advise against it unless you have huge amounts of traffic.

In 90% of cases you’re much better off just running the tests one by one and making more significant changes. Eg. Testing different layouts, full colour schemes, element positioning, etc.

This will allow you to get to statistic significance much quicker and make more decisive changes to your website to improve it.

0