Multivariate Testing - A Real Life Example

Posted By: cody

Last Updated: Sunday February 21, 2010

About a year ago I posted a little tutorial / guide on how to get a simple A/B or Multivariate test running using Google Website Optimizer. Unfortunately I never made a follow-up post, though we still use this internally to help improve our conversions. I figure shedding some insight on why this is useful and how we utilize this internally may be of interest to some people, so here it is.

Now if you’re not familiar with Google Website Optimizer or A/B / Multivariate testing I would highly recommend stopping here and taking a quick moment to get the jist of what they are. In simplistic terms you have several changes on your website with defined goals (here a sale, or conversion). You send traffic to these changes and after awhile you will gain enough statistics to determine if a particular change helps you or not (or in some cases does exactly the same).

So now that you have an idea of what the point of the testing is let’s take a peak at a current Hawk Host one that’s not quite conclusive yet.

The Variates

In this particular test we decided to go with one of the more prominent sections on our website: the portion above the folder and in the center (mostly). The reasoning behind this was it’s one of the first things people see and in theory would be the first thing that could potentially be improved to get people to order. Here is the “block” that we’ll be playing with:

In particular these are the elements we’ll be changing and testing inside of the above section:

Now in this particular test we decided to go with some generic changes - as you can see there are two elements in particular we’re focusing on. The main heading and subheading (we decided to lump these together as they can compliment each other - with variate testing you learn to appreciate that the smallest change can result in experiments that run months longer) and the image.

The headings / subheadings we are testing are the following:

And the images we are trying:

Now one thing you must realize with variate testing is what you think will do well (or not do well) isn’t always what the results show. Luckily for us that smiley face was weeded out within the first few days due to its lack of performance (and utter creepiness).

The Results (thus far):

Now you need to wait for the results - this can take awhile depending on the amount of impressions / conversions / combinations you’re running. In this particular test we started out with 16 combinations (4*4) 1 of which was the original website. We’ve whittled it down to 8 combinations after a few weeks of letting it run and seeing which combinations are consistently doing poorly (you need to gather ample data to make this decision, patience is your friend).

So what can these small changes do for your business’ bottom line?

As you can see numerous combinations are out performing our original - some to a very large degree! I’ll let you interpret these results but at a glance you can see that several of these changes are converting significantly better. We’ll let this run until we gain enough data to be 100% sure that a particular change will yield better results but it’s looking promising.

In case you’re curious here is the original vs. the winning variate (at this time):

Original

Winning variate

That’s right, in this instance the combination with only the image different is converting well 2x-3x better than the original. Always keep testing - small changes can make large splashes.

-Cody

Ready to get started? Build your site from
$2.24/mo
GET STARTED NOW