One of the most popular forms of testing is something that you have probably already been doing for a while now. You measure which you currently have and then you make a change. Then you measure what happened after that change was made and you compare the two and see if there is a difference. This is known as sequential testing and it’s not exactly statistically valid. To see why not, let’s take an example.
Let’s say that you own a baby supply e-commerce site and you want to test two versions of your product page. You might be experimenting with tabular design, adding reviews or are playing with your add to cart button things like that so you run version A for a couple of weeks and you measure your conversion rate let’s say you’re at 1.5 percent you make your changes and you push them live. You let this version run for a couple of weeks and then you measure. Now your at 5.5 percent well that’s great news right? Well you should be celebrating because you just more than tripled your conversion rate.
Well let’s look at those dates a little more closely if you happen to be bottle feeding a young one at that time you’ll remember headlines like these ‘toxic baby bottles’. I do not think there is a stronger marketing message than that now what I didn’t tell you about that product page is that it’s for a BPA free baby bottle for those of you not up to speed on your baby bottle specs that means it’s not one of those toxic ones. In reality that tripling of conversion rate had nothing to do with your new design and had everything to do with the external factors going on in the world around us.
People wanted that product so badly that you probably would have seen more sales if you would try and hiding your buy now button. If we know that external factors can have an effect on our websites performance then how do we control for this during a test? The easiest way is to randomly and evenly split apart traffic in real time and here’s why it works. As visitors request the pages of our site they have an equal and random chance of seeing any of the variations.
If you and I visit the same site at the same time we might both see a different version. If an external factor does occur its impact will be felt in the same way across all of the variations. So you are probably saying yourself ‘yeah but how often does something like that really happen?’ Most external factors aren’t that obvious and they don’t have to be huge to impact your websites performance. More often than not we will never even know about them.
As an example we were working with a client a number of years ago whose services revolved around travel in a specific geography. We saw a huge double-digit increase in conversion rates one particular week. Don’t get me wrong everybody was happy but the problem was we couldn’t explain it. You see we hadn’t changed a single thing that week.
Now I found out what it cause it almost a full year later when a friend of mine was telling me about a week long Discovery Channel special that was highlighting our clients target geography. Sure enough as it turns out that special had aired the same week we had seen the mysterious boost in performance. Now if I hadn’t had that random conversation we may have never known what caused it.
And if you are still not convinced than just think of the seasonality of your business. Even the time of year can have an effect on how visitors interact with your website. Just like things as seemingly small as the weather or the state of an economy thousands of miles away.
At the end of the day we are really never sure what external factors are out there affecting our business so to make sure that our experiments are not impacted by them we want to avoid sequential testing and split our traffic in real time. You can rest easy now because that is one thing that Google’s website optimiser automatically handles for us.
