AAB Testing

A couple of days ago somebody told me I should A/B test an idea. Take two versions, show each to half the visitors, crown a winner. The A/B test. Every product manager's favorite way of not having an opinion.

I have done my time with it.

At Mister Spex I was responsible for conversion rate optimization and usability, which is a fancy way of saying product management. We were running TV commercials back then, and TV commercials bring millions of people to your website. Millions of people are a lovely thing for a product manager, because the math gets simple. If I managed to lift the conversion rate of the site by a tenth of a percentage point (yes, 0.1), that was 125,000 euros more revenue. Per month. You are welcome.

The first question was how to measure the conversion rate at all. Not as obvious as it sounds. If I took the overall traffic of the site, then one day a marketing manager would flood us with a few hundred thousand visitors from some bargain-basement channel, the conversion rate would drop by one or two percent, and all my testing was garbage. So I went looking for traffic that behaves. It turned out that SEO traffic was by far the most consistent in quality. People who search for your product and click on you convert at the same rate today, next week and next quarter. SEO traffic became my yardstick. The marketing managers could keep their floods.

The second question was what actually drives a sale. I drew a Venn diagram, as one does, with three circles. Trust. Offering and usability. Inventory and availability. That is it. With those three levers alone you can do wonders.

The levers already had owners. Marketing was constantly tinkering with pricing, which is the offering. A freshly founded department called category management was looking after inventory and availability. Which left me with usability and trust, and I was happy with that. Trust is the lever nobody else wants, because you cannot put it on a slide.

And this is where the trouble started. A lot of folks in the company told me we should test every change we could come up with. A noble idea. A data-driven idea. The reasoning was sound, too: we had a million ideas and wanted zero regressions. Who wants to go backwards on conversion rate and revenue?

As noble and as data-driven as it was, it was a bad idea.

Back then there was a thing called Google Website Optimizer. A tool by Google, nowadays discontinued, for good reasons. I cannot even fathom how much harm this tool has done to the e-commerce industry. We had a couple of A/B tests running where I could have sworn that the new version was performing badly. You could see it in the numbers, you could see it in the support tickets. And still, according to Google Website Optimizer, it was the winner. Every time.

So I built a test of my own. A was the original. The second A was another copy of the original, pixel for pixel. B was the new version. Three variants, two of them identical.

Lo and behold, the copy of the original won against the original.

How is that possible? The sample size was far too small to say anything conclusive, and the test setup itself was broken. The tool was happy to declare a winner anyway. Of course it was. A tool that says "not enough data, come back in three months" does not get used, and a tool that does not get used does not get a roadmap.

From that day on we didn't do A/B tests anymore. And when we did them, they were AAB tests. If your two identical versions do not come out identical, you do not have a test. You have a random number generator with a nice dashboard.

I still recommend it. Before you test your idea, test your test.