A/B Testing: Peeking Inside the Black Box
“Black box – any complex piece of equipment, typically a unit in an electronic system, with contents that are mysterious to the user.”
It can solve problems ranging from finding a cure for scurvy to choosing the best pick-up lines. A/B testing is much more than one of the ways to optimize your website.
Origins of A/B testing
It is said that the first A/B testing took place in the 18th century, when physician James Lind was trying to find a cure for scurvy. One day trying to find out which one worked best he started giving different remedies to different groups of sailors suffering from scurvy. This way he managed to find out that fruits rich with vitamin C were the most effective. What Linden did was a basic A/B or split test – he tried different variations of treatment and monitored the results without understanding cause and effect.
One can argue whether Linden was the first to use this method. In every situation where the input (ill sailors, fruits) and the output (healthy sailors) are known but what’s between is pure magic, split testing is the natural approach. Linden didn’t know a thing about chemistry and vitamins (it was his ‘black box’) but he knew that some sailors were doing fine after eating citrus. Now we’ve still got plenty of areas full of black boxes, mostly in situations where the ‘so-called human factor’ is involved.
People are not easy subjects because it is expensive and often illegal to experiment on them. And the results are almost always shady. In 2010 an interesting test was conducted on the homeless in the USA. The Federal Department of Housing and Urban Development started an 18-month research project on 3,000 families stationed in homeless shelters assigning them to different support programs.
Each family was given a few options (different models of subsidies, ‘leave-me-alone’ option etc.) and then tracked to learn the outcome of different approaches.
The effect: a decline in homelessness in the USA in the beginning of 2011.
Of course there are many less serious applications of the A/B testing method: blogger and marketing specialist Ian Fernando did a split test on a beggar and increased his conversion rate by 100%. He discovered that changing patterns (e.g. holding white paper instead of a piece of carton), engaging the givers and modifying common call-to-actions can have a tremendous effect on that man’s income.
Another blogger stated that AB testing is the obvious method to find the best pickup lines and I can’t argue with that. It is the basic non-simultaneous split test – when one tries different approaches in similar conditions (some stranger vs. attractive woman), with no clue how it works.
A few years ago A/B testing became very popular among web designers. The idea was to randomly show two (or more) different variations of a website and measure the behavior of a site’s users. It is said that the development of platforms, such as Facebook, was based on A/B testing. Some would ask why they didn’t hire researchers and designers to design what’s best. The answer is simple. Even the best researchers and designers don’t know everything and only letting the users see your design can give you solid answers.
You can conduct AB testing experiments on completely different versions of a site or several versions of call-to-action buttons. The most famous anecdotes mention the color of one button magically doubling or even quadrupling conversions. In fact there are some outstanding AB testing success stories like the one in which changing the position of the Facebook share button on AMD main page resulted in 3600% increase in shares.