Lots of people are running AB tests. They can be a hassle to set up and they can be a bit complex to understand. For some of you, this is daily business. If so, there probably won't be new stuff in this post, as we are starting with the basics.

Basically: with an AB test you’re going to show two different variants of a page, a button or a piece of text to your visitors, to learn which version works best.

Here is the absolute minimum you’ll need:

  • You need some kind of way to divide the visitors in multiple groups
  • You need to display different content to these groups
  • You need to track some kind of action that needs to take place: the user visits a specific page, clicks a button or places an order

It is probably the easiest to get a tool to help you with these things. There is a great post on Quora listing all kinds of tools. Integrating these tools is pretty easy. I’ve played with VWO and Optimizely and they both have an extensive editor which allows you to change your website in real time. This works by loading your website in an iframe and storing the things you change on their end. It is a safe process.

When working with AB tests, there are two basic keywords you should understand in order to know how to run a successful test:

Statistical Significance: If a test is significant, it means that you gathered enough data to feel confident about introducing changes based on the test. The more people actually go trough your test, the more reliable your outcome will be. Obviously you don’t want to run your test forever, and that is where your confidence interval comes in.

Confidence Interval: This is your margin of error. Let’s say you researched a red button versus a green one. If 65% of your audience clicks the green one, with a confidence interval of 5%, the actual percentage for the green button will be somewhere between 60% and 70%. Again, the more people run your test, the more precise your outcome will be.

You could do the math yourself, but there are dozens of really good online calculators that can help you find out how many people should complete your funnel to reach statistical significance. This is a nice one: http://www.hubspot.com/ab-test-calculator

Happy testing!