KOBIT Kompact Series: Is A/B Testing Really Worth It?

cnxt_dev
cnxt_dev
2019/06/09 20:00

ab test

If you’re a digital marketer, you know the importance of A/B testing. However, even though A/B testing has become the norm, many marketers are starting to question whether it’s really worth it at all.

The thought of your conversion rate hitting double digits overnight simply because you used a single A/B split test is certainly enticing but unfortunately, this is mostly a myth.

The reality is that most A/B tests fail to deliver any significant benefits. The good news is that they have the potential to.

The Problem with A/B Testing

Most A/B tested campaigns have one or more of the following problems:

  • You’re not waiting long enough for the results. A/B testing is not a tactic that’s going to produce results overnight. You need to give your campaigns at least 3 weeks to a month to take effect before you decide whether they’re a success or not.
  • You’re basing the tests on your own assumptions. When you add a bit of bias to the equation, you’re the results of you’re A/B split testing is going to be slightly misleading.
  • Statistical significance takes time to attain. Since your results are going to be dependent on a large sample size, it’s going to take a while to achieve true statistical significance.
  • You’re using too many tests. Running your tests at high speed is going to get you the wins you’re hoping for. Decide whether you want to achieve short-term results or gather data that will serve you well in the long run.
  • You’re not focusing on the right metrics. Naturally, every business wants to achieve a higher conversion rate by A/B testing campaigns but this isn’t the only metric that matters. Not only should you be tracking the impact that a variation has on your conversion rate but how it’s affecting visitor behavior too.

A/B testing is definitely valuable, just make sure you have realistic expectations and stick to the rules for running a successful test.