How Long Should I A/B Test?

A/B testing is an important in the optimization of your company’s marketing strategy and elements. You can research keywords and plan out landing pages and consider vision tracking for every detail, but until you put your design in front of a potential customer, you don’t know if it will work. Even when you do, it’s difficult to tell if you’re getting your best results.

A/B testing is one way to narrow down what’s working well about your website or marketing elements, and what isn’t.

What is A/B testing?

459916_Images to support blog post_2_070919.jpg

A/B testing is, essentially, an experiment to see which version of a website, landing pages, or other element performs better. This is often used on all types of marketing materials - ads, sales pages, and more. You might also hear this process called split testing or bucket testing.

Split testing and A/B testing are generally the same concept, but executed differently. The goal is to collect data by showing a visitor two different versions of the same element and to see which element gives you the result you want. With A/B testing, you adjust a simple element (e.g., the color of a button, heading copy, a single image, etc.), which is a relatively simple process. Split testing, however, involves redirecting the customer to a completely separate page.

If you're just looking at a couple of discrete variables, A/B testing is probably enough. Split testing is more effective if you're testing website redesigns, for example. 

Configuration of split testing in Ontraport

Configuration of split testing in Ontraport

If you have multiple elements you want to change at once, you will need to conduct multivariate texting. This is much more complicated than A/B or split testing, and is recommended for those who are much more experienced with marketing. The process generates so much data that it can be hard to go through unless you're very familiar with statistics and what you're looking at.  

Ultimately, the point of these different testing types is to use data to drive your marketing, allowing you to make fact-based decisions instead of going with your gut (and potentially wasting a lot of money).


Why is this important?

When you’re designing a website, especially at first, it’s easy to get overwhelmed by all the choices. Particularly if you’re designing the website yourself, without a lot of experience, you will probably end up modeling your website after others that you like, whether they’re really appropriate for your business or not.  

That’s fine as you’re getting started, but as your company grows, it’s important to make data driven decisions about how to scale your business. You’ll need stronger architecture for your website, and often times, you’ll need a better, more modern website.  

When you’re starting out with your business its common to set it up in a way that won’t allow it to grow past a certain point. When we work with new and emerging companies the first thing we focus on is setting up the right systems that will support growth in the long run. And often times that means having something a little more powerful than you need right now, but allows you the capabilities such as A/B testing when you’re ready to implement it.


How long should you do A/B testing?

Ask a room of experts how long you should do A/B testing, and you’ll get a huge variety of answers. 

There are a couple specific factors to consider: 

  • Having two separate versions of your page up for a long time can make your overall Google ranking drop. Google recommends that complete your testing as soon as possible. They don’t share an actual timeline for how long it’s safe to test.

  • How frequently do people come to your page? If you get hundreds or thousands of hits a day, you’re probably going to gather statistically significant results fairly quickly (but you may also want to consider multivariant testing!). If you only get a few dozen hits per day, it’s going to take longer to get you the kind of information you need.

  • How familiar are you with coding? If you’re doing your own A/B testing, there are certain details you can use to help keep Google from dropping your ranking, but they are not for those new to web design. Working with an experienced marketing company to help complete your A/B testing is probably ideal at that point.

  • If you determine that you test for more than a few elements at a time, you will probably need to conduct multivariate testing. Again, if you're not a very experience marketer, this is a time to work with a company like Automated Dreams so that we can help you understand the data that you will collect.

In general, most experts believe that you should look at your data after a week and see if your results appear to be statistically significant. If not, then test for another seven days. If you're testing a website, two weeks seems to be the maximum timeline before your page may start looking fishy to Google. Then, it’s time to choose an option for the time being while you consider your data and determine if there are other factors you want to test.


Is A/B testing the end of the conversation?

Of course not. A/B testing is just one type of data collection used to determine whether or not a website is doing its job. Regardless of what industry you’re in, the job of a website is to convert customers. You might want them to sign up to take a course, buy your latest book, follow you on social media, or purchase your SaaS offering.  

In many cases, your goal is to move the customer further down the sales funnel. You want them to sign up for a free trial, give you their email address, or call you for more information about your services. Without methodically testing your website, you’re just guessing if it’s going to do its job converting customers. When you conduct thorough testing, you know that you’ve got the best version of your website and can trust that you’re doing the best work you can.  

At Automated Dreams, we want to help you take your business to the next level. We’re here for businesses that are ready to re-evaluate their systems so that they can continue to scale without the growing pains and roadblocks, but aren’t quite sure how to take the next steps to make it happen. Expanding your business takes consideration and experience; we know how to get you there.

Contact us today to learn more about our services and what we can do for you.