3 Tips to Improve Your A/B Testing Process


tips to improve ab testing process

In the long-term, your A/B testing process is one of the most important factors driving performance of your conversion rate optimization efforts.

I want to share three tips that will make your A/B testing process more impactful. We use these techniques on daily basis to convince companies to invest in CRO, spot real customer problems, and run long-term optimization programs more efficiently.

The three techniques described below enabled us to finish the process quickly, bring conversion rate uplift, and keep ourselves well-oriented through the multitude of tests.

1. Start testing as soon as possible

The problem we’ve struggled with in the past was over-analyzing data before running the first A/B test for a particular website. It was taking us two months on average before we could show results of our work. Sometimes after this long time we delivered conversion rate drop – you can imagine how disappointed was the client. The same applies to your boss, who invests money in CRO.

Your goal should be to start a test as soon as possible, because every hour you lose traffic you could have tested. Not to mention the opportunity cost of running less performing version of your website.

Goal flow report in Google Analytics is one of the places where you should head first in order to quickly find elements worth testing.

In many cases you can spot severe problems with your website quite fast. A look into standard Google Analytics reports should give you a couple of ideas worth testing. Landing Pages, Goal Flow and Funnel Visualization reports are good places to start looking for low-hanging fruits.

Use guerrilla user testing to identify most cumbersome usability issues with your website. Show your site to someone who is not working with it on daily basis (it could be your friend or even a random stranger in the nearest coffee shop) and ask him to complete the most important user scenario.

Another way to gather ideas for A/B tests is interviewing your sales and customer success departments to learn about your customers’ most frequently asked questions. We’ll talk more about this in tip #2.

Remember, don’t over-analyze in the beginning. Try these ideas to come up with a few quick ideas and start your test. It’s the most effective way to use your time when you’re starting conversion optimization process.

2. Do interviews with sales and customer care departments

We are an agency, so client interviews at the beginning of a project are standard protocol for us. If you are a part of an internal conversion optimization team though, it’s easy to think that you know everything about your customers. Be careful! I know a lot of marketing specialists that were completely detached from reality before they spoke with customer care department (CC).

Recommended for YouWebcast: Sales and Marketing Alignment: 7 Steps To Implement Effective Sales Enablement

You should constantly keep in touch with members of sales and CC departments. They talk with customers on daily basis and have the most knowledge about their fears, doubts, problems, needs, and motivations. This knowledge is the foundation of effective conversion optimization.

Here is a list of questions I always ask sales / customer care specialists:

  • What are the reasons customers look for a product like this?
  • What are the expectations of end effect of using the product?
  • What are the main doubts about the product?
  • How do you tackle these doubts? Which answers work best?
  • What features and details are most important for customers?
  • Which emotions do people express when talking about the product?
  • Which incentives work best on customers?
  • What problems are most common?
  • How do you respond to these problems?

Why do we put emphasis on this particular part of analysis? This story explains it: We were working for a dating site. A customer care specialist told us that users often asked about how the payment for the dating service would appear on their bank statement. Why? Apparently people felt nervous that someone would be able to see they were using dating site by looking at their account history. It’s an abnormal doubt, but a crucial one because it concerns users who are about to convert. Fixing it was an important step towards higher profits.

3. Keep track of your A/B testing

You should keep a record of every single A/B test. You may not think about it in the beginning of your A/B testing process, but the more tests you conduct, the longer the list of lessons becomes.

With this A/B testing record on hand, you can look back and quickly check what you have optimized before and the impact. Without one, you may end up forgetting what you’ve tested and find yourself testing the same hypothesis after a couple of months.

Evernote 2015-10-13 10-16-40.png

It’s an example of an A/B test note, that we use to keep track of our conversion optimization activity. The one above describes a test of removing unnecessary elements from product page for traffic coming from price comparison engines. We ran it on awa24.pl an e-commerce site for electronics.

It’s also an invaluable asset to have when you want to add another person to the conversion optimization team. You simply grant the new teammate access to the spreadsheet or wherever you’re keeping your record, and they can instantly get up to speed on the historical testing and results.

An A/B testing record should include:

  • domain, number of test, dates
  • test hypothesis (+ metrics you will use to verify them)
  • result
  • screenshots of original website and all the challengers (mark your treatments)
  • test analysis
  • lessons learned from the test (what can you improve in future tests?)

I also encourage you to create additional spreadsheet where you can list lessons learned from all the tests. It’s a valuable resource to recap your experience, as well as for new team members to get on board effectively.

After working with dozens of clients we’ve learned, that these three tips stand out as the most helpful tactics for a solid conversion optimization process. Start with basic analysis to get your first A/B test up and running quickly, spend time with your customer success and sales team to really learn the problems customers face, and keep a track record of your tests. Committing to these steps will help you secure executive buy-in, run more successful experiments and build a culture of continuous improvement.

Digital & Social Articles on Business 2 Community


The Lunacy of Our Election Process Puts Too Much Emphasis on Iowa and New Hampshire


When the nominees are in place, I’ll go into the other crazy aspect of the electoral college that makes my vote meaningless. For now, I’ll discuss why the use of primaries spread out over months is utterly ludicrous.

As a California Republican, I literally have zero vote for President. By the time June, 2016 rolls around, there’s a pretty good chance that the GOP nominee will be all but locked in. Since the Republicans will absolutely not win California in the general election, my vote will be meaningless next November as well. How can we consider this process logical, fair, or remotely intelligent when many people have absolutely no influence on the outcome?

Then, there’s Iowa and New Hampshire. There’s a reason that some candidates spend dozens of days in these states while only coming to California for the occasional fundraiser. The voters in these first two primary/caucus states have exponentially more influence over the Presidential election than the vast majority of Americans. Their vote literally counts more than mine. They don’t have special voting knowledge. They haven’t done anything to earn their spot of having “supervotes” compared to my meaningless votes other than the fact that they live in a state that the system has deemed to be more important than mine (and probably yours, depending on where you live).

They will be able to vote for candidates who will no longer be an option for me. That’s assuming that I have an option at all. By the time they make it here, there’s a possibility that both parties will have nominees and American citizens in several states will have had no say whatsoever. In other words, our votes simply don’t count.

There is no argument that can be made to prove to me that it makes sense to have a system like this. Before anyone thinks that it’s sour grapes because I live in California, keep in mind that I would be griping just the same if I lived in Iowa or New Hampshire. The system is completely broken. The people in those states will have hundreds of opportunities to meet the people who are trying to lead this country while most people will be lucky to have a single opportunity.

This all made sense back in the 18th century when communication and travel were much slower and the size of the country was much smaller. Today, it makes absolutely zero sense. I’m certain that someone can find a logical, legal way to fix this system, but apparently nobody’s trying hard enough.

Chris Christie has spent 44 days in New Hampshire. Have you seen him in your state? Is your vote less important than the vote of someone in New Hampshire? Unless you live in Iowa, the answer is yes, and that’s simply not right.