• images

    4.98 Rating

  • images

    5.00 Rating

  • images

    4.2 Rating

  • Home
  • » Insights » 10 Huge Mistakes Marketers Make While Split Testing
  • Feb 08, 2019
  • Insights

  • images2830 Views

Team TIS

Share this article

For any business to run effectively, a major part is its marketing analysis and research which many entrepreneurs prone to miss out. The tendency of mere selling without factoring in what is working and what is not is determinate to the site’s growth. It blocks the deeper requirement for intensive market research, and as a result, in the long run, you lose customers. The best way to conduct client research is by split testing.

Why split test?

Split testing or A/B testing, by definition, is a method of conducting controlled yet random experiments with the definitive aim of improving the website traffic. To make it simpler, it is the comparison of two versions of the same page to determine which is more effective. Ideally, there will be just one or two differences between the two versions to pinpoint the effectiveness accurately.

Common actions like content, clicks, form completion, and purchases are analyzed to see what variant delivers better results to a pre-defined marketing objective. The common marketing methodologies used are signup forms, registration pages, call to action buttons or a redirect to a different page. It is proven that even as small a change as updating one word in the CTA has helped increased conversions by a whopping 77%. This proves the importance of doing split testing to improve your marketing campaigns’ efficacy.

As enticing as it might sound, sometimes marketers are seen likely complaining about getting false negative analytics or couldn’t even generate adequate data. If you are fazed by split testing, chances are that you might be conducting it the wrong way. Here is a list of 10 most common mistakes that testers are likely to make along with their possible fixes.

Mistakes and fixes worth knowing about

There can be a multitude of errors when a tester tries to comprehend a split analysis. The most common ones are jotted below.

1. Arbitrary Testing

The biggest problem that most testers are bound to make is conducting a split test without a reason. For instance, you might be apprehensive about testing the size of the ‘call to action’ button, in which case you can design the variants with specific focus. If you are conducting a split test just for the sake of it then please refrain as you are about to get doomed.

The Fix:  Use heat map software to discover the potential areas that aren’t focused much or aren’t garnering much traffic. Conduct a split test and form a hypothesis first. Now conduct the trial and make sure to conduct it for the proper phase of time, compare the new heat map data sets and analyze. Keep repeating until you get satisfactory results.

2. Calling off the test early

This is a big rookie mistake that most of the testers are bound to make. Let’s assume that your site conducts high traffic and hence within 3 days of split test you generate 98% confidence and around 250 conversions per variation and you pull off the test. Here is where you test gets a false positive result as you have not taken in account the seasonality parameter, even which day of week you conduct the test can lead to significant variation in test curve.

The Fix: Another important parameter for any statistical analysis is the sample size taken. For getting proper results make sure to take your sample size large enough viz. 100 or 1000 conversions. Too small a sample size taken can lead to wrong implications.

3. Multi-element testing

Website heat mapping analysis might suggest that there is more than one area that needs focus however conducting multiple tests at same time ends up nowhere.

It is always better to opt for split test instead of a multivariate one. This is because when you are conducting a split test on two pages having differences in only one parameter it is easy to judge which page is working better.

The Fix: However in case of multivariate testing, assume that you are testing four different web pages with two or three parameters different. In that case when you garner the data you cannot fix the actual deciding factor. In such occasions, you have to compare the data from all pages and also analyze the correlation between different parameters.

4. Focusing only on traffic conversions

When you are testing for a particular or a couple of parameters make sure to be deep rooted instead of just thinking short term. Which means if you see certain changes are gathering more traffic on your site, don’t get all complacent about it. If the high traffic comprises of low quality customers then this eventually might yield negative results for your business.

The Fix: So whenever you are opting for a split test check your conversion metric then correlate that traffic with actual leads and see how many generates to potential customers.

5. Opting for a random hypothesis or just blindly following split test practices

A statistical test has no significance without a proper hypothesis. So, before even wasting your time conducting a test make sure that you have a proper hypothesis. In case you are unsure of the credibility of your hypothesis go for market research, analyze the results for your chosen variable, check your competitors’ strategy, and know your targeted customers.

The Fix: Also while checking for competitors’ strategies make sure not to opt for blindly following them. What worked for someone else isn’t going to work for you. Check your competitors’ strategies but at the same time know your own USPs and strategize accordingly.

6. Eliminating the confound variables

The confounding variables are those elements that are not part of your significant hypothesis parameters and are likely to mess up your test results. Some of these include new product introductions, marketing campaign launch, and website redesign.

This generally happens when you change some test parameters in the middle of the test to generate more significant variation. This might generate traffic from places outside your potential target pool.

The Fix: When performing a split test make sure to eliminate such confounded variables, and make sure that rest of the factors remains constant throughout the test.

7. Testing only incremental changes

There is a significant difference between how large websites operate and how rest of the small entrepreneurs need to deal with. For large websites some minute incremental changes can lead to generate big ROI.But for startups and smaller companies, this activity may not yield the expected outcomes. For example, it’s not quite feasible to test the website’sor the CTA button color and its various shades, as it is going to add very little to the overall site facelift.

The Fix: Split tests provide for minute improvements but that won’t give significant turnover in case of small businesses. So don’t get focused only on incremental changes, instead, focus on huge performance lifts. What will be needed is to go for a radical change at an overall level. This is more intensive than a narrow A/B testing technique. This may entail a major page redesign which can call for substantial efforts. It is also important to note that because of the multiple elements being re-designed, it might be difficult to note which particular element has resulted in a spike in traffic after the re-designed page is live.

8. Doing split test even without any traffic

If you are running the business for only a couple of months, it is recommended to achieve higher traffic,before you start running a split test. For startups and new ventures, performing split tests with a few beta users will be ineffective. The testing of different hypotheses is a game of statistical significance achieved by an optimum sample. But if you don’t have adequate sample, this underlying objective will not be fulfilled.

The Fix: Go for split testing only when you have met 3 distinct conditions

1- You have adequate representative sample

The testing needs to run for 3-4 week duration to cater to diverse sales periods. Calling off the testing before that will not depict a universal testing, rather it will show a selective/cyclic testing outcome.

2- You have sufficient sample size

A website with more than 1000 transactions (leads, signups, or subscriptions) can do with one A/B test in a month to improve traffic. Factors like conversion rate, total visits, and overall transaction will determine how much should be the adequate sample size.

3- When you achieve a p-value

Once point# 2 and 3 are met, it is important to look at the p-value (as per a common misconception p-value is not the probability of B being greater than A). Typically, an acceptable level of significance is 5% (or 1 out of 20 times, the sample will show an extreme result given that the null hypothesis is true)

9. Tests not run for full weeks

For split tests to yield proper result, seasonality, weekly and even diurnal parameters matters. So the time taken period for split test should be carefully chosen. Not factoring in full weeks test will skew the results which might show the correct picture of the result of the testing

The Fix: For instance, if you are an ecommerce site owner, thenonweekends your targeted population are more inclined to your products whereas on Mondays or rush hours of weekdays your site might not generate proper traffic.So, make sure that you end the test on the same day as that of the beginning. This ensures that you test a full week in one iteration. This will also align with our earlier recommendation – running tests for 2-3 week duration

10. Test data not sent to Google Analytics

Conversion metrics generally shows average data and in the world of marketing averages lie. You can never get a full picture if you just work with percentage data.This is because time, seasonality, diurnal phases and many other factors contribute to a proper customer graph.

The Fix: So if you have catered a significant amount of data send it to Google Analytics. Run advanced segmentand custom reports. The results will show you the path to run advanced tests and you might get an idea on where to take your site testing from there. Utilizing the new GA features allows marketers to run upto 20 concurrent A/B test data analyzed. Make sure to use a distinct Custom Dimension (or Custom Variable in GA classic mode) for one active experiment. Tools like Optimizely Classic can help in this case.

To sum it up

Marketing and business analytics are pretty complex in their own ways. So instead of checking what others are doing or webbing the most common testing strategies, go the smart route and adopt the cost-effective A/B testing to boost ROI from your marketing strategies.

You might also like...

Top 15 Marketing Automation Tools to Look at This Year

Top 15 Marketing Automation Tools to Look at This Year

Marketing is the indispensable business function that powers up the visibility of the brand. Marketing enables it to acquire new customers and build business accounts. The success of every business depends on how powerful its marketing is. Advantages of marketing: 1) It creates brand awareness There is no doubt...

Read More images
A 12 Point Check List Before You Hire a Digital Consulting Agency

A 12 Point Check List Before You Hire a Digital Consulting Agency

Your dream venture is the outcome of creative brainstorming, sheer hard work, and effective resource utilization. It’s your brainchild and you will surely want it to achieve the highest feat. Reaching out to potential customers, captivating their attention, and popularizing your products are some of the tasks involved in...

Read More images
Things To Know About Online Reputation Management

Things To Know About Online Reputation Management

There was a time when having a social media presence was something just those “cool kids” down the block did. It was seemingly started as a medium of free gossip. But looking at the staggering numbers, 2.95 billion people have logged into some social media platform or another. Clearly,...

Read More images