A/B testing: tips for online retailers + example

You are an online retailer and 100% convinced of your brand and your products, but the sales figures are not as expected? Maybe you are using the wrong marketing strategy or customers are leaving your site after only a few clicks. There can be many different reasons for this. A/B testing (or AB testing) can be used to find out the real cause and then eliminate it.

In this article we will show you what these A/B tests are all about, how you can carry them out and what advantages this can have for your company.

A/B-Testing Definition

An A/B test is a test method for evaluating two variants of a system by testing the original version against a slightly modified version. This method is mostly used in software and web design with the aim of increasing certain user actions or reactions. Over the years, it has become one of the most important testing methods in online marketing. However, A/B testing is also used in other areas, such as comparing prices, designs or elements in online shops.

A/B-Testings: Advantages and disadvantages

With A/B testing, you can test every change to your marketing activity, website, etc. and decide on the more successful version based on solid data. This brings the following advantages for you:

Advantages of A/B testing

• Higher conversion rates

• More satisfied customers/website visitors

• Optimised time and budget management

• Better insight into the needs of the target group

• Possibility to implement qualified results immediately

However, besides its opportunities, A/B testing also brings a few challenges:

Disadvantages of A/B testing

• Several tools necessary

• Only one hypothesis possible per test

• Confusion among clients

• For small sites: time-consuming and statistical significance difficult to achieve

Procedure for A/B testing

In A/B testing, target groups (such as website visitors or newsletter recipients) are divided into two subgroups: Group A and Group B. This division must be random. Depending on the target group, test objects such as landing pages or advertisements are also divided into two parts: the original variant and the modified variant. The two variants should only differ in one component, because this is the only way to clearly attribute response differences to the changes. Then use the original version for group A and the modified version for group B and compare the reactions. Reactions here mean the desired effect, such as subscribing to a newsletter or ordering a product.

In addition to improving the user experience, A/B testing is also a means of increasing conversion rates. Statistical testing techniques used for A/B testing depend on the characteristics of the data used.

A/B testing: You can use these statistical approaches

There are two statistical methods used in A/B testing around the world: The Frequentist Approach and the Bayesian Approach. Both methods have their advantages and disadvantages. The following comparison between the two methods will help you understand the differences.

A/B Testing with Shopify: How to run A/B tests step by step

All websites can benefit from A/B testing, as each has at least one measurable goal. Whether you have an online shop, a news site or a lead generation website, your goal is usually always to increase the conversion rate.

So A/B testing can be done on any page, but also for all elements of the website: From push messages to design, to buttons like "search" or filters, and many more. Here are 10 examples that might inspire you:

• Titles and headlines: For example, start by changing headings or article content to find out what appeals to users. Changing color or font can also make a difference.

• Call-to-action: the CTA button is critical. Color, font, placement, and words used ("buy," "add to cart," "order," etc.) can significantly impact conversion rates.

• Buttons: Other buttons can also be important. They can change size, shape and color to attract more visitors.

• Images: Images are just as important as text. Try different images and vary the size, aesthetics and placement.

• Page structure: the structure of your pages, be it the home page or categories, should be well thought out. You can change elements like carousels, banners or popular products.

• Algorithms: Use different algorithms to turn visitors into buyers or increase shopping cart value.

• Business model: rethink your business model to generate higher profits, e.g. by adding suitable products or services.

• Forms: Forms need to be clear and concise. Test different wording, optional fields and placements.

• Pricing: A/B testing with prices is challenging. Use your imagination to test the impact on conversion rate, e.g. by offering savings or variants.

What types of A/B tests are there?

There are different types of A/B tests that should be carried out depending on the page or test objective:

• Classic A/B testing: In classic A/B testing, your visitors see two or more variations of a page at the same URL. This allows you to measure the success of different variations of a particular element.

• Split test or redirect test: Split tests redirect your traffic to another URL or several different URLs. This can be particularly interesting if you are hosting a new site on a server.

• Multivariate tests (MVT): Multivariate tests can measure the impact of multiple changing elements on the same page. For example, you can change banners, text colours and even your design. This allows you to determine which variation works best for you.

• A/A tests: With an A/A test, you can test two identical versions of one or more elements. The hits on your website are divided into two groups, each of which sees the same changes. This way you can see if the conversion rate is similar for each group and confirm that your solution is working properly.

How to carry out an A/B test step by step

Below we go through the individual steps to help you achieve successful results in your A/B test.

1. Identification of problems and potentials

At the beginning, it is important to determine what is to be determined in the first place. In this step, make sure to only note things that are really verifiable and where there is potential for optimisation. Based on this, you can collect your user data with the help of interviews or heat maps, for example.

2. Selection of the test group

For certain test objectives, it can be worthwhile to include a test group. An example of this is the optimisation of your newsletter. Including existing subscribers there would not be very effective.

3. Formulate the hypothesis

Based on the problems identified in step 1, you now formulate a hypothesis for each. The individual variables must always be verifiable and measurable and the hypotheses should not contradict each other.

4. Selection of the tools

Step 4 is now about choosing the right tools. In the next chapter we present a selection of well-known tools for conducting A/B tests. It is important that you take the time to choose the right tools for your skills, test types and requirements. Shopify also offers built-in A/B testing tools that allow you to create variants and manage the test.

5. Performance of the test

Once you have decided on one or more tools, it is time to conduct the A/B test. Make sure you have a sufficiently large test group and a meaningful runtime. Particularly in e-commerce, certain phases such as Christmas could strongly influence visitor numbers. Here, too, tools help to determine the so-called reliability rate. To exclude coincidences as much as possible, this rate should be at least 95%.

6. Evaluation of the results

After the implementation, the results have to be considered with regard to the hypothesis. Depending on which tool you have chosen, this may already include an evaluation and archiving function. If the results turn out to be very different from your previous assumptions, you should review the initial data and your assumptions again and, if necessary, adjust the test criteria accordingly.

7. Implementation of the results

If your test result meets your expectations, you can implement all optimisations. After that, it is important to keep an eye on the website and all changes.

8. After the test is before the test

Once you have completed a test, you can start a new test with another hypothesis. For example, you can gradually test and optimise different elements of your onlineshop.

Examples of well-known A/B Testing Tools

There are now many different A/B testing tools for websites and online stores. Well-known examples include:

• Google Optimize

• Optimizely

• Ab Tasty

• Kameleoon

• HubSpot

• Omniconvert

• Freshmarketer

• Adobe Target

• Unbounce

• Visual Website Optimizer

• Convert

• SiteSpect

While tools such as Visual Website Optimizer and SiteSpect tend to be more cost-intensive due to their additional functions, there are also programmes such as Kameleoon where a free freemium account is possible for up to 2,500 visitors per month. In addition to the price, user-friendliness or usability is of course also decisive for many. Optimizely, for example, is considered a beginner-friendly option.

However, if you run a Shopify or Shopify Plus shop, there are also a whole range of great plugins available in the Shopify App Store. There is something for you there for every type of A/B test.

A/B Testing Example: Webshop

Let's say you run an online shop where you sell high-quality skin care products. You notice that you have a lot of visitors on your site, but hardly any of them make a purchase. This observation is the first step in the right direction, because with existing data you have at least already been able to identify this problem.

So the next step is to find out what you could A/B test to eliminate the problem. Surveys can be very useful for this, as they also save you from creating new variants that end up completely going nowhere.

Once you have completed your survey, you can build two different variants based on it. Let's assume that the product description has revealed a potential for improvement. In this case, you could create two different variants of this description, such as "Pore refining face cream with vanilla scent" and "Vanilla Whipped Cream" and test which one is better received, i.e. generates more sales.

Common mistakes in A/B tests

When conducting A/B tests, errors often creep in that can significantly impair the validity of the results. A common mistake is insufficient planning and definition of the hypothesis, which results in the test objectives remaining unspecific. Without clearly defined objectives, it becomes difficult to interpret the results correctly.

A test duration that is too short is also a common problem. Many merchants terminate A/B tests prematurely before significant results are visible, which can lead to false conclusions. In addition, not enough users are often included in the tests, which reduces the statistical significance of the results.

Another mistake is to test too many elements at the same time, so that in the end it is not clear which changes have positive or negative effects.

To avoid these mistakes, you should establish clear hypotheses from the outset, plan the test duration sufficiently and ensure that the sample is large enough to guarantee significant results. Regular monitoring and documentation of the tests also helps to recognise and correct unexpected problems at an early stage.

A/B tests and their impact on SEO

A/B tests are not only an important tool for optimising conversion rates, but also play a crucial role in search engine optimisation (SEO).

Through targeted testing of landing pages, call-to-action elements and content, companies can gain valuable insights that not only improve the user experience but also increase visibility in search engines. However, a common mistake in A/B testing is that retailers do not adequately consider the impact of their changes on SEO rankings. For example, changes to URLs, meta tags or structured data can have both positive and negative effects on rankings.

Therefore, it is crucial to integrate SEO aspects already in the planning phase of A/B tests. You should also use canonical tags to ensure that search engines index the correct version of a page, especially when testing multiple variants. Continuously analysing test results in terms of organic traffic and ranking changes is also important to support long-term SEO goals. By harmoniously integrating A/B testing and SEO strategies, companies can not only optimise their conversion rates, but also sustainably increase their online visibility.

Legal aspects and data protection in A/B testing

When conducting A/B tests, legal aspects and data protection issues are of great importance, especially with regard to the EU's General Data Protection Regulation (GDPR). The GDPR sets strict requirements for the collection and processing of personal data, and this has a direct impact on the implementation of A/B tests. Firstly, companies must ensure that they obtain users' consent to process their data before they take part in the tests. This means providing clear and understandable information explaining how and why the data will be used.

Another important point is the anonymisation and pseudonymisation of data. In A/B tests, no directly identifiable personal data should be processed without the explicit consent of the user. Instead, data should be aggregated or anonymised in such a way that it cannot be traced back to individual persons. In addition, companies should ensure that they only use the collected data for the intended purpose and do not store it for longer than necessary.

Compliance with the GDPR also requires that users have the option of withdrawing their consent at any time and requesting the deletion of their data. A transparent privacy policy that explains users' rights is therefore essential. By taking these legal aspects into account during A/B testing, companies can not only avoid legal problems, but also strengthen user trust and protect their brand in the long term.

Conclusion

Even small changes can make a big difference. With the help of A/B testing, you can eliminate all weak points in your website, your online shop, your app or even in marketing measures, provide the best optimised version and skyrocket your conversion rate. So take advantage of this opportunity, conduct A/B tests. Simply follow our step-by-step instructions in the article and make sure that you determine the data precisely. We hope this article was helpful for you and wish you good luck with A/B testing.