The Definitive Guide to A/B Tests in Your Email Marketing

We spend a lot of time on this blog discussing how you can optimize your email campaigns for your audience. We do this with tips on deliverability, email design, and optimizing your team. But, an important aspect of not only email marketing, but really any kind of marketing – is testing. 

You wouldn’t release a new product without first testing if it works, so why waste time sending emails that don’t work. 

In order to test what campaign works best, what message resonates with your audience, or what CTA generates the most clicks, you can use A/b testing in your email campaigns.

If you’re asking – “What is A/b testing” or “how in the world can I do that”… Well, we hear you and help is on the way. We’ve compiled the definitive guide to A/B tests in your email marketing. 

A/B testing explained

A/B testing allows you to compare and contrast two versions of the same piece of content. You can use it to test everything from website copy, to paid search ads, and of course marketing and transactional emails.

In the context of email, A/B testing allows slightly tweak a part of your emails to test which version more often generates opens, clicks, and conversions. For example, does adding an emoji to a subject line increase opens? Does a bright red button (instead of a white button) increase clicks?

Split testing varies from simple to complex testing. Simple A/B testing includes one or two elements which are easy to customize like described above  (e.g. subject line, button color and size).

More advanced testing includes the customization of multiple elements in your email campaign like picture placements, overall messaging, personalization, or comparing different email templates against each other.

The benefits of split testing your email campaigns

To get the best results possible, you need to test and analyze all of your email campaigns (marketing email, transactional emails, and email automation workflows). While the cost of acquiring new customers and newsletter subscribers can be high, the incremental cost of improving your email conversion through A/B testing is minimal.

A well-planned split test increases the effectiveness of your email marketing efforts. By using controlled tests, you will figure out which content and visual arrangements work best for your target groups. If you know what works best, then it is easier to send the most effective email for your audience. You can do this manually, by monitoring performance and sending future campaigns based on your analysis. Or, better yet, you can automate your A/B tests with tools like Mailjet to send your campaign to a small sample audience (e.g. 20% of your list) and once you collect enough data, Mailjet’s A/B test tool would send the best performing version to the remaining 80% of your list.

Even with small testing and optimization afterwards, the Return On Investment of A/B tests can be massive. Split testing allows you to significantly increase the open rate and click rate. The result is a significant increase of leads, sales and revenue.

What you can test

You can test nearly everything! You can test design and text related elements. In fact, to understand which campaign performs best, you need to test both.

Make sure that your audience falls in love at first sight. You can reach this with a killer designed email. However, the design may be the reason they do not delete your email right away, but what really drives the conversions is the content you provide. In the following section, we give you a step-by-step guide outlining which elements you should focus on when starting your A/B tests.

A/B test subject lines

The very first thing you should test is your subject line. Your hard work may be for naught if your audience does not even open your email. So, make sure you create a subject that encourages them to open your email.  You can try out clear messages (“Our special Christmas offer for you”) or subject lines which are more mysterious (“You really do not want to miss this offer”). You can even play around with emoji’s, pre-header texts, personalization to drive clicks. 

A/B test images

Now that your killer subject line drove a ton of opens, you can now focus on optimizing the content of the email. Pictures and other visuals will be the first thing that catches the eye of your reader.  Try different banners, product pictures, and other captivating images. etc. You could also experiment with GIFs, and video previews, and other visuals to drive

The internet is in love with video, and we assume your audience is too. So, do you include videos in your marketing campaigns? If not, then you really should. It can really make the difference. And if you do so, then try some A/B testing. You can start testing the size and placement.

A/B test copywriting

Sometimes, fancy visuals and funky subject lines aren’t enough to convince your audience and your paragraph texts need to win them over. Try different wording, text length,, and placement. Focus on the key messages and wrap the other elements around it.

If you are an online shop, you could test price or different discounts, headings, text sizes, colors, placement.

A/B test calls-to-action

When your headlines, subtitles, and paragraph text are optimized, then your audience is willing to click on your Calls-to-Action (CTA). CTAs buttons are one of the most important elements in a marketing email. This is what it all comes down to. Your mailing is meant to get their attention, but above all, it has to generate leads to your website.

So, do not forget this in your testing. Play around with colors, sizes, text, placement etc. Keep in mind that the CTA text also needs to be on point.

A/B test links

Besides CTAs, there are other links you can include and test. An example of these links is Social Media buttons. Getting in touch on multiple channels is necessary. Most brands now include Social Media buttons in their marketing emails, linking to other channels like Twitter and Facebook. To see how to get the best engagement, test different formats, colors, and sizes. 

A/B test sending time

Last but not least is the ability to test the date and time of your campaigns. Does your audience prefer to open their emails in the morning, in the evening, during the week or at the weekend? Use your testing to find out, and in fact, you can then create segments for each, to maximize engagement going forward. 

The 4 steps to your A/B tests

The best way to run a successful A/B test is to follow a strict process. It will help you to get profound insights from your campaigns. A proper process need to include the following steps:

1. Problem identification:

Study your email campaign statistics. Define the user’s behavior and find the problem areas in your conversion funnel. Include the landing pages your audience reaches after clicking a link in your email.

2. Defining a hypothesis:

Based on your analysis, build a hypothesis. Define which result you expect from which changes. For example, a hypothesis could be:

“My customers do not like to scroll down. Putting the Call-to-Action button at the top will increase their attention and results in conversion.”;

Or…

“Most of my readers open my newsletter on their smartphone. Increasing the size of the CTA button will make it easier for them to click on it which results in more conversions.”

3. Testing the hypothesis:

Based on your hypothesis, set up the split testing. Create a variation and A/B test it against your current email template.

4. Analyzing the test date and draw conclusions:

Once you’ve successfully sent out your split email campaign to the defined target groups, now it’s time to monitor the results. Which variation performs best? If there is a clear winner, then go ahead with its implementation. If the test remains inconclusive, go back to step number two and rework your hypothesis.

The 7 rules to A/B testing

To do a proper split test, you need to follow these 7 rules.

Rule 1: Set goals – know what and why you want to test in your email campaigns

Testing without a specific goal is just wasting time. Don’t pull randomly select an A/B test for now reason. Know the reason why you want to use split testing (increase open rates, increase click rates, test new messaging, pricing models) and think about what changes may get you the desired results.

Rule 2: Focus on frequently sent emails

The moment you start conducting A/B tests, you will be on fire and want to test every single email campaign you are sending. But stay calm, take a deep breath. To start, only focus on the emails you are sending most frequently.

Rule 3: Split your list randomly

Choose a smaller, randomized portion of your contact list to test for the most optimized email version before sending the campaign to the rest of your contact list. To get conclusive results make sure you choose the same sample sizes.

Rule 4: Test one element at a time

To best be able to identify which variation works best, focus on just one element at a time and leave all other variables the same. For example, create a few different CTA colors, but do not change anything else. This way you can identify whether an increase in engagement is because of the CTA color. If the color and the text are both tested at the same time, then how can you tell which change drove the most clicks?

Rule 5: Wait the optimum amount of time

If you are automating your A/B tests, by sending to a small sample first and then the full list after the test is complete, then it’s important to wait long enough to gather enough data. Usually, we recommend waiting 3-5 hours after sending to your sample before sending to your full list. 

Rule 6: Check if results are statistically significant

The struggle with doing A/B testing is having a large enough sample size. Use a A/B Sample Size Calculator to find the right sample size.

Rule 7: Test and test again

After the testing comes more testing. Now you know the best subject line, calls-to-action, and hopefully more. Now, you can try testing another element. Rise and repeat.

Now you know everything you need to start A/B testing your marketing campaign: the elements you should focus on, the best practice process and the 7 rules you need to follow. Sign up for a Mailjet account and let’s get the split testing party started.

What is your experience with A/B testing? Which elements do you want to test? What improvements do you see? Tell us all about it on Twitter with the hashtag #emailmarketing.