Content A/B testing

With content split testing, you can test 2 different mailings against each other with a portion of your audience. We'll send the mailing with the highest response to the remaining recipients. It's a smart and easy-to-use tool to learn what works best for driving up metrics with your audience and get the best results from your mailings!

Content A/B testing is only available for Administrator and Full permission levels.

Creating a content split test

Before you get started, you'll need two separate mailing drafts. Once you're ready, you'll head to the Campaigns tab of your account. Click the Create a New Campaign button and select A/B test on the modal that appears.


This will take you to the setup page where you can choose your mailings, name your test, choose your audience, and more:

First, choose the mailings you want to test. Click ‘Choose version A’ and then choose your draft from the dropdown menu. You’ll then see a modal that will allow you to enter your preferred subject line, preheader text (optional), and the name and email address your message will be sent from.

 

Once you’ve inserted both of your email versions, you’ll see a preview of each that includes your subject, preheader text, and from information.

Once you’ve chosen your mailings, you can fill in the test and send settings for your A/B content test.

• Content A/B test name: Like a mailing name, this is only used internally for your reference and is not visible to recipients.
• Test Group Size: When you send a content split test, Frankie will distribute your mailing variations to the test group first. Here you can select the size of that test group. For example: if you're sending to an audience of 60 contacts and set the test group size at 50%, we'll test with 30 contacts: 15 will receive Mailing A and 15 will receive Mailing B.
• How should the winner be chosen? Frankie sends to your test group and the remaining members of the selected audience will receive the mailing with the best results, based on your selection here:
  • Highest click rate on entire mailing: the mailing with the most overall clicks wins. If this option is selected, you’ll need to enter the link URL for both version A and version B exactly as they appear in your mailing
  • Most clicks on a link in each mailing: the mailing with the most clicks for a specific link wins
  • Highest open rate on entire mailing: the mailing that's opened the most wins
  • Choose winner manually: if you prefer to choose the winner
• Send winner after: You'll also set the length of time the test runs. For example, if you choose "After 1 hour", Frankie will send to your test group and gather results for one hour before automatically sending the winning mailing to your remaining audience.
• Audience: For split testing, you'll have to send to at least 30 recipients. When you select your audience, you'll have the option to send to your entire audience or any groups / segments you choose. If you need to suppress a portion of your audience from receiving, you can check "Exclude audience segments" and then select one or more of your audience segments.
• Send test at: You can choose the date and time you’d like your test to start.

Once you've made all of your selections, you'll need to review your A/B test. You’ll see a modal that will show your A/B test settings, subject lines, from information and preheader text from both versions, your audience, and when your test is scheduled to send.

The modal will also include a full preview of your campaign, allowing you to toggle between version A and version B.

 

From here, you can edit your A/B test, or you can confirm and schedule.

Reading the response to your test

Once you've begun the test, you'll be taken to the response. Here you'll find a summary of your test, including the criteria, send time, time remaining, and audience used. You'll know your test is in progress because you'll see this in the top right of the app:

 

If needed, you can cancel the split test by choosing Stop test. You will be prompted to confirm this before Frankie stops the test.

The response for each mailing is displayed; you'll need to refresh the page each time you want to see updated results.

If you've set your test to autosend the winning version, it will be sent at the end of the testing period. You can manually pick a winner any time by clicking the Choose As Winner button. Once testing is finished, the winner will be indicated on this page:


 

Frequently Asked Questions

I selected "Choose Winner Manually" but forgot to choose a winner. What happens?
If you forget to choose a winner, your mailing will not be sent to the remaining members of your audience.

Can I add Google Analytics to my mailings?

Yes, you can add Google Analytics to the A/B test by selecting the ‘Add Google Analytics’ box under Test Settings. You can choose names for the winning version as well as versions A & B.

Can I use the Compare Mailings tool with the mailings from a content split test?
Currently, mailings from a content split test are not available for use with Compare Mailings.