A/B Testing: Overview
When to use it
A/B Testing allows Salsa campaign managers test different versions of an email blast to see which one performs the best. To do this, follow these simple steps:
- Create your email blast as normal.
- Copy the original email into two test versions of the email (using the tools described below).
- Send the variations out to a small percentage of the total targets for the email blast.
- Monitor the open rates or conversion rates of each variation and determine which was the better performing email.
- Send the content from the better performing email out to the rest of the target.
How to get there
Complete your email in the Salsa HQ as normal. At Step 7: Summary, click the button before scheduling or sending the email.
Please note: First, do not use Percentile targeting and A/B testing together, as this will result in 0 targets receiving emails. Use only one of these functions at a time. Second, if you are considering using different email templates between the main blast and the test blast, make sure that both templates you are using are 'uneditable' type templates.
Here's a quick slide show that outlines this process (which is documented in more detail below).
Create Split Blasts
Click the primary and then the secondary button to create split blasts.
You've just created two copies of your main blast:
- Test A
- Test B
Target Test Blasts
By default, your targets will be as follows:
- Test A will be sent to a random 5% of the supporters you targeted for this email blast. You cannot specific which 5% - it's random by design.
- Test B will be sent to 5% of the supporters you targeted for this email blast. You cannot specific which 5% - it's random by design.
- When you're done testing, the "winner" of the test will be sent to 90% of the supporters you targeted in your initial email blast setup.
Use either the editable percentage fields or the slider tools to vary these percentages, if desired.
Editing Test Blasts
Click the edit link next to the test blasts and the following box will drop down:
Once you've made changes to the test blast, don't forget to save them!
Once you have made edits to all of your test blasts - submit them using the buttons. This will launch the test blasts to the specified % of people and bring up the open rate statistics on this summary page.
Note: After you send out the test blasts, you need to give the test some time. You're waiting for real humans to receive your email so that you can evaluate their response. Depending upon your supporters and your content, wait at least one hour, but up to several days for best results.
Evaluate Test Results and Send Main Blast
After waiting for the appropriate length of time, you'll see test results on the summary tab.
NOTE: Only open and click rates are available in this view. If you'd like to see conversion rates, check the email summary report.
If Test A performs better than Test B, replace the main blast content with the content from Test A. To do this:
- Go to the main blast drop down labeled: Replace Main Content with:
- Select the version you want to send to the remainder of your targets (usually the best performing in your test)
- Accept the confirmation "Are you sure?" message
- Once completed, the drop down will change to "Leave as is" again.
Your blast is now ready to be sent with the better content. Once you've selected the content for the main blast, schedule and submit the email as you would with any other blast.
Best Practices for A/B Testing
Only test one variable at a time! If you change multiple things, you won't know which thing resulted in better performance and your test will be invalid. Some ideas for varying your email blasts:
- Vary subject lines.
- Dynamic content and merge fields (personalization of content)
- From name
- Vary the color on call to action buttons
- The time the email blasts are sent