Perform experiments and continuously optimize your communication campaigns with A/B testing. This feature is available both in Broadcast and Flow and there are differences between the marketing experiments that can be done using these two communication tools.
|Compare how different channels perform with your audience
Use different channels for A/B Testing
|All channels available in Flow.
|Pretest different versions and choose the winner
|Metrics which may be used to choose a winner automatically
• Open Rate
• URL Click Rate
• URL Click Rate
• Open Rate
|Test one or multiple version variations
• Name and Address (From)
• Subject and Preheader
• Character Set may be adopted to the Content
• Push Message and/or In-App Notification
• Custom Payload
|Any parameter or attribute of a message may be changed for different test versions in Flow.
|Compare testing results for different versions
||A report for each version contains Delivery Rate information and other channel-specific parameters.
||Statistics in Flow are channel-specific. However, the Number of Engaged Persons and Performance Ratio are available for all channels.
A/B Testing in Flow
Testing audiences in the Flow tool will let you see how they react to your content, what channel they prefer, or what is the best time to send communication.
Set Up an A/B Test
We will show an example with different channels involving similar content - a discount code that a company would like to communicate to its customers. After a few hours or days (depending on the timeframe specified for this communication), the metrics will show which channel would be the best choice in the future.
- In the Moments module, select an entry point for your communication (for this example case, we will use Item Added to Wishlist event as our communication trigger), and then add the Split Audience element - you can select how many audiences you want and split them in proportions of your choice. Separate branches for each subset audience will be created as seen in the image below, with people randomly being assigned to these branches.
- Select channels and messages that will best serve your desired testing scenario.
- Next, include the Evaluate Message element with keywords that will let you know the customer reaction. For Email, select link clicked instead of using the keyword.
- If the customers responded using the keyword, send another SMS and Viber message with instructions involving the promo code. Also, you can send a message to customers who clicked the link in the email message.
- Use an event such as Item Purchased. This event will let you know whether the customer has reacted to your message and completed the purchase. Think about how long you want to wait before sending another message to them based on their actions. For example, you could wait 2 days.
Finally, if you want, you can send a thank you message to those who made a purchase. You could also decide to continue the Flow by sending another reminder to those who have not reacted to your communication.
After you Validate and Launch the Flow, you can go back to it and check the Flow Performance.
The outcome of the A/B Testing shown in this article offers a couple of possibilities, including the continuation of the same campaign or planning future communication with customers.
Use Performance Data For Future Campaigns
In the Moments module, click on your campaign to see how the channels performed. If, for instance, SMS was the most successful one, your next campaign could be focused on the SMS messages to customers.
The data you will see when hovering over each channel (in this example case SMS, Viber, and Email), involves the number of customers that the message was sent to, the total number of engagements that includes possible multiple entries to the Flow by some customers, and the total number of the sent messages.
Pretest and Send Winner Message
Another A/B Testing option in Flow is to add a tag to your communication. The tag would represent a winning segment (tag can have any name). The first three audience branches are used for pretesting, and the best performing message you can later send to the remaining audience branch via a new campaign that you would create in Flow or Broadcast. This we will explain in the following paragraphs.
New Campaign in Flow
Go back to the Moments page and create another flow same as the winning one or duplicate the existing one. But this time use the tag that involves the remaining audience.
- Select the Predefined Audience starting point and select tag winning_segment.
- Add the SMS message.
- Add Evaluate Inbound SMS.
- Add another message and a Wait for an event element; in the side panel, select Item Purchased event. Select the wait time 2 days.
- The final Flow would look something like this:
New Campaign in Broadcast
Another option would be to use the AB testing result for Broadcast, if the one-way communication is enough for your scenario.
Use the winning_segment tag (the name for the tag is really up to you) as an audience for your communication.