A/B Testing

Perform experiments and continuously optimize your communication campaigns with A/B testing. This feature is available both in Broadcast and Flow and there are differences between the marketing experiments that can be done using these two communication tools.

A/B TESTING	BROADCAST	FLOW
Compare how different channels perform with your audience	No	Yes
Use different channels for A/B Testing	SMS Push Email	All channels available in Flow.
Pretest different versions and choose the winner	Automatically	Manually
Metrics which may be used to choose a winner automatically	Email: •    Open Rate •    URL Click Rate SMS: •    URL Click Rate Push: •    Open Rate	Not available
Test one or multiple version variations	Email: •    Name and Address (From) •    Subject and Preheader •    Content •    Attachment SMS: •    Content •    Character Set may be adopted to the Content Push: •    Title •    Content •    Image •    Push Message and/or In-App Notification •    Custom Payload	Any parameter or attribute of a message may be changed for different test versions in Flow.
Compare testing results for different versions	A report for each version contains Delivery Rate information and other channel-specific parameters.	Statistics in Flow are channel-specific. However, the Number of Engaged Persons and Performance Ratio are available for all channels.

A/B Testing in Flow

Testing audiences in the Flow tool will let you see how they react to your content, what channel they prefer, or what is the best time to send communication.

Set Up an A/B Test

We will show an example with different channels involving similar content - a discount code that a company would like to communicate to its customers. After a few hours or days (depending on the timeframe specified for this communication), the metrics will show which channel would be the best choice in the future.

1. In the Moments module, select an entry point for your communication (for this example case, we will use Item Added to Wishlist event as our communication trigger), and then add the Split Audience element - you can select how many audiences you want and split them in proportions of your choice. Separate branches for each subset audience will be created as seen in the image below, with people randomly being assigned to these branches.
    

 

2. Select channels and messages that will best serve your desired testing scenario.

3. Next, include the Evaluate Message element with keywords that will let you know the customer reaction. For Email, select link clicked instead of using the keyword.

4. If the customers responded using the keyword, send another SMS and Viber message with instructions involving the promo code. Also, you can send a message to customers who clicked the link in the email message. 

5. Use an event such as Item Purchased. This event will let you know whether the customer has reacted to your message and completed the purchase. Think about how long you want to wait before sending another message to them based on their actions. For example, you could wait 2 days.

 

6. Finally, if you want, you can send a thank you message to those who made a purchase. You could also decide to continue the Flow by sending another reminder to those who have not reacted to your communication. 

7. After you Validate and Launch the Flow, you can go back to it and check the Flow Performance.  

Test Results

The outcome of the A/B Testing shown in this article offers a couple of possibilities, including the continuation of the same campaign or planning future communication with customers.

Pretest and Send Winner Message

Another A/B Testing option in Flow is to add a tag to your communication. The tag would represent a winning segment (tag can have any name). The first three audience branches are used for pretesting, and the best performing message you can later send to the remaining audience branch via a new campaign that you would create in Flow or Broadcast. This we will explain in the following paragraphs.

 

New Campaign in Flow

Go back to the Moments page and create another flow same as the winning one or duplicate the existing one. But this time use the tag that involves the remaining audience. 

1. Select the Predefined Audience starting point and select tag winning_segment.
2. Add the SMS message. 
3. Add Evaluate Inbound SMS.
    

 

4. Add another message and a Wait for an event element; in the side panel, select Item Purchased event. Select the wait time 2 days.
    

 

5. The final Flow would look something like this:
    

 

New Campaign in Broadcast

Another option would be to use the AB testing result for Broadcast, if the one-way communication is enough for your scenario.

1. Create Broadcast.

2. Use the winning_segment tag (the name for the tag is really up to you) as an audience for your communication.