In this document we’ll cover:
- How the A/B testing plugin works
- When to use the A/B testing plugin
- Setting up an A/B test
- Useful testing tips
How it works
The A/B testing plugin will help you test different flow versions of your bot as a tool to calculate and improve conversions.
You’ll find it in the plugins list in your Chatfuel Dashboard.
When to use the A/B testing plugin
Let’s say you have a funnel of three blocks linked with buttons (Block 1 → Block 2 → Block 3), and you see that Block 2 is not performing very well. You can:
- Create an alternative “Block 2.1”
- This will help you lead 50% of users to Block 2 and 50% to the Block 2.1
- By setting a goal, you'll have the option to track conversion to Block 3 by setting up user attributes in the beginning of Block 3—or, set a deeper goal if you’re already tracking this.
Setting up an A/B test
In this step, you’ll choose which values and attributes you want to use and track. For example, you could select Attribute: Lead Status, value: ‘Completed,’ which signals success (user went through a purchase process, or a lead became a customer in some other way).
- Build an alternative flow
Before adding the A/B testing plugin, let’s set up our flow and attributes. For example:
Let's say you have a flow with two options:
First, define the attributes. If user clicks ‘Yes,’ set attribute to ‘Completed.' If the user says ‘No,’ add the ‘Incomplete’ attribute. (Note that you can use more than two options here.)
Adding attribute using the button Actions is only available for Pro users.
3. Creating an A/B test
In this step, you'll be adding the A/B testing plugin to divide people into two, three, or four groups to test the main and alternative flows for your bot.
You'll see the option to set a user distribution percentage, and decide what percentage of user traffic you will be directing to each variant.
4. Set up a goal
For this step, you'll set your goal based on the desired attribute and values. In this case, we’re looking to see our success rate for Lead status: Completed.
5. Collect data
Wait until at least 10 subscribers have gone through each flow, so you'll have enough data to compare results for the variants.
6. Review the results
You will then be able to check stats for each flow. You'll see the total number of users, number of conversions, and conversion rate for each variant, like this:
- Do not edit the experiment after launching, because it may skew results.
- Use 50%/50% split for faster results.
- Check conversions on different levels—not only right after the alternative version you’re testing, but also some deep-level conversions (i.e. real purchase that can take place in a day).
- Note that the A/B testing plugin remembers each person that comes to the plugin and preserves their "bucket." For example, if you received bucket A the first time, you’ll still receive bucket A even if you re-enter the plugin later.
- Chatfuel creates an attribute for each A/B testing, so you can always see which subscriber visited which path.
- If you'd like to start another test, rename the block and create AB test, or create new block with AB test.
If you have any trouble with the A/B testing plugin, reach out to us at firstname.lastname@example.org and we’ll do our best to help out. 😉