Split testing
Use A/B testing to improve flow metrics.
Last updated
Use A/B testing to improve flow metrics.
Last updated
To get a quick start on traffic splitting within Formsort itself, you can assign weights to individual variants and direct your responders to the flow URL, which looks like https://<client-id>.formsort.app/flow/<flow-name>. This will serve a pseudo-random variant according to the weights given.
Assigning weight to deployed variants will distribute flow visitors according to the weight given to the variant. For instance, say a flow has two variants - variant A and variant B - and both have a weight of 50. When the flow URL is loaded, half of the responders will be sent to variant A and the other half to variant B.
Only new traffic is split among variants according to weight. Once the user has interacted with a specific variant, that variant UUID will be stored in their browser.
If they revisit the form from the same browser at a later time, the cached variant will be loaded for them and weighted variants will no longer be randomly served.
Once you have your variants appropriately weighted and your Custom Domain configured, you can point a flow to your domain and have random variants served to new traffic.
In the Domains tab of your Studio, pick a flow from the flow to load for this Hostname option. Do not specify a variant.
The flow should have at least 1 deployed variant that can be served to users.
This is the same process as using a flow URL to serve weighted variants, only with an extra layer of addresses. The custom domain points to the flow URL, and Formsort handles splitting incoming traffic.
Another, more comprehensive pattern is to add unique variant URLs to an upstream testing tool such as Optimizely. These purpose-built tools are much better at handling traffic splits and can aggregate experiment data that spans across the user journey - either upstream, downstream, or inside of Formsort.
This pattern does not require weight to be added to individual variants, since traffic should be split upstream of Formsort. The only thing you'll need is the deployed variants you want to use, and the URLs for them!
The variant URL looks the same as a flow URL, only with the path to the variant specified:
https://<client-id>.formsort.app/flow/<flow-name>/variant/<variant-name>
In the process of testing different variants, there will come a point where you'll want to move off a particular variant entirely. The thing to be aware of is that returning responders will still get directed to the variants they have previously interacted with (due to variant UUID caching), so it's likely your analytics will pick up events on variants that you thought you deprecated long ago. This can be a problem!
The solution here is to archive the variants you want to deprecate, and set a redirect to URL. This pulls old or unwanted variants out of circulation for all traffic, and ensures that returning visitors are pushed to the variant you want them to see.
But wait, won't Superseding on deployment push all users onto the most recent variant deployment?
For the most part, yes. Deploying a superceding update will:
ensure new users have the most recent variant revision served and,
push users that have previously interacted with the variant onto the most recent variant revision.
The users that will not get pushed to the "superceded" revision are those that are returning after visiting a different variant.
Let's say you you're ready to move on from variant A, so you:
Push a superceding update to variant B.
Point all your links and custom domains to variant B.
Remove the weighting from variant A.
However, there are users have visited variant A already!
The returning "A" variant users will still have variant A served when they return to finish their form, even if you've removed weigthing from variant A. This is because the UUID for the variant is still cached in the returning user's browser.
At this point, it's time to deprecate variant A entirely:
Archive variant A.
During archiving, set the redirect to URL to either variant B, the flow URL, or your Custom Domain (which in turn may point to the flow URL).
Once A is archived, no users will be able to access it, and returning A variant users will be directed to variant B!
