Feature toggles, such as unleash-hosted, are a great tool to enable A/B/n experiments. Talking to customers, we often find that many are really eager to start experimenting but they often find it a bit difficult to get started. Blockers may be many, one that we repeatedly hear is inter-team dependency to get up and running. At unleash-hosted we truly believe that development teams shall be as autonomous as absolutely possible. What is important is to empower the team with the tools and mandate to be as independent as possible from other teams and their priorities. Our experience is that it is common to put the ownership and responsibility of the shared analytics tool into one dedicated team. The result is that this analytics tool team gets a bottle-neck for the teams that are to enable A/B testing. This guide will show you how you can get started with A/B/n experiments in 3 simple steps using unleash-hosted – all handled within the team

 Before we get started, there are some basics that you should have in place

  • Unleash-hosted – this is to create and manage the A/B test in your application
  • A preferred analytics tool. In this guide we provide step-by-step guide using Google Analytics
  • Clear understanding of the business goals of the development team – e.g. the development team need to understand the KPIs they want to improve through their A/B testing

Our reference case is built on a joint effort we did together with the Norwegian Labor and Welfare Organization (NAV). NAV is using unleash self-hosted part of their software toolbox.

Step 1: Define an alternative to the targeted KPI

The first enabling step for any A/B/n experiments, is to define an alternative that we believe will perform better on the decided target KPI. For simplicity – let’s assume that the experiment will test the color of the call-to-action button, the team expect a green button will perform better than the existing blue.

The team then need to decide on what part of the traffic they want to perform the experiment on. In this case, the team decides that they want to run the experiment on 50% of the traffic. 

  • Log into your unleash-hosted instance
  • Create a new feature toggle by pressing the “+”
Create new feature flag
  • Give the feature toggle a name according to your naming convention. In this example we have chosen “AB_experiment”
  • We also recommend that you give the feature toggle a short description – this is for your reference and documentation 
  • Now you need to choose an activation strategy. For this example we have chosen to run the experiment on 50% of the traffic – hence we choose the “gradualRolloutUserId”. This activation strategy means that the unleash-hosted engine picks 50% of the users to be part of the experiment. We now this – as the percentage is set to “50%. We also want to make the experiment sticky – which means that if a users have seen the experiment once, we want this user to have a consistent experiment. In this example, we are using the UID to create the stickiness. UID is part of the user context in the Super Insurance web application.
Choose gradualRollout activation strategy
  • Now we want to create the variants that contains the actual experiment.
    Here we define a new variant as well as a control variant. It is worth noticing that all the variants will be weighted equally. You are able to define many variants – in this example we have created 2. “BlueButton” is the control variant, while “GreenButton” is the alternative we want to test.
    Enable two or more variants

    Step 2 - Set up the experiment in Google Analytics

    To visualize your A/B/n experiment results in Google Analytics, we choose to use “Custom dimensions”. 

    You will find this by logging onto the Google Analytics admin console. Click “Custom definitions”, “Custom dimensions” and then “New custom definition”.

    Setting it up in Google Analytics

    In this example, we have chosen to call the Custom dimension “featureToggleName”. The name itself is irrelevant to the Unleash SDK, but the user need to know the id of the dimension reserved for the Unleash data. We have also chosen to use the “Session” scope on the dimension. Usually a user that returns to the site after a day or two, the experiment might not be that relevant to that specific user. This article provides additional details on the available scopes for Custom dimensions.

    Step 3 - Connecting the dots

    The last step in setting up the A/B/n experiments, is to connect the experiment with the metrics. The developer in the team now need to enter a few lines of code to get the magic happen.

    In the source code, where both the Unleash SDK and Google analytics SDK is running.

    const toggleName = ‘AB_feature_toggle’;  
    const variant = unleash.getVariant(toggleName);  
    console.log(variant.name);  

    // Maps ‘dimension4’ to ‘unleash’.  
    gtag(‘config’, ‘UA-YOUR_ID-1’, {  ‘custom_map’: {‘dimension4’: ‘unleash’}  
    });  

    // Sends an event that passes ‘variant’ as a parameter.  
    gtag(‘event’, ‘unleash_dimension’, {‘unleash’:`${toggleName}:${variant.name}`});

    Your experiment is now up and running. You are now able to explore the results of the experiment using your already existing reports and views in Google Analytics. The Custom dimensions are available as a secondary dimension that you can easily access.

    If you are using Google Tag Manager you can pass the data to the data layer and onwards to your analytics tool such as Google Analytics.

    First, create your event JSON and push it to the data layer like so:

    dataLayer.push({ //send data to GTM

    ‘event’: ‘unleash’, //name your event

    ‘feature’: ‘new-logo’, //name of the feature

    ‘variant’: ‘control’ //name of the variant the user sees

      })

    Then, in Google Tag Manager, create 4 tags

    • One custom event to trigger when your Unleash event is passed
    • One data layer variable to store the name of your feature
    • One data layer variable to store the name of your feature variant
    • One Google Analytics event tag to send the data to Google Analytics when the custom event triggers

     

    Ready to get started with unleash-hosted?