A/B Testing Using Contentstack and Optimizely

A/B testing is a digital marketing practice that helps businesses to evaluate their conversion funnel and marketing campaign by getting data directly from the customer behavior.

In A/B testing, two versions of a particular web page and other digital assets are compared and then measured to gauge the difference in their performance.

Once you have created your variations, you can have one set of the group, let's say group A, try out the first version and the other group, let's say group B, try out the second one. Through this, you can gauge if there are any changes in the behavior between group A and B.

Prerequisites

• Contentstack Account
• Optimizely full-stack account
• Working knowledge of React.js
• Node 8+ installed on your machine

Note: Free trial account will not work for this exercise. You'll need a full-stack Optimizely account. We assume that you already have one before starting with the exercise. If not, create a new Optimizely account.

Steps for Execution

1. Download the code
2. Add and publish entries
3. Implement A/B Testing
4. Try it out

1. Download the Code

   First, clone or download the GitHub repository. Once you have the code, you'll find a zip containing the content types needed for this example. Unzip content-type.zip and import them into your stack, as discussed in our Import a Content Type documentation.

   

2. Add and Publish Entries

   Before adding and publishing your entries, you need to create an environment.

   We have created 3 content types for the exercise. The most important one is the Home content type. Add data Now while adding data in the entry, you'll find a group name Home Variation. In this group, add different images.

   • For this exercise, we're using three variations for the home page banner.
   • These images will be assigned to the users based on their cookies setting when they load the home page.
   • For every user who navigates to the home page, he will see a different banner image as per the cookie assigned to that user.
   • Cookies here act as a unique identity for every individual user who navigates to the home page.
   • Based on that cookie, the banner image may appear different to different users based on the traffic distribution which is assigned to each variation in Optimizely's experiment A/B Test.

   The Banner group will look somewhat like this:

   

   Add data in all other content types and publish them to your environment. We have used the "development" environment for this exercise. 

3. Implement A/B Testing

   Follow the steps given below to implement the A/B testing for your website:

   1. Log in to your Optimizely account and once you reach the dashboard, click on Experiments from the left-side panel.
   2. Then, click on Create New and select A/B Test.
   3. On the New A/B Test page, provide a suitable Experiment Key (we have used buyOption in our example) and an optional Description, and click on Create A/B Test. 

   4. You'll get two default variations, that is, variation_1 and variation_2 with traffic distribution 50% each as shown below:

      
   5. Now add a metric by clicking on Metrics which is basically an event like clicking on Buy option, the bounce rate of the user visiting the website.
   6. This can be done by selecting Create New Event from the top-right side of the events section.
   7. Provide an event key name, in our example we have named it "buyOption".
   8. Then save the event. After saving the event navigate to the Metrics section within your experiments. There will be a block from where you can select your created event as a metric for your experiment.

   9. Select the event and save the metric. Now your metric is set with the specified event.
      

      
   10. Save and add that metric to the experiment. Now run the experiment on an appropriate environment (we have used the production environment in our example) by clicking on the Run button. And based on your environment, make sure you have your SDK key handy which is required while creating an instance of Optimizely's react SDK.
   11. To get the SDK key, click on the Settings option located on the left panel. You will find the required SDK key.
   12. Now that the experiment is running, open the project in any code editor of your choice.
   13. Open the .env file and enter your stack credentials, content type UID, the Optimizely SDK key, and the experiment/feature key. By default, the system supports the US region. If you want to add support for the Europe region, Azure North America region, or Azure Europe region, replace the value of the NA region to Europe (eu), Azure North America (azure-na), or Azure Europe (azure-eu) respectively in the .env file.

   Additional Resource: For more information, refer to the Contentstack regions documentation.

   14. Now open your command prompt and move inside the project root directory. Then, install the required dependencies by running the following command:
       
   15. npm install
       

   16. Note: Usually, the above command will install the Optimizely SDK. But you can get it installed otherwise using the following command: npm install @optimizely/react-sdk

   17. Now staying in your project's root directory start the app on port 4000 using the following command:

       npm start
       

   Open the browser and hit the localhost at port 3000, you'll see a banner image based on the userID from the cookie.

   So whenever a new user opens the home page in a new tab, a new cookie is assigned to the user based on which the banner image may vary.

4. Try it Out

   You can open the Home route or page in multiple tabs to see the difference in banner images. Now you can click on the Buy option to know which user is clicking more on that Buy button based on the variation displayed to the user.
   
   Based on the click, the result can be determined which variation is better and on which variation the user will more likely click on the Buy option as shown below:

   

Additional Resource: To implement personalization for your Contentstack-powered websites using Feature Test, refer to our Implementing Personalization Using Optimizely reference guide.