Optimizely Experimentation: A Comprehensive Guide To The Leading Platform

In 2020, Episerver made the decision to purchase the worldwide experimentation leaders, Optimizely, and later rebranded the platform to Optimizely. The strategic union of leading forces in digital experience optimisation has been a turning point for DXP’s and since then Optimizely has continued to grow and been recognised in  Gartner’s magic quadrant as market leaders year on year whilst competitors like Sitecore, have fallen behind. Optimizely emerged as a pioneer in the field of web experimentation and optimisation and this fusion empowered enterprise level organisations to elevate their online presence, delivering personalised and data-driven experiences.

What is Optimizely Experimentation?

Don’t just guess, test with the most effective way to make data driven decisions and achieve personalisation.

Optimizely experimentation is a process that involves testing and optimising various aspects of digital products and user experiences. Experimentation allows organisations to make informed decisions by conducting controlled tests on different versions of webpages, app features, or other digital elements to determine which variations perform better based on specific metrics or objectives. Optimizely is the fastest experimentation platform in the world. Features of the platform include A/B testing, Multivariate testing (MVT), personalisation campaigns, feature flagging, and gradual rollouts.

What Are The Experimentation Types?

There are two types of experimentation products available from Optimizely, whether you want an easy to use client-side experiment or a more in depth full-stack analysis, you can access this through Optimizely experimentation.

Optimizely Web Experimentation:

Web Experimentation has revolutionised the way online organisations create their digital platforms and optimise personalised experiences. 

Optimizely Web Experimentation is a simple out of the box experimentation software solution used by marketing teams to do things such as optimising the user experience to drive Conversion Rate Optimisation (CRO). This is known as client-side experimentation as it focused on website visitor behaviour and improving conversions in marketing or purchase funnels. 

The beauty of the Web Experimentation package is that it’s very easy to operate, after practising a handful of times almost anyone can utilise its capabilities. No technical tools are required to run experiments or interpret the results, it has a WYSIWYG visual editor to amend webpages for building experiments and user friendly dashboards displaying results.

The experiments you build with Optimizely Web Experimentation usually run in one channel, such as your organisation’s website. Building an experiment is as simple as injecting a snippet into your website’s header, or alternatively you could use Google Tag Manager. If you want to check your experiment is running properly, you can use the ‘Optimizely Assistant’ plugin to view each variation of your page from a customer’s perspective. During the experiments you can monitor the results with live view to see the traffic and monitor conversion rates. 

Optimizely Feature Experimentation

The other type of experimentation product that can be purchased is Optimizely Feature Experimentation. This is a more complex experimentation software that relies on SDK’s and requires intricate development implementation. It’s an experimentation tool for rolling out new features for a site using feature flagging. This form of experimentation is less concerned with CRO and instead is designed to optimise the release of new functionality, providing the ability to revert back changes quickly, ensuring that end users have the most reliable digital experiences possible.  

Optimizely Feature experimentation can take longer to implement in comparison to Web Experimentation because it is important to adhere to code development standard practices, like code reviews and testing. However, the benefits include being able to experiment across multiple channels simultaneously such as web, mobile, SMS and email thus providing richer results. The experimentation tool can also be linked with the Optimizely Data Platform or other third party customer data platforms such as Segment and Tealium to help improve targeting and personalisation.

Ultimately, this server-side experimentation provides additional capabilities that work in tandem with the features of Web Experimentation, to enable targeting of new website features to specific audiences. 

Feature flagging is available to allow you to enable or disable a feature without changing the code. When experimenting, another useful element is the Stats Accelerator. Optimizely uses bespoke algorithms to reach statistical significance faster by allocating traffic to the variation that has demonstrated the largest impact in your target metrics. With the latest Feature Experimentation update, Optimizely has now made experiment scheduling available, as well as notifications to tell you if your experiments have hit statistical significance. All of these aspects combined means employees can spend less time logging in and out of the dashboard when launching and monitoring experiments which in turn improves productivity.

Optimizely testing facilities:

A/B Testing: 

A/B testing, also known as split testing, is a valuable technique used primarily in marketing to experiment with and compare multiple variations of a web page or application. Its main purpose is to determine which variant performs better based on predefined metrics. This process aids marketers in identifying the most effective content and design choices.

Key elements of A/B testing include:

• Random User Assignment: Users are randomly assigned to different variants, ensuring a representative and statistically valid sample of the target audience.

• Metric Collection: Metrics related to user engagement, conversions, or other predefined goals are collected and meticulously analysed for each variant to identify performance differences.

• Statistical Significance: The gathered data is rigorously evaluated for statistical significance to determine whether observed differences are meaningful or simply due to chance.

• Insight Generation: Based on the results, meaningful insights are derived to understand which variant is more effective. These insights then guide further optimisations to enhance the digital product or user experience.

The key benefit of using A/B testing is it allows organisations to make data-driven decisions to optimise the user experience, ultimately leading to a more effective and user-centric digital presence.

Included within your A/B testing is:

A/B/n Testing

A/B/n Testing is a method of website experimentation to evaluate website design to improve CRO. Multiple variations of a web page are pitted against each other to identify the one with the highest conversion rate. In this testing approach, traffic is randomly and equally allocated across the various versions of the page to assess their performance and determine the most effective variation.

A/B/n testing expands on the concept of A/B testing, where two versions of a page (A and B) are compared. However, in an A/B/n test, more than two versions of a page are simultaneously compared. The ‘n’ in A/B/n signifies the number of versions being evaluated, ranging from two to the ‘nth’ version. It is important to help marketers determine which website design is optimal rather than guessing based on opinion.

A/A Testing

A/A Testing is used to check your experiment results are valid. A/A testing employs the principles of A/B testing to compare two identical versions of a page with each other. The primary purpose of A/A testing is to verify the statistical fairness of the experimentation tool being utilised. In an A/A test, when properly executed, the tool should indicate no significant discrepancy in conversions between the control and variation, validating the accuracy of the testing process.

Multivariate Testing:

Optimizely Multivariate Testing (MVT) is an experimentation method that involves simultaneously testing multiple elements on a webpage or app screen to discover the best combination that maximises desired results. In contrast to A/B testing, which assesses one element at a time with a few variations, MVT involves modifying several elements, like headlines, images, layouts, and buttons, in different combinations to comprehend how they collectively influence user behaviour and engagement. This type of testing is unparalleled in bolstering digital strategies.

Key aspects of (MVT):

• Element Variation: Identifying multiple elements within a webpage or app screen for potential variation.

• Variant Combinations: Creating different combinations of these elements to represent diverse design options for testing.

• Randomised Distribution: Assigning users randomly to various combinations to comprehensively evaluate the performance of each combination.

• Data Collection and Analysis: Collecting data on user interactions and engagement metrics for each combination to analyse how these variations impact user behaviour.

• Statistical Significance: Applying statistical analysis to ascertain the significance of the variations and their influence on user interactions.

• Insights and Optimisation: Utilising insights derived from the analysis to guide optimisation strategies. 

Optimizely MVT provides organisations with the tools to optimise their digital products by understanding how different elements interact and influence user experiences. It offers a more comprehensive view of user behaviour and preferences, enabling informed decisions to improve the overall design and functionality of digital assets.

Multi-page funnel testing:

Multi-page testing is a variation of MVT and is often referred to as “funnel” testing. It shares similarities with A/B Testing, with the primary distinction being that instead of applying variations to a single page, modifications are consistently applied across multiple pages. Visitors participating in a multi-page test are categorised into one version. By monitoring the interactions of these visitors with the various pages they encounter, you can discern the most effective design style. A critical aspect of obtaining meaningful data in a multi-page test involves ensuring that users do not encounter a mix of variations but rather experience a consistent variation across a set of pages. This setup enables a thorough comparison of one variation against another.

Best Practice For Experimentation:

• Define Clear Objectives: Clearly articulate what you aim to achieve with each experiment, whether it’s increased conversions, higher engagement, or improved user satisfaction.

• Segment Your Audience: Understand your audience and segment them appropriately to tailor experiments that resonate with specific user groups.

• Prioritise Hypotheses: Develop hypotheses based on data, research, and user insights, and prioritise them based on potential impact and feasibility.

• Maintain Experimentation Discipline: Stick to a structured experimentation process, including pre-experiment planning, execution, analysis, and iteration based on results.

• Continuous Learning and Improvement: Always learn from your experiments, regardless of the outcome, and use those learnings to refine future experiments and strategies.

Benefits of Optimizely Experiments:

• Make data driven decisions. 

• Enhance user experience and retention rates.

• Significantly boost conversion rates. 

• Reduced risk (risk migration): Feature rollouts with gradual access help mitigate risks associated with deploying new features.

• Cost effective optimisation: Once you have purchased the software, you can conduct experiments on any website you have access to.

• Developer friendly: With over 9,000+ developers, the developer experience is well loved and well documented in comparison to competitors.

• Competitive edge: organisations using Optimizely stay ahead in the market by consistently optimising their products and services.

• Create specific targeting and personalisation to create unique experiences. 

• Access easy to interpret analytics and harmonised data.

• Easily integrate the experiment software with third party systems through API’s.

Is It Time For Your Organisation To Start Using Experimentation?

We are Ultimedia, the first Optimizely partner in the UK and the Middle East. With a rich history dating back to 1995, we have been collaborating with enterprise-level organisations. We have a team of experts including Optimizely Certified Experimentation Strategists who possess extensive experience in digital transformation and excel in implementing digital solutions through Optimizely. Contact us today to book an Optimizely demo and start your journey with Optimizely.