About A/B testing
When you are making changes to your experience (for example, a website or app), you can use A/B testing to understand two main things:
- How much of an effect did my change have?
- How do I know whether I just got lucky (or unlucky)?
For example, consider web traffic. Websites often draw in visitors to sell a service or introduce an important idea. Small changes in messaging and component placement can have a material impact in guiding a visitor to take some action (for example, complete a purchase).
We can run an experiment on just a landing page: the control is the original website, and one or two variants are experiments. A visitor to the website is directed to either the original or one of the variants. The visitor then becomes part of the population of people experiencing that landing page.
We can measure the performance of the landing pages by looking at their conversion rate. That is, the percentage of visitors who opt to go to the next part of the website, or who complete a purchase.
Before starting, we want to estimate how long we need to run the experiment to be sure of the results. When we get results, we want to know not only which page performed the best, but also certainty around the repeatability of those results.
How do I know whether my change has a measurable effect?
With A/B view, you can:
- Create multiple experiments to compare to an original control.
- Define multiple performance metrics (called measures here) to compare absolute and relative population performance.
If you have the data infrastructure to define your populations before your data is ingested into a table, then it might be as easy as specifying a categorical experiment (that is, a column) property and the particular label. For example:
You can use A/B view to:
- Create multiple experiment populations to compare to a control.
- Define multiple measures to compare absolute and relative population performance.
- Interpret results in alternate all-in-one or single experiment comparison tabs.
- Understand experiment validity using p-values or rely on cell shading. Green or red cell shading indicate more or less significant results, according to the experiment's p-value.
Use Interana A/B view
A/B view is disabled by default. To enable A/B view, contact your technical account manager.
Access A/B view in the left menu bar:
When you access A/B view, it automatically fills in fields to create a valid (if of dubious interest) experiment based on actors in your data set. Use the dropdowns to create a more interesting experiment.
Ways to extend A/B view
An experiment population is not limited to people. For example:
- The population can be actors such as devices or other entities that experience events.
- Events can be categorized into populations and measured on different performance criteria.
- Flows are similar to actors, in that they also are described by a collection of events.