Use Interana’s A/B Test view to understand the results of your A/B tests. For example, you can examine the results of tests for new layouts, user flows, email subjects, recommendation algorithms, colors, rankings, or new features. Then use filters to drill down into your data and identify the statistically significant results.
The A/B Test view is not enabled by default. Contact your Interana representative for more information.
The A/B test workflow
The basic A/B Test analysis workflow is:
- Identify the groups that you want to compare. These can be identified by a flag in the data, by filtering to specific sub-populations within the data, or by building cohorts or filters from defined start and stop times of your test.
- Select a list of Measures you care about.
- Specify Filters that define your A and B groups.
Interana will show you the selected measures of both groups, then the difference and percentage difference between the groups for each measure. Note that the first group you define (group A by default) is the control group; all measurements will be in relation to the first group.
If you select the Average measure or create a custom ratio measure, Interana will also display the confidence interval, both as a numeric value and in the graph view.
Preparing your data
We recommend using tags to identify your data in your logging for your A/B testing.
For example, if you have a field named "experiments,” for each event (row) that's part of an experiment, add a string with the name of the experiment group to that tagset field. In JSON, this would be:
Use consistent naming conventions to differentiate between experiments and groups. For example, "myexperiment_control", "myexperiment_group_A", etc. Then you can filter to `"experiment" "is in tagset" "my_experiment_group_A"`, and use the filter compare feature in the Interana Explorer or A/B view to compare the groups.
Interana does not currently support other methods of logging or demarcating experiments (for example, events that signify the start of an experiment for a user), and we recommend using the experiment tag for every event that is part of the experiment.
If you want to use a lookup table to assign users to an experiment group, be aware that this does not retain start and stop times for the experiment. If using a lookup table, make sure you track the start and end times when you perform the query.
Configure the A/B test analysis
See the following sections for more information.
Select one or more Measures
Start by selecting one or more measures, then the event data you want to analyze.
This example uses two measures: the Average of the itemInSession data and a custom measure of the ratio of males in the data.
Select your Compare groups
Select the groups that you want to compare. Note that the first group you define (group A by default) is the control group; all measurements will be in relation to the initial group.
Click next to the default name of the group to rename the field. This name will appear in the graph views.
Use the p-value and confidence intervals to analyze your data
The p-value represents the statistical confidence of the results. This tells you how likely the observed difference between the groups is due to chance. The initial hypothesis, or assumption, is that the two groups are not different. The lower the p-value, the more “real” or statistically significant the difference.
Interana allows you to set the statistical significance cutoff for your test. Use the following p-value settings to configure the data displays:
- 0.05: This is the standard measure of statistical significance, indicating that there is a 95% probability that the two groups are distinct in the selected measure. This is the default value.
- 0.01: This setting shows data with a higher statistical certainty, indicating a 99% probability that the two groups are distinct in the selected measure.
- 0.001: This setting shows data with an even greater level of statistical certainty.
This setting sets the significance cutoff for all applicable measures in the view.
Interana displays confidence intervals only when using the average measurement or certain custom ratio measurements. The confidence interval represents the range of values that our measure likely lies within, to the probability defined by our p-value (the p-val setting controls the width of the bands).
For example, with p-value setting of 0.05, we can be 95% confident that the actual difference between groups A and B for average itemInSession is 1.69 +/- 0.11, or between 1.58 and 1.80.
Use the confidence intervals to better understand when the difference between groups is statistically significant. The intervals show the distribution of values over all events.
View the analysis results
Interana displays the analysis results in rows on the right side of the window.
Group data cells
When you hover over a group Value cell, Interana displays statistical information including the degrees of freedom and the standard deviation or standard error, depending on the measure that you are using.
Difference and percentage difference cells
When you hover over a cell that shows the difference or percentage difference between the groups, Interana shows you the standard error (both when assuming that groups A and B are equivalent and for B-A), the z-statistic, and the p-value. This is the information Interana uses when evaluating the significance of the data.
Interana applies shading to these cells to indicate values that are statistically significant based on the p-value setting (p-val).
In this example, the Average itemInSession measure shows a statistically significant difference between the control and test groups. Interana shades the cells to indicate this. The Ratio of males measure does not show a statistically significant difference (and is unshaded) because the B-A difference is not significant, as determined by the p-value.
When you hover over a shaded cell that displays the difference or percentage difference between the groups, Interana displays statistical information about the cell, including degrees of freedom, the t-statistic, and the approximate p-value of the difference. This information helps you evaluate the significance of the data.
The analysis graph
Click any row to open the graph for that analysis. Use the graph to view the events over time, and explore the analysis by adjusting the graph parameters.
You can change the graph display by selecting the y-axis represents option:
- Value: The graph shows the value of the data points over time.
- Change: The graph shows the change between the groups (the B-A values) over time.
- % Change: The graph shows the percentage change between the compare groups over time.
You can also set the y-min and y-max values to scale the y-axis of the graph. This is useful to zoom into the graph or to sections of the graph.
You can always toggle the second (or B) compare group graph. Interana does not display a graph of the initial group (the A group) because it is implicitly 0 for most comparisons. However, you can toggle the A group when you using the Value display setting. This allows you to compare how the values of the groups change over time.
When you hover over a data point in the graph, Interana displays a popup with more information about that data point.
- Starting and ending times: The time window for this event data.
- p-value: The current p-val setting.
- Value and confidence range: The value of this data point, and the confidence band for this value.
- Difference between the compare groups: The difference between the two groups at this data point and the percent difference.
- Degrees of freedom: Typically, the number of data points in the set.
- Standard deviation or Standard error: Interana displays one of these statistical values, depending on the measure you are using.
Set a filter (Optional)
You can use filters to drill down into your data and identify statistically significant results. The filters you select will apply to all groups in the A/B test.
Select a filter, a condition, and a value (or a range of values), then click Go to apply the filter.