Reactive A/B Testing - A New Approach to A/B Testing
Background
During the last 2 years we have been engaged in many advising work for A/B Test for various Internet companies.
We have also tutored our first A/B Testing Course.
We have built few analysis engines both using Bayesian A/B Testing approach and Frequentist A/B Testing approach.
When calibrated well, both methods could be satisfactory, yet both have pitfalls of a static test.
Hence, internally, we have developed a concept called Reactive A/B Testing.
The idea is to have a continuous reactive A/B Test which maximizes revenue.
You’re always in the test, no need to wait and then optimize.
What Is A/B Test
A/B Test is basically a decision making test.
It is more than 100 years method which was initially used in Agriculture.
A/B Test in Agriculture
Testing 2 treatment variants and a baseline
The methods is used for comparison of 2 variants and measure the difference effect on the desired result.
A/B Test in Medicine
Testing if a biological treatment works is done using A/B Test
In our days it is the Go To method to measure the effect of a web site design on the users engagement.
Though the name A/B Test implies 2 variants, the approach can be easily extended to many variants.
There are many approaches to solve it:
- Binary Classification
One may build a classifier to classify the winning strategy. - Frequentists Approach
Estimating parameters of a distribution and measuring the probability of being wrong given a baseline assumption. - Bayesian Approach
Computing the posterior of the distribution of the data and calculating the credible intervals.
Internally, out default choice is the Bayesian A/B Test unless we are asked to get the real
The Pitfalls of Current A/B Tests
Whether the classic A/B Test is used or more modern methods, the issue with them is being static.
There 3 main stages to the test:
- Build and deploy variants.
- Split traffic and collect data.
- Analyze results.
Once the results are conclusive (Try not to peak before time!) we deploy the winning variant and the revenues are optimized.
The issue is the period during the test and after it:
- During the test by definition the revenue is not optimized.
- After the test, results are valid if the data we collected is stationary, namely there is no shift in the observed population.
The effect of both can make the final outcome “not worth it”.
This is what we have fixes with the new concept of Reactive A/B Testing.
Reactive A/B Text Concept
How can it be done? As pedagogically one must wait for the results to be credible in order to make decisions.
Without going into our concept, the idea is to use methods form Game Theory which are used in the context of A/B Testing.
It is a different approach yet we have successfully used it in many cases and polished it.
It requires some adaptations to how data is collected and the algorithm used. Yet the effect, when examined on real world data was staggering.
Revenue during the test is not lost, the output is adaptive hence even with population shift the revenue is optimized.
A/B Test Course
We packed all our knowledge in A/B Testing into A/B Test Course.
This course covers all theoretical and practical knowledge with many know how secrets.
It includes some optional subjects such as advanced simulations, data generation and Reactive A/B Test.
If you want to know the secrets of this method, getting the course for the algorithms teams is the way to go.
We also do knowledge transfer and code transfer for Reactive A/B Testing.