Practical and Modern A/B Test Methods

While the concept of A/B Test is simple and intuitive deployment is much trickier. After having experience with deployment of A/B Test systems we built a single day (The 2nd day is optional) course in order to give the engineers and data scientists the required knowledge to develop and deploy a successful A/B Test system.

The course deals with practical and modern A/B Test methods. It focuses on the Bayesian approach which is the modern way of handling the A/B Test since it allows answering business questions naturally. The course is accompanied with Jupyter Notebooks with real world data examples.

• Practical and Modern A/B Test Methods (PDF)

Overview

The course:
${\color{lime}\surd}$ Introduces the problem and its probabilistic formulations.
${\color{lime}\surd}$ Explores different concepts of solving the problem.
${\color{lime}\surd}$ Demonstrates results on real world data using battlefield learned tricks with Jupyter Notebooks and hands on approach.

Main Topics

A/B Test Models	Frequentist Approach, Bayesian Approach
Key Performance Indicators (KPI)	Binary (Clicks, Sales, etc…) vs. Continuous (Revenue)
Answering Business Questions	Defining and simulating results, Confidence level
Modern A/B Test System Architecture	High Level Blocks, Low level implementation considerations, guide lines
Advanced Bayesian Models (Optional)	Hierarchical Bayesian Methods with MCMC, Advanced models for A/B Test
Reactive A/B Test (Optional)	Fixel’s own built battle proven algorithm to A/B Test to maximize revenue

Goals

• The participants will know fundamentals concepts in A/B Test.
• The participants will be aware of the different assumptions for different models for A/B Test.
• The participants will be able to choose the proper model for a given A/B Test case.
• The participants will be able to implement a deployment oriented A/B Test algorithm.

Pre Built Syllabus

Day	Subject	Details
1	A/B Test Models	Frequentist Approach, Bayesian Approach, Distributions, Priors
	Key Performance Indicators (KPI)	Views, Clicks, Sales, Revenue, Binary vs. Continuous
	A/B Test by Frequentist Approach	Assumptions, T-Test (Continuous and Binary), Estimating number samples required, Stopping criteria
	Notebook 001	Frequentist Approach for Binary KPI, Manual implementation of T-Test
	Notebook 002	Frequentist Approach for Continuous KPI, Manual implementation of T-Test
	A/B Test by Frequentist Approach	Assumptions, Priors, Models (Conjugate Priors), Answering business oriented questions
	Notebook 003	Bayesian Approach for Binary KPI, Implementation of the Bayesian Posterior
	Notebook 004	Bayesian Approach for Continuous KPI, Implementation of the Bayesian Posterior
	Real World Practice and Considerations	Frequentist vs. Bayesian, Change detection, Prior learning, Update rate, Granularity of data
	Notebook 005	Prior learning, update rate and change detection
	Notebook 006	Answering business questions by simulations
	Modern A/B Test architecture	High level architecture, Building blocks, Visualization and reports
2	Introduction to MCMC	Concept, Frameworks, Using MCMC for A/B Test
	Notebook 007	Simulation and inference of Bayesian A/B Test model with MCMC
	Introduction to Hierarchical Bayesian Model	Concept and visualization, Example of models for real world data
	Notebook 008	Implementation of models for real world data
	Hierarchical Bayesian Model for A/B Test	Binary Model, Continuous model, Comparison with Non Hierarchical Bayesian Models
	Notebook 009	Implementation of Binary and Continuous models, comparison to non hierarchical models

• Practical and Modern A/B Test Methods (PDF)

Prerequisites

• Linear algebra (Basic).
• Probability / Statistics (Basic).
• Python / MATLAB / Julia.