# Talking Bayes to Business AB Testing Use Case Shopify --- [[AB Testing|A/B Testing]] Agenda - Motivation - Getting the right answers with Bayes: concepts and toolkits - Beyond AB testing - Problem Forward vs. Solution Backwards A PM talks about impact, tracking & KPIs before planning the future. In the real world - We have a model of population and causality (eg better feature -> more usage) - We have well defined KPIs (clicks, sales) and understanding of effect size - Sufficient volume for significance and power - Sufficient velocity for timely answer - Good randomisation and user tracking infra for AB tests - Cross domain tracking is sometimes not allowed - Ad blockers What do we want to know? - Is the new feature better than the old one? (or better than not doing anythign) - The Intra-Ocular Trauma Test - I would need to see the results with my eyes only. If I need statistics to see if something is working, then something has gone wrong somewhere P-VALUE quantifies surprise in a universe when there is no change ![[bayes_pvalues.png]] - Replication crisis >[!note] Prior means you have an opinion >*... the probability distribution that would express one's beliefs (yes, it's subjective) about this quantity before some evidence is taken into account.* [Talking Bayes to Business: A/B Testing Use Case | Shopify](https://www.youtube.com/@DataCouncil) - Translating business insights into distributions