Business Experiments with R. B. D. McCullough
Читать онлайн книгу.
by Harrison and List (2004) and Gerber and Green (2012), yet the experiments reported tend to focus on testing economic theory. In many ways, the types of tactical experiments that are valuable in business are much more similar to clinical trials and other biomedical experiments, where there is a tradition of tactical experiments designed to determine, “Should we give this treatment to patients?” and a well‐established culture of randomized controlled trials as the gold standard of evidence. While much of this work from other disciplines is relevant to business experiments, it doesn't provide students with concrete examples of how to apply these methods to answer important business questions.
This book has drawn from all of these literatures selecting the key ideas that business students need to know about experiments. This book approaches experimentation as a skill that one becomes better at over time, like playing a sport or an instrument, which requires equal measures of creativity and technical skill. To help students practice, the book is structured around a series of examples, focusing first on a specific business question, then turning to the design or analysis of an experiment that then will answer that question, and then building up the appropriate statistical methods. This way, the technical material is presented in context. This approach provides an ideal “second course in statistics,” where students can review material they have seen before (specifically two‐sample tests, power, ANOVA, confidence intervals, and regression) in the context of business problems. My hope is that students who work through this book will move beyond the mechanics and develop their abilities to frame ill‐defined problems, determine what data and analysis would provide information about that problem, and examine the evidence for or against a particular business decision.
For clarity in the use of pronouns, I have adopted the convention that the analyst is always female. Other persons are male.
Suggested Courses Using This Book
This book can be used both with general business students, such as MBA or executive education students, and with the growing number of master's programs in business analytics and marketing analytics.
The first four or five chapters (depending on instructor preference) provides students with the tools they need to design and analyze two‐treatment experiments (i.e. A/B tests) to answer business questions, focusing on the strategic and technical issues involved in designing experiments that will truly affect organizations. The book begins with a discussion of the importance of causal analysis in business and the risks of observational data analysis in Chapter 1. By design, the next chapter begins with the analysis of A/B tests, with a review of the statistical methods required for analyzing experiments, focusing on visualization and hypothesis testing. Once students understand how experiments are analyzed, in Chapter 3, they can move on to understanding issues involved in designing experiments including selecting treatments, measuring responses, planning sample sizes, and the importance of randomization for causal interpretation. Chapter 4 returns to the analysis of A/B tests, covering several advanced analysis techniques including analyzing tests with matched pairs, analyzing tests with more than two treatment groups, analyzing subgroups within an A/B test, and determining the minimal detectable effect for an experiment that has been completed. Chapter 5, which covers issues involved in the design of A/B tests when the number of units available for tests is small, as is the case when doing geo‐testing or testing across store locations. This coverage makes Part I ideal for a half‐semester graduate course for second‐year MBA students or strong undergraduate students. These chapters can be covered over five weeks, skipping some of the more technical material and allowing time for the students to apply the material to a project where they plan, execute, and analyze an A/B test.
Part II builds on the foundation in Part I, expanding to multivariable testing. Chapter 6 begins Part II with a review of the primary tool used to analyze multivariate tests: regression analysis. Next comes full and fractional factorial designs in Chapters 7 and 8, which builds students' understanding of the core issues in multivariate design: interactions and confounding. Fractional factorial designs are useful for building understanding. Next comes custom design, which is ideal for designing experiments in constrained business contexts. The development of software for optimal design has made this sophisticated tool accessible to users, and my goal for Chapter 9 is to provide students with a basic introduction to optimal design, while avoiding the algorithmic details that would be more appropriate to a “technical manual.” Parts I and II together form the core of a full‐semester course that is appropriate for more analytically oriented graduate students including MBA students concentrating in analytics and master's of business analytics students.
Not all experiments are technically complicated. There is a great deal of “low‐hanging fruit” in the business world that can be profitably plucked by persons whose knowledge of experimental design does not exceed 2k designs. The goal of this book is to enable students who have taken business statistics to pluck that fruit. In particular, after finishing the first half of this book, the student should be able to conduct an A/B test. After finishing the second half, the student should be able to design, execute, and analyze a 2k experiment. To this end, instructors should require students to complete a project where they design, execute, analyze, and report on an experiment. Students benefit greatly from the experience, which helps them see the challenges involved in designing a conclusive experiment and cement the ideas in the book by applying them in practice. A/B tests are easy to dream up, multivariable tests not so much. Hence, Box's helicopter experiment is very important. It is perhaps the only easily executed multivariable experiment available to students that is amenable to screening designs, full factorial designs, and even custom designs. By the end of this book, the reader should be able to execute Box's helicopter experiment Box (1992a) even if the professor does not make it part of the class.
Philadelphia, PA
B. D. McCullough
Acknowledgments
Without constant support and encouragement over the past three years from my esteemed colleague, Dr. Elea Feit, this book would never have been started, let alone finished. She allowed me to sit in on the business experiments course that she developed at Wharton and brought to Drexel. Despite the publish‐or‐perish demands of an assistant professor's life, she took the time to read and comment on some chapters, suggested many books and articles for me to read, and always had time for me when I ran into difficulties. Her influence can be found in every chapter, but any errors or omissions are mine and mine alone.
I thank Russell Lavery and three sections of STAT 335 students for reading some of the early chapters, and I thank Malcolm Hazel for reading all the chapters.
B. D. McCullough
Bruce McCullough
Bruce McCullough passed away in Fall 2020, just before this book was published. Bruce was an extraordinary scholar and friend, who will be greatly missed. He was fair, direct, true‐to‐his‐word, and expected the highest level of rigor and integrity from himself and those around him. He was also unusually protective of those he took into his care including junior colleagues like me, as well as his wife and children. I'm sure it troubles him greatly that he can't be here in‐person to see us all through.
Based on many discussions we had about the book, I know he felt just as protective towards his readers. He devoted the last few years of his professional life to providing you with a book that would open up the potential of experimentation in business and prevent you from making