Business Experiments with R. B. D. McCullough
Читать онлайн книгу.
of firefighters is highly correlated with the damage caused by the fire. Adding more firefighters doesn't increase the amount of damage (the variables are really unconnected). Rather, the lurking variable “intensity of the fire” connects them. A lurking variable (say,
• Variables are confounded when we cannot separate their respective effects on the response. A confounding variable
Confounding can occur in a poorly designed experiment. Suppose you wish to determine the effects of fees and interest rates on credit card use. Suppose you offer a low fee and low interest rate to one group and a high fee and a high interest rate to the other group. The first group will have more credit card use and the second group less use, but you won't be able to tell whether the low fee or the low rate caused more use in the first group or whether the high fee and the high interest rate caused less use in the second group. The rate and the fee are confounded.
On the other hand, we could isolate the effect of rate by offering low fee and high rate to the first group and low fee and low rate to the second group. In more advanced designs, we sometimes will have many effects and be unable to isolate them all. In such a situation, we will deliberately confound the effects that we don't care about so much so that we can isolate the effects that we do care about. We will address this in Chapter 8.
Section 1.3 “Case: Salk Polio Vaccine”
• A layman's overview of the Salk trials is given in Meier (1989).
• The source for the polio data is http://www.post-polio.org/ir-usa.html. The source for the “under 18” US population is https://www.census.gov/data/tables/time-series/demo/popest/pre-1980-national.html. The source for the US population data is US Current Population Reports Series P25.
Section 1.4 “What Is a Business Experiment?”
• The financial services example is based on Watson‐Hemphill and Kastle (2012).
• The Progressive Insurance example comes from Chapter 22 of Holland and Cochran (2005).
• The Anheuser‐Busch example comes from Ackoff (1978).
• The number of conditions necessary to establish causality varies from discipline to discipline and even author to author. For example, the epidemiologist Hill (1965) gave nine rules. We stick with just three.
• While observational data can be useful, they are no substitute for an experiment (if an experiment can be conducted!):
But even if done perfectly, an observational study can only approach, but never reach, the credibility of randomization in assuring that there is no missing third variable that accounts for the differences observed in the experimental outcome. (Wainer, 2016, p. 48)
• The book by Schrage (2014) is entertaining and describes numerous business experiments; most of these are small, inexpensive experiments. The book by Holland and Cochran (2005), which is written for laymen, describes several larger, more complicated experiments. The article by Ganguly and Euchner (2018) describes the approach of the Goodyear Tire Company to experimentation and gives many interesting examples, including one where they conducted an experiment to determine whether tire‐pressure monitoring equipment could generate more than enough savings to pay for the monitoring equipment by reducing roadside breakdowns of tractor‐trailers. It is worthwhile to read materials such as these, for it is important for the novice experimenter to develop an idea of what has been done and what is possible.
• The idea that small sample sizes, small effect sizes, and lots of noise can lead to false positives and even sign reversals (truly positive coefficients being estimated as negative) is discussed in terms about as nontechnical as possible in Gelman and Carlin (2014), but you'll have to know what “power” is to follow the argument, so maybe you should wait until after Chapter 2 to read it.
Section 1.5 “Improving Website Design”
All the web tests, in particular the results in Table 1.6, are real. Due to difficulties obtaining permissions for the original web ads, some of the web ads were mocked‐up to simulate the real ads. The photograph in Figure 1.5 is from pixabay.com, and the photograph in Figure 1.7 is from pexels.com.
GuessTheTest.com is a resource for digital marketers who want objective A/B test case studies and helpful information to get split‐testing ideas, insights, and best practices. There are many aspects of A/B testing on the web that are not covered in this book, and the interested reader may profitably spend some time at this website. Also, if you think you're any good at predicting the outcome of an A/B web test, to disabuse yourself of such an errant notion, try guessing at a dozen or so of the many cases presented at this website and see if you can beat 50% accuracy by a statistically significant amount.
• We barely scratched the surface of A/B testing, which, according to two recent surveys, is the most important topic in business: a survey of online marketers found “Conversion Rate Optimization” to be a top priority for the foreseeable future (SalesForce.com, 2014); a survey of businesses that engage in conversion rate optimization used A/B testing more than any other method (Econsultancy, 2015).
• An entertaining layman's article on the rise of A/B testing can be found in Wired magazine (Christian, 2012). On A/B testing and the Obama presidential campaigns, see the interesting article in Bloomberg Businessweek by Joshua Green (2012). This is of historical interest because the Obama campaign was the first to really use analytics for fundraising and get‐out‐the‐vote activities. For those who want to learn more about the technology behind website testing and the types of tests that are possible on websites, we recommend the chapter on web testing in Waisberg and Kashuk's book titled Web Analytics 2.0 (Waisberg and Kaushik, 2009) or the succinct book by McFarland (2012) with the catchy title Experiment!.