Preface

      The path that led us to write this book began in 1999. We wanted to build an investment portfolio that would perform well across a wide range of market environments. We quickly came to the view that we needed more reliable estimates of volatilities and correlations—the inputs that determine portfolio risk—than the estimates given by the conventional method of extrapolating historical values. Our thought back then was to measure these statistics from a subset of the most unusual periods in history. We reasoned that unusual observations were likely to be associated with material events and would therefore be more informative than common observations, which probably reflected useless noise. We had not yet heard of the Mahalanobis distance, nor were we aware of Claude Shannon's information theory. Nonetheless, as we worked on our task, we derived the same formula Mahalanobis originated to analyze human skulls in India more than 60 years earlier.

As we extended our research to a broader set of problems, we developed a deep appreciation of the versatility of the Mahalanobis distance. In a single number, his distance measure tells us how dissimilar two items are from each other, accounting not only for the size and alignment of their many features, but also the typical variation and covariation of those features across a broader sample. We applied the method first to compare periods in time, each characterized by its economic circumstances or the returns of financial assets, and this led to other uses. We were impressed by the method's potential to tackle familiar problems in new ways, often leading to new paths of understanding. This eventually led to our own discovery that the prediction from a linear regression equation can be equivalently expressed as a weighted average of the values of past outcomes, in which the weights are the sum of two Mahalanobis distances: one that measures unusualness and the other similarity. Although we understood intuitively why unusual observations are more informative than common ones, it was not until we connected our research to information theory that we fully appreciated the nuances of the inverse relationship of information and probability.

      Our focus on observations led us to the insight that we can just as well analyze data samples as collections of pairs rather than distributions of observations around their average. This insight enabled us to view variance, correlation, and R-squared through a new lens, which shed light on statistical notions that are commonly accepted but not so well understood. It clarified, for example, why we must divide by N – 1 instead of N to compute a sample variance. It gave us more insight into the bias of R-squared and suggested a new way to address this bias. And it showed why we square distances in so many statistical calculations. (It is not merely because unsquared deviations from the mean sum to zero.)

      But our purpose goes beyond illuminating vague notions of statistics, although we hope that we do this to some extent. Our larger mission is to enable researchers to deploy data more effectively in their prediction models. It is this quest that led us down a different path from the one selected by the founders of classical statistics. Their purpose was to understand the movement of heavenly bodies or games of chance, which obey relatively simple laws of nature. Today's most pressing challenges deal with esoteric social phenomena, which obey a different and more complex set of rules.

      The emergent approach for dealing with this complexity is the field of machine learning, but more powerful algorithms introduce complexities of their own. By reorienting data-driven prediction to focus on observation, we offer a more transparent and intuitive approach to complexity. We propose a simple framework for identifying asymmetries in data and weighting the data accordingly. In some cases, traditional linear regression analysis gives sufficient guidance about the future. In other cases, only sophisticated machine learning algorithms offer any hope of dealing with a system's complexity. However, in many instances the methods described in this book offer the ideal blend of transparency and sophistication for deploying data to guide us into the future.

We should acknowledge upfront that our approach to statistics and prediction is unconventional. Though we are versed, to some degree, in classical statistics and have a deep appreciation for the insights gifted to us by a long line of scholars, we have found it instructive and pragmatic to reconsider the principles of statistics from a fresh perspective—one that is motivated by the challenge we face as financial researchers and by our quest for intuition. But mostly we are motivated by a stubborn refusal to stop asking the question: Why?

      Practitioners have difficult problems to solve and often too little time. Those on the front lines may struggle to absorb everything that technical training has to offer. And there are bound to be many useful ideas, often published in academic articles and books, that are widely available yet seldom used, perhaps because they are new, complex, or just hard to find.

      Most of the ideas we present in this book are new to us, meaning that we have never encountered them in school courses or publications. Nor are we aware of their application in practice, even though investors clearly thrive on the quality of their predictions. But we are not so much concerned with precedence as we are with gaining and sharing a better understanding of the process of data-driven prediction. We would, therefore, be pleased to learn of others who have already come to the insights we present in this book, especially if they have advanced them further than we do in this book.