Nate Silver’s “Signal and the Noise”

signalb

Nate Silver is famous for the accuracy of his predictions of US election results. His book The Signal and the Noise addresses prediction methods across a very wide front to develop his core arguments: that forecasts should be probalistic (i.e. forecasting that there was a 91% probability of Obama winning last year’s election rather than just predicting he would win); that Bayesian forecasting (i.e. reassessing prior probabilities  on the basis of new evidence) is more accurate than frequentist approaches; and that in many cases the risk of rare but significant events (e.g. earthquales and terrorist attacks) can be modelled using a power law (i.e. log of frequency of earthquakes above a certain strength is  inversely correlated with earthquake strength). To get these points across, Silver peregrinates on a journey across the  forecasting of elections, baseball, terrorism, epidemics, share prices, earthquakes, weather, poker, climate change and chess. This is a long expedition through very varied terrain, and describing it takes Silver 454 pages; this book is not called “A Short Introduction to Prediction” for a good reason. And as on a round-the-world cruise, some of the points of call turn out to to be more interesting than others. Throughout Silver writes clearly and all of the chapters are very well supported by citations, but Silver is most insightful and persuasive in the domains in which he is most experienced (baseball, poker and US elections) and less persuasive in areas where his experience is weaker and the issues around forecasting are very complex (financial markets and climate change). The chapter on baseball assumes a knowledge of the game that is way beyond me, and I would guess most European readers, but perversely Silver explains the rules of Texas Hold’em in enough detail for a complete beginner to put down the book, head to the poker tables and lose (Silver includes an oblique reference to Joy Division, so he is not afraid of looking obscure).

The book includes a short but clear description of Bayes Theory, but as a book on a quantitative subject, this is a book almost devoid of detailed explanations of quantitative methodologies, but I have been around long enough to realise that the numerophobic are not an excluded minority. More specifically, it would have been useful to explain in detail the advantages of Bayesian forecasting over alternative methodologies, most obviously game theory and maximum liklihood estimation.

So why is this book relevant to operations management? It covers a lot of ground, but one field not crossed is the management of operational risk. The management of operational risk is often based around the identification of scenarios and then the calculation of the scale of the risk and the probablity of occurrence. So the analysis will identify that the risk of bad thing X happening is Y% in the next Z years: so far so inline with Silver’s approach. But Silver’s argument is that this analysis should take account of prior beliefs and should take account of new information, and this is not always the case. The attraction of building complex models built on past data runs the risk of overfitting, where the model is following the random noise in the past data leading to over-belief in its accuracy, with the importance of new data overlooked. Silver makes the claim for thinking about the power-law relationship between liklihood and severity, persuasively arguing that rare but spectacular failures are either ignored or assigned too low a probability because no one has experienced them before (an availability bias). This argument is obviously close to Taleb’s concept of Black Swan events, but the breadth of Silver’s examples and his relentless linking of it to Bayesian statistics will drive this point into the consciousness of any reader who is also involved in managing operational risk, which nowadays is just about every operations manager. Negative Amazon reviews are often from specialists directly involved in the management of risks critcising the lack of equations and models, but in most operations scenrio analysis the process of evaluating risks and assessing new information is not a quantitative exercise undertaken in isolation, it is an exercise involving a few quantitative specialists and many others less quantitatively skilled but with domain specific knowledge that should be incorporated into the forecasts. For the mathematical mavens this book will not tell them too much they do not know already, but for the larger number of managers who have to think about risk this book will communicate several very important lessons.

Leave a Reply

Your email address will not be published. Required fields are marked *