[book review] The Signal and the Noise: Why So Many Predictions Fail — but Some Don’t

Nate Silver’s The Signal and the Noise is a great entry to know how much information is out there and how much we don’t know about it. All the data in the world will not make us predict the future yet knowing how to analyze it is a something that I think all of us has to know.

I think we don’t value statistics enough and this book tells me how much I need to know more about it.

The Signal and the Noise

As much as I want to be a great fan of statistics, I was never really good in math but luckily the statistics subject wasn’t that hard during university. If you read The Black Swan by Nassim Nicholas Taleb then you would have an idea how much we under utilize statistics in our daily lives. This book will give you insight on how statistics is being used in different industries and how wrong we are in analyzing the data that we are given.

Nate Silver was already known in the baseball world before the US presidential elections last 2008, but the elections made him a household name after successful predicting the winner in 49 out of 50 states. If you follow just a bit of US politics like I do (via The Daily Show and Colbert Report 😛 ), then you would have definitely heard of him.

A former accountant, Silver created his predictive baseball algorithm on the side which eventually led him to better opportunities. The book discusses several industries where prediction and forecasting is their main role. Jobs like political pundits, weather forecasters, chess computer programmers, and even poker players rely a lot on statistics. This varied mix of industries got me really into the book and showed me how important statistics is.

The book was released in 2012 and a good start to any “statistics” book is to start with the 2008 economic crisis and how nobody saw it coming. Of course, he does give the typical hindsight is 20/20 routine but Silver does have a bunch on insights on why a lot of people saw it coming and how fiscal irresponsibility and ballooning property prices were good indications of the things to come.

A ironically funny chapter in the book is about political pundits and how they constantly get things wrong. Silver is in the pundit game and he knows what he is talking about especially about television pundits. It’s just hilarious for me especially since I get my “news” from comedy news show.

The book does have a good analogy on two types of thinkers: hedgehogs and foxes.

• Hedgehogs are personalities that believe in Big Ideas. People who won’t change their minds on predictions even though new evidence presents otherwise.
• Foxes on the other hand are people who believe in little ideas and use the data to formulate their predictions.

The book has better examples and explanations of this.

Moving away from politics, the book goes into weather forecasts and how this industry has improved over the years. If you are like me and you look at weather predictions all the time then I’m sorry to tell you that we’ve all been had. I used to live in tropical countries and man the weather forecasts are at best “a guide”. Unless it’s for a major storm, 20% of rain means nothing to me and I would always carry an umbrella. Having only two seasons, tropical countries have a hard time predicting the weather (or so I thought). When I moved to Japan, I was quite surprised on how “good” their predictions are and I would always follow their advice on what to wear and whether to bring an umbrella or not. Alas, a few months later, I still carry my umbrella all the time. Expecting a good weekend picnic is not an accurate science too. Apparently, weather forecasts are much much better now than they were 30 years ago which I totally believe. The advancement of weather forecasting technology would definitely have given us more data to work with. However, one key point I got from the book is the biases that are introduced into the forecasts as well as 5-day forecasts. The book explains this more, let’s just say that I only look at forecasts with confidence two days in advanced now.

Living in Tokyo, the topic about earthquake prediction definitely got me interested. If you’ve lived here for a few weeks, you’ll know that earthquakes is as common as the wind. Silver discusses how earthquake predictions are always wrong and why we always want to look for patterns in earthquake predictors. I really believe that the technology to predict earthquakes are still far off and until then, all we can do is be prepared. I felt a couple of moderate earthquakes here and to be honest, I won’t know how I’ll react once another big one comes. I am ready with my emergency pack and all, but looking at the statistics a big one will definitely hit sooner or later. The book has a good analysis of earthquake occurrences that he will later discuss but on the topic of terrorism.

Another chapter that got me piqued was the chapter about climate change. Although I still don’t understand why there are still people who don’t believe that climate change is a real thing is beyond me, he does have some information on why. Comparing date from the past, sample sized, and different predictive algorithms has a big part on this. Although just looking at what is happening with our weather patterns should make you think that climate change is real. Larger storms, longer winters, hotter summers, and random freak fests should give us an indication that something is wrong with our nature and we are contributing to this destruction should be freaking obvious. The recent storm in the Philippines, Haiyan, is another single point example of this.

I really enjoyed this book and if you are like me who likes to know how much I don’t know, then you would appreciate this book at lot. Looking at different industries and professions in a different angle will just make you appreciate them more and all the thought that goes into their jobs that we take for granted.