The Signal and the Noise
Tuesday, March 25th, 2014 | Books
Nate Silver is the man who correctly predicted 51 of the 52 states in the 2008 US Presidential Election, and then all 52 in the 2012 election.
With an increasing number of people recommending I read his book “The Signal and the Nose”, I decided to give it a read. It looks at why we, as a society, are pretty bad at making predictions. Why did nobody see the 2008 financial crisis coming? Why is our best guess at when the next earthquake will hit no better than random chance? Why can’t we even predict if it will rain or not?
Actually, the last one, we can. Weather forecasts have become far more accurate over the last few decades. However, they are one of the few fields in which the large scale application of data and computing power to process that data has truly been effective.
Silver claims that one of the biggest problems is that as we now live in the “information age”, there is simply too much data to work out what is actually a useful predictor (the signal) and what is merely correlated (the noise). A great example of this is that the Super Bowl winner (AFC or NFC) was an accurate predictor of how the economy would do. But obviously that is just random chance and has proved erroneous in the past few years.
Ultimately the book has a simple message – you need to use a Bayesian model and apply regression. None of this is a new concept to me, nor indeed you would hope anyone working in the field of statistics. But judging by some of the meetings I have had recently, it is shocking the amount of people that do not follow this advice.
Nate Silver is the man who correctly predicted 51 of the 52 states in the 2008 US Presidential Election, and then all 52 in the 2012 election.
With an increasing number of people recommending I read his book “The Signal and the Nose”, I decided to give it a read. It looks at why we, as a society, are pretty bad at making predictions. Why did nobody see the 2008 financial crisis coming? Why is our best guess at when the next earthquake will hit no better than random chance? Why can’t we even predict if it will rain or not?
Actually, the last one, we can. Weather forecasts have become far more accurate over the last few decades. However, they are one of the few fields in which the large scale application of data and computing power to process that data has truly been effective.
Silver claims that one of the biggest problems is that as we now live in the “information age”, there is simply too much data to work out what is actually a useful predictor (the signal) and what is merely correlated (the noise). A great example of this is that the Super Bowl winner (AFC or NFC) was an accurate predictor of how the economy would do. But obviously that is just random chance and has proved erroneous in the past few years.
Ultimately the book has a simple message – you need to use a Bayesian model and apply regression. None of this is a new concept to me, nor indeed you would hope anyone working in the field of statistics. But judging by some of the meetings I have had recently, it is shocking the amount of people that do not follow this advice.