Posts Tagged ‘forecasting’

The Black Swan

Friday, August 15th, 2014 | Books

A number of books on probability I have read recently talk about Nassim Taleb’s book, The Black Swan: The Impact of the Highly Improbable.

Sometimes I wonder if some of the citations and due to the Black Swan meme. It is a great term to use for unexpected events. Capturing the phrase makes it more citable. However, that is not to take anything away from the book. Many have called it one of the most influential books of the past 50 years. The Nobel laureate Daniel Kahneman is quoted as saying

The Black Swan changed my view of how the world works

Interesting Taleb goes on a small rant about the Nobel Prize. He questions the validity of some of the winners (though this is increasingly looking correct to do so) and also quotes others who have suggested that the Nobel Prize was a PR stunt designed to put economics on the same step as the natural sciences. Of course, he has not won one yet…

Anyway, the premise of the book is that we often assume that we live in a world known as Mediocristan in which distribution is on a bell curve. Outliers can only go so far. Height for example, you can only deviate so far from the average. However, the great challenge for society is that we actually live in Extremistan, where outliers can deviate significantly. These are the unknown unknowns as Donald Rumsfeld would put it.

How we deal with them is a difficult one. They are the unpredictable, and therefore by their very nature, we cannot predict them. Rather, we need to be prepared to handle them when they inevitably do happen. Forget trying to predict the next outlier that is completely missed by our models and instead try to robust enough to cope when negative ones happen (as well as taking advantage of the positive ones).

The book also deals with human thought processes, in particular our need to turn everything into a narrative. Most skeptics will know that one of the problems with the world is that anecdotes are more easily accepted than data, which makes it so far to get the skeptical point of view across. The issue also causes a lot of bad thinking – we fit things into a narrative that simply do not belong in one.

It is however, something we can turn to our advantage when we recognise it. For example, when going over an unpleasant situation or memory, insert it into a narrative that makes it unavoidable. Also writing down your problems in a narrative can make you feel less guilty about them. Anyway, that is just a small aside.

One of the key messages that I think we should take from Taleb, is the same message that we can take from most of the books I have read recently – that our thinking is flawed, but by recognising those flaws and trying to spot the weakness we know are there, we can be a little less stupid.

The Black Swan

The Signal and the Noise

Tuesday, March 25th, 2014 | Books

Nate Silver is the man who correctly predicted 51 of the 52 states in the 2008 US Presidential Election, and then all 52 in the 2012 election.

With an increasing number of people recommending I read his book “The Signal and the Nose”, I decided to give it a read. It looks at why we, as a society, are pretty bad at making predictions. Why did nobody see the 2008 financial crisis coming? Why is our best guess at when the next earthquake will hit no better than random chance? Why can’t we even predict if it will rain or not?

Actually, the last one, we can. Weather forecasts have become far more accurate over the last few decades. However, they are one of the few fields in which the large scale application of data and computing power to process that data has truly been effective.

Silver claims that one of the biggest problems is that as we now live in the “information age”, there is simply too much data to work out what is actually a useful predictor (the signal) and what is merely correlated (the noise). A great example of this is that the Super Bowl winner (AFC or NFC) was an accurate predictor of how the economy would do. But obviously that is just random chance and has proved erroneous in the past few years.

Ultimately the book has a simple message – you need to use a Bayesian model and apply regression. None of this is a new concept to me, nor indeed you would hope anyone working in the field of statistics. But judging by some of the meetings I have had recently, it is shocking the amount of people that do not follow this advice.

the-signal-and-the-noise