‘Why I believe statistics can beat fake news’
Data editor of Guardian US, she has previously worked at the polling company FiveThirtyEight, the Bank of England, the Economist Intelligence Unit and a statistical arm of the United Nations.
Many of us have grown wary of statistics. Scepticism is healthy, says Mona Chalabi, but we must not give up on them altogether. Instead we need better tests to help us to spot bad data.
When it comes to numbers, especially now, you should be sceptical. But you should also be able to tell which numbers are reliable and which are not. I am not talking about claims like: “Nine out of ten women recommend this anti-aging cream.” What is different now is people are questioning statistics like: “The US unemployment rate is five percent.” This claim does not come from a private company, it comes from the government.
About four out of ten Americans distrust the economic data that gets reported by government. Among supporters of President Trump it is even higher; it is about seven out of ten. There are a lot of dividing lines in our society, and a lot of them start to make sense once you understand people’s relationships with these government numbers.
On the one hand, there are those who say these statistics are crucial, that we need them to make sense of society as a whole in order to move beyond emotional anecdotes and measure progress. Others say these statistics are elitist, maybe even rigged; they do not make sense and they do not really reflect what is happening in people’s everyday lives.
“We need these government numbers, but we also have to move beyond blindly accepting or rejecting them”
It feels like that second group is winning the argument now. We are living in a world of alternative facts, where people do not find statistics a starting point for debate. This is a problem. There are moves in the USA right now to get rid of some government statistics altogether.
Statistics come from the state; that is where they got their name. The point was to better measure the population in order to better serve it. So we need these government numbers, but we also have to move beyond either blindly accepting or blindly rejecting them.
I want to give you three questions that will help you be able to spot some bad statistics. Question number one is: Can you see uncertainty? A lot of data visualisations will overstate certainty, and it works — these charts can numb our brains to criticism. When you hear a statistic, you might feel sceptical. As soon as it is buried in a chart, it feels like some kind of objective science, and it is not.
The second question is: Can I see myself in the data? Part of the reason why people are so frustrated with these national statistics is they do not really tell the story of who is winning and losing from national policy. It is easy to understand why people are frustrated with global averages when they do not match their personal experiences.
The point is not that every single data set has to relate specifically to you. The point is to get as much context as possible. It is about zooming out from one data point, like the unemployment rate is 5%, and seeing how it changes over time, by educational status or by gender.
The final question is: How was the data collected? Methodologies can be opaque and boring, but there are some simple steps you can take to check this. This is one of the reasons why government statistics are often better than private statistics. A poll might speak to a couple of hundred people, maybe a thousand — or if you’re L’Oreal, trying to sell skin care products in 2005, then you spoke to 48 women to claim that they work.
How do you question government statistics? You just keep checking everything. Find out how they collected the numbers. Find out if you are seeing everything on the chart you need to see. But do not give up on the numbers altogether: if you do, we will be making public policy decisions in the dark, using nothing but private interests to guide us.
This is an extract from a recent talk to TED NYC. Follow the link under Become An Expert to watch the presentation in full.
- Do you trust government statistics?
- Find a recent news story which has reported the results of a poll or survey. Write five questions you should ask to help you understand how accurately it has been reported. Present your answers to your class.
- Alternative facts
- A phrase popularised by Kellyanne Conway, one of President Trump’s advisers, after Trump claimed that the size of the crowd at his inauguration had been significantly under-reported.
- For example, a draft law currently passing through the US Congress says government money should not be used to collect data on racial segregation.
- In recent months some political polls — in the UK, Italy, Israel and the USA — have proved less accurate than many pundits expected. Polling has become inaccurate for many reasons: for example because polling samples are not demographically representative and because people are reluctant to respond to pollsters.
- For example, ensuring that statistics are understood in their full context; asking how the survey was carried out; and finding out how many people were polled.
- Government statistics often draw on bigger sample sizes than private ones. The unemployment statistics the author cites are from the US bureau of labour statistics, which spoke to over 140,000 businesses around the country to get their results.