When scientists want their data fudged
and why you should care
Some statisticians have been asked to falsify
significance of research results, new study finds
Kelly Crowe · CBC News ·
A shocking revelation during a casual conversation prompted a New York epidemiologist to investigate the ethically dubious things statisticians are sometimes asked to do when analyzing research findings. (sasirin pamai/Shutterstock)
This is an excerpt from Second Opinion, a weekly roundup of eclectic and under-the-radar health and medical science news emailed to subscribers every Saturday morning. If you haven't subscribed yet, you can do that by clicking here.
How often are you asked to do something unethical? That was the blunt question at the heart of a surprising survey of statisticians released this week.
The answer? Many statisticians reported that scientists routinely asked them to fudge data.
McGill University bioethicist Jonathan Kimmelman said the results should be concerning to everyone.
"We use [statistical] information to make decisions about what drugs to take, what foods to eat, what policies to make, what chemicals to ban," Kimmelman said. "It's crucial to protect the integrity of that data."
Ralph Katz, a New York epidemiologist, got the idea for the study after he casually asked a statistician if scientists ever wanted him to manipulate their data to get a better result.
"I just asked him one day when we were chatting. And he said 'frequently.'"
"We use [statistical] information to make decisions
about what drugs to take, what foods to eat,
what policies to make, what chemicals to ban.
- Jonathan Kimmelman, McGill University bioethicist
Shocked, Katz asked other statisticians and realized it was a common experience among the mathematical whizzes who analyze research results.
"They talk about what they've been asked to do over beers, after the meetings of the day." Katz said.
But there was no data to reveal the extent of the problem, so Katz conducted a formal survey sent to a randomly selected group of statisticians.
Published this week in the Annals of Internal Medicine, the ethically dubious things the statisticians were asked to do included altering data records, falsifying statistical significance, and stressing only the significant findings.
"Greater than 20 per cent, sometimes up to 50 per cent, said these had occurred a multiple number of times over the last five years," said Katz, adding that this does not mean all research is compromised.
"There are a lot of consultations being done where this is not happening, but it is alarming how often it is happening."
Most people are unaware that statisticians have a critical role in research. It's their job to analyze complex arrays of data to determine if research findings are genuine.
"What a statistician is helping a scientist do is to cut through randomness and to cut through bias to see relationships that are likely to be real," said Kimmelman.
Part of the problem is that researchers consult statisticians too late, expecting them to fix problems after the data has already been gathered.
'Jarring' experience
It's a frustration for Andrew Althouse, a biostatistician at the University of Pittsburgh.
"I feel like I've been asked to do quite a few of these at least once" said Althouse. "I do my best to stand my ground and I've never falsified data."
Althouse describes one troubling experience when a surgeon pressured him to provide data on 10-year survival rates after a particular surgical intervention. The problem — the 10-year data didn't exist because the hospital hadn't been using the procedure long enough.
"The surgeon argued with me that it was really important and pleaded with me to find some way to do this," Althouse said. "He eventually relented but it was one of the most jarring examples I've experienced."
Sometimes researchers are asking because they just don't know enough about statistics. But Kimmelman said there is a more disturbing possibility.
"A less benign interpretation is that they actually know what they're doing and they have questionable professional integrity."
"Dishonest statistical analyses can lead to false discoveries," said Althouse, adding that young statisticians are more vulnerable.
"If you're a junior person who's faced for the first time telling a surgeon you work for that you don't want to do something they told you do to, that can be a pretty intimidating situation."
'Strong incentives for people to fudge'
The survey did not ask how often the statisticians agreed to tinker with the numbers. But the results warrant further investigation, said Russell Localio, a biostatistician at the University of Pennsylvania and lead author of an editorial published along with the survey.
"Given the number of respondents and the frequency and nature of the reported requests, these findings suggest that requests for inappropriate statistical methods is a real issue that needs to be studied further and addressed," he said in an email to CBC News.
"If statisticians are saying no, that's great," said Kimmelman. "But to me this is still a major concern."
Kimmelman does his own research using statistics. And he's not surprised that there's pressure to embellish results.
"Everyone has had papers that are turned down by journals because your results were not statistically significant," he said.
"Getting tenure, getting pay raises, all sorts of things depend on getting into those journals so there is really strong incentives for people to fudge or shape their findings in a way that it makes it more palatable for those journals."
"And what that shows is that there are lots of instances where there is threat of adulteration of the evidence that we use."
It is not just in medical science where data are fudged, although I am confident in the lack of integrity of some pharmaceutical companies, but there has been proven to be manipulation of data even in the atmospheric sciences. I do believe there is global warming. I also believe that there is very little real data linking it firmly to carbon emissions. And, I believe there is far too much hyperbole and hysteria being promoted by scientists and pseudo-scientists.
Many an environmentalist believes scientific data are sacred. Anything that can be manipulated by man for personal gain is vulnerable to fraud, even science.
