But then, 90% of everything is crap; so we knew that
Apparently, someone has done an interesting meta-study of scientific publishing, and titled his paper, “Why Most Published Research Findings Are False” (PLoS Medicine).
As a sarcastic philosopher type, it is my duty to observe that Dr. Ioannidis’ claim is self-stultifying. If most published research findings are false, it follows that, probably, Ioannidis’ research finding is false. Therefore, probably, most published research findings are true!
Summary by WSJ Online.
LOL.
One of my favorite self-refuting statements is “There is not such things as truth.”
Is that true?
Comment by Stephen Ewen — September 18, 2007 @ 11:53 am
My favourite line from the linked article is: “He has done systematic looks…” Maybe this is acceptable US English, but do Americans really “do looks”? I always try to take my looks:-)
More seriously, once you start paying people to do things, they tend to do them with some enthusiasm. This is the disaster now internationally with academic work and publishing, where careers are made by quantity publishing and there is little quality evaluation. Of course, there is this absurd mythology of “refereed journals” that Americans and Brits go on about, but it is difficult not to laugh at their sincerity on the issue. OF COURSE 90% IS CRAP: WHY WOULDN’T IT BE? After all, nobody did anything to stop it!
Comment by Martin Baldwin-Edwards — September 18, 2007 @ 2:42 pm
The ideals which inspired the Age of Enlightenment are less of a guiding light in today’s competitive world of scientific career building. On the positive side, there is less difficulty in scrutinizing data published in science journals. Before the internet, non-academics would have to make travel arrangements to visit a specialist library that kept all the major journals for inspection by the public. They were few and far between — mostly in large cities in Europe and America. Now, nearly all major journals make abstracts available on their websites and as someone pointed out, the full text can often be found on faculty websites. Shortcomings can no longer be concealed.
The Ioannidis meta-study was covered in New Scientist magazine when it first appeared in 2005. It followed another survey by the Health Partners Research Foundation in Minneapolis, which sent questionnaires to researchers listed by the US National Institutes of Health. 3247 were returned. The results were published in Nature (vol 435, p 737). 15.5 per cent admitted changing their design, methodology or results to suit a sponsor, and 6 per cent admitted suppressing data. More than a quarter owned up to inadequate record keeping, and 10 per cent confessed to inappropriately giving credit to an author. See:
http://www.newscientist.com/article/mg18625033.400.html
In 2006, an Editorial in New Scientist lamented that corporate-sponsored research has undermined the standards of evidence-based medicine:
http://www.newscientist.com/channel/opinion/mg19025472.500.html
The peer-review system is subject to confirmatory bias too, as this 1977 experimental study demonstrated:
http://www.stern.nyu.edu/~wstarbuc/Writing/Prejud.htm
But what made me laugh out loud was the reader comment in this blog post about genetic bias.
Comment by Eire Islander — September 19, 2007 @ 9:26 am
Anyone interest in bad science should read the article titled “Do We Really Know What Makes Us Healthy?” By GARY TAUBES and published on September 16, 2007 in the Sunday Times magazine.
http://www.nytimes.com/2007/09/16/magazine/16epidemiology-t.html?_r=1&ref=magazine&pagewanted=all
The following three quotes from the article are interesting.
“In January 2001, the British epidemiologists George Davey Smith and Shah Ebrahim, co-editors of The International Journal of Epidemiology, discussed this issue in an editorial titled ‘Epidemiology — Is It Time to Call It a Day?’ They noted that those few times that a randomized trial had been financed to test a hypothesis supported by results from these large observational studies, the hypothesis either failed the test or, at the very least, the test failed to confirm the hypothesis: antioxidants like vitamins E and C and beta carotene did not prevent heart disease, nor did eating copious fiber protect against colon cancer.”
“As John Bailar, an epidemiologist who is now at the National Academy of Science, once memorably phrased it, ‘The appropriate question is not whether there are uncertainties about epidemiologic data, rather, it is whether the uncertainties are so great that one cannot draw useful conclusions from the data.’ “
“Richard Peto, professor of medical statistics and epidemiology at Oxford University, phrases the nature of the conflict this way: ‘Epidemiology is so beautiful and provides such an important perspective on human life and death, but an incredible amount of rubbish is published,’ by which he means the results of observational studies that appear daily in the news media and often become the basis of public-health recommendations about what we should or should not do to promote our continued good health.”
Comment by Chris D — September 19, 2007 @ 7:23 pm