Stats >> Details >> Steve Simon's Consulting Services
This page was written by Steve Simon and edited by Linda Foland. This page was last modified on 11/21/03 by Linda Foland . Send feedback to ssimon@cmh.edu.
Stats >> Journal >> What's Wrong with Medical Research
This is a draft version of a speech I gave at the May 4, 2000 meeting of Bluejacket Toastmasters.
Introduction
"I can't believe all the junk that gets published in top notch medical journals!" That's a comment I hear all the time. I work with some prominent researchers at Children's Mercy Hospital and they don't trust a lot of the stuff that is published in the Journal of the American Medical Society, Lancet, the New England Journal of Medicine, and other prominent medical journals.
That's just an opinion, of course. As a statistician, I always trust cold hard data to opinions. In this case, though, the cold hard data backs up what my colleagues are saying.
Thornley and Adams Overview
One piece of data about medical research is an article by Ben Thornley and Clive Adams that appeared in the British Medical Journal in October of 1998. The title of this article is "Content and Quality of 2000 Controlled Trials in Schizophrenia over 50 Years." Thornley and Adams work for the Cochrane Collaboration Group, a group that provides systematic reviews of the results of medical trials, so they are in a good position to write such an article.
Thornley and Adams actually found over 2500 studies, but decided to summarize only the first 2000 that they uncovered. Only the first 2000. I still am very impressed at the amount of work this must have taken.
The research covered fifty years, starting in 1948 through 1997. The research covered a variety of therapies: drug therapies, psychotherapy, policy or care packages, or physical interventions like electroconvulsive therapy.
What did Thornley and Adams find? It wasn't a pretty picture. For the most part, researchers in schizophrenia studied the wrong patients, they didn't study enough of them, they didn't study their patients long enough, and they didn't measure them properly. Let's look at each of these assertions in turn.
They studied the wrong patients
First, the researchers studied the wrong patients. Only 14% of the studies of schizophrenia are community based, but that's where most patients are treated. Community based studies are hard to design, because you have so much less control than you would with a pool of hospitalized or institutionalized patients. But research in a hospital does not extrapolate well to other settings. It's like the story of a man who lost his wallet at night in a dark alley, but was searching for it out in the street underneath a street lamp. Why was he searching in the street, you might ask? Because the light is better out here.
They didn't study enough patients
Second, the researchers didn't study enough patients. The precision of the research studies depends on the number of patients studied. Thornley and Adams showed that a typical study of schizophrenia would require a total of 200 patients to achieve adequate precision. What was the average number of patients studied in these 2000 trials? 65. The researchers needed 300 subjects, but on average got only 65. Only 3% of the researchers met Thornley and Adams' goal of 300 patients.
They did not study these patients long enough
Third, the researchers did not study these patients long enough. According to Thornley and Adams, a good research trial should last at least six months. More than half of the trials lasted six weeks or less. Clearly it costs less to run a short trial than a long trial, but short term improvements in any medical study are much less important than long term changes.
They did not measure their patients properly
Finally, the researchers did not measure their patients properly. Measuring improvement in a schizophrenic patient is indeed difficult, but the researchers blew it badly here. In 2000 trials, the researchers developed and used 640 measures of these patients. This shows a serious lack of standards in schizophrenia research. The use of 640 different measurements shows is no consensus on how to measure the severity of schizophrenia or how much a patient might have improved under a certain therapy. This makes it very difficult to compare results across different studies. You might say that the research about schizophrenia is schizophrenic.
Conclusion
The problems that Thornley and Adams raise are not limited to schizophrenia or even psychiatry. There is plenty of objective evidence in many other medical and scientific areas that the reports published in research journals has many of the same problems.
It's hard to do good medical research, especially in an area like psychiatry. But as Thornley and Adams have shown, we could be doing a lot better.
- We study the wrong patients. We grab the easy to get hospitalized and institutionalized patients when we should be doing more studies in the community.
- We don't study enough patients. A good study would require 300 patients, but the average study only had 65.
- We don't study patients long enough. A good study would require six months of monitoring, but more than half of the studies lasted only six weeks.
- And we don't measure things well. There were 640 different ways of assessing how the patients in these studies were doing.
So the next time you hear news reports about the latest research findings, be sure to be a little bit skeptical. Just because it appears in the New England Journal of Medicine doesn't mean that it is true.
There is a handout which I used during this talk.
Stats >> Journal >> What's Wrong with Medical Research
This page was last modified on 08/25/2003 . Send feedback to ssimon@cmh.edu.
Stats >> Journal >> How to read a medical journal article (November 2001 version)
"Still, it is an error to argue in front of your data. You find yourself insensibly twisting them around to fit your theories." Sherlock Holmes in The Adventure of Wisteria Lodge.
The medical journals are filled with research on new medical therapies. What should you look for in this research? How do you gauge the strength of evidence? When should you change your medical practices?
The answers lie not in how the research data was analyzed but in how it was collected. Simple factors like how the research subjects were recruited determine the strength of evidence in a research paper. When you are reading a journal article, just ask yourself five simple questions: Who did the choosing?; Was there a plan?; Who knew what when?; Who was left out?; and How much did things change?
Important Disclaimer.
This presentation will review several published journal articles. The intent is to gauge how much evidence each article presents in favor of the efficacy of a new therapy. Some articles will provide a greater level of evidence and some will provide a lesser level of evidence. But articles which provide lesser levels of evidence are still valuable and important.
Nothing stated in this presentation about a particular journal article should be construed as a statement about the quality of that article. The very nature of research requires a series of steps from very preliminary and speculative levels of evidence to more definitive levels of evidence.
Furthermore, when I point out limitations in the evidence presented in a journal article, more often than not, the authors of the article delineate these same limitations in their discussion. But in general, you need to be aware of these limitations because not every journal author is going to be open and honest about the limitations of their research.
Here are five questions you should ask yourself when reading a journal article.
1. Who did the choosing?
2. Was there a plan?
3. Who knew what when?
4. Who was left out?
5. How much did things change?The first five chapters of this presentation will discuss each of these questions in detail. There are two additional chapters.
7. Special guidelines for meta-analysis.
8. A resource list.
Chapter 1: Was there a good comparison group?
Introduction
Almost all research involves comparison. Do woman who take Tamoxifen have a lower rate of breast cancer
