The Reminder is making its archives back to 2003 available on our website. Please note that, due to technical limitations, archive articles are presented without the usual formatting.
Most of the health stories you read in newspapers or magazines are built around some sort of medical research. But how do you know if the research itself is any good or if it has been accurately reported? Jim Pollard explains how to read between the lines. Firstly, what sort of research is it? ItÕs amazing how many medical breakthroughs have been announced on the back of experiments on mice, rats or on body tissue alone. If it is that sort of research, you can bet it wonÕt be available for human treatment for years, if ever. The most important sort of research so far as the treatment or prevention of disease in humans is a clinical trial Ð a trial involving real people and real treatments. But some trials are more reliable than others. When you see the headline ÔIrradiated cow dung can change your lifeÕ, here are the questions to ask before you rush off to your local farm. 1. Is the trial randomized? This means patients are selected at random not according to the doctorÕs bias which can lead to those who are most likely to be ÔsuccessfulÕ patients being selected. 2. Is it Ôdouble blindÕ? Double-blind means that neither the patient nor the doctor or other professional observing them knows whether that particular patient is receiving the treatment on trial or not. Again this reduces bias. 3. How big is the study? Small studies are unreliable. The rarer the problem under investigation, the more unreliable a smaller study is. 4. Are the results statistically significant? You can do a degree in stats and still be left with questions about statistical significance but put simply, findings are statistically significant if they cannot be attributed to chance alone. HereÕs an example. The odds on getting a head or a tail when tossing a coin are evens (50-50). That means that if you toss a coin 100 times, youÕd expect 50 heads and 50 tails. But in reality this is probably not what would happen, it would be 51-49 or 52-48, perhaps more. However, that would not mean that the odds on the next toss would be different, they are always 50-50. Statistical significance tries to allow for this variation between theory and reality in an experiment. Example: 100 men and 100 women are given an IQ test. The men score 102, the women 104. Does this prove that women are better at IQ tests than men? Probably not. ItÕs not statistically significant. However, if the men score 98 and the women 120 you might be tempted to draw different conclusions. The allowable variation depends on what is being studied and how but five per cent is common. Good research will make it clear that its findings are statistically significant. 5. How was study conducted? The way an experiment is done will nearly always distort the results in one way or the other. You need to look at how an experiment was done and see where the problems and distortions might arise. Patient questionnaires, for example, are unreliable Ð one manÕs ÔexcruciatingÕ pain is anotherÕs Ômild niggleÕ. In an example like this it might also depend on how the questions were asked Ð if by a female researcher, men might play down the pain. Even asking someone if they feel better is fraught with problems. People often feel better simply because someone is bothering to ask them. 6. Did the trial really measure what it claims to have measured? (Or at least, what journalists claim it measured.) Take the example of the IQ test above. If men scored 98 and women 120, the headline could well be Ôwomen more intelligent than menÕ but that would not be true. It presupposes that IQ tests measure intelligence which is debatable. All that can accurately be said is: Ôwomen better at IQ tests than menÕ Ð trouble is thatÕs not such a good headline 7. Who was paying for it? If research findings appear to be in the commercial interest of the firm who sponsored them, it doesnÕt mean theyÕre not true, it simply means youÕre right to be skeptical about them. The increasing commercialization of science with research departments dependent on the publications of their staff for funding can also encourage the ÔspinningÕ of results. 8. WhatÕs the real risk? In the reporting of stories, journalists will often talk about ÔriskÕ. The question is: is this a relative or absolute risk. Absolute risk is what you should be most interested in but relative risk, because itÕs more dramatic, is what is usually reported. Example: you have a one in 10 million chance of catching X. Research shows that drinking tea doubles your chance of catching X. This creates a great headline: Ôtea-drinkers at twice the risk of XÕ. But what does it really mean? Doubling the risk means the chance is two in ten million or one in five million. In other words although the relative risk has doubled, the absolute risk is still very, very small. If, after youÕve asked these eight questions, the research still looks pretty good then you can think about believing it.
