Additional resources
Randomised controlled trial of cardiotocography versus Doppler auscultation of fetal heart at admission in labour in low risk obstetric population ?/font> Commentary: changes between protocol and manuscript should be declared at submission ?/font> Commentary: research governance must focus on research training ?/font> Commentary: Approach to power calculations has to be realistic
Gary Mires, Fiona Williams, Peter Howie, Sandy Goldbeck-Wood, Gordon D Murray, and Britt-Ingjerd Nesheim
BMJ 2001; 322: 1457-1462. [Abstract] [Full text]This reference talks about a change in power calculations after the data was collected.
Chapter 3: Who knew what when?
Introduction
Knowledge of group membership, either before or during the data collection can bias the study.
When you are trying to figure out who knew what when, ask the following two questions:
3.1 During the study, did the patients know which group they were in?
3.2 At the start of the study, did the patients know which group they were going to be placed in?
Accupuncture
Acupuncture is an example of a therapy that is difficult to blind. One study of the effect of accupuncture on the prevention of recidivism among alcohol and other drug abusers used a placebo accupuncture that placed needles 5 mm away from the designated accupuncture point. Because of the nature of accupuncture, the accupuncturists were aware of which patients were which, making this a single blind study.
A critique of this study pointed out that there were significant interactions between the accupuncturists and the patients, with opportunities for indirect suggestion and nonverbal communication to occur. One indication that subjects became aware of who was in which group was the fact that there was a far greater tendency for control subjects to drop out of the study.
3.1 During the study, did the patients know which group they were in?
In an experimental study, it is desirable (but not always possible) to keep the information about the treatments hidden from the patients and anyone involved with evaluating the patient. This is known as "blinding." Blinding prevents conscious or subconscious biases or expectations from influencing the outcome of the study.
Unfortunately, there are many situations where blinding is impossible. For example, if you are comparing oral versus rectal administration of a drug, that's pretty hard to conceal from the patient. In general, observational studies cannot be blinded, because the patient and/or their doctor selects the treatment group.
Unblinded studies are still useful, but they are considered less authoritative than blinded studies.
The placebo effect.
Positive effects of a treatment are sometimes due to a placebo effect. The placebo effect is a product of "belief, expectancy, cognitive reinterpretation, and diversion of attention" that can lead to psychological and sometimes physiological improvements in situations where the treatment is known to have no effect, such as sugar pills (Beyerstein 1997).
Johnson (1997) lists three specific situations where the placebo effect is of particular concern: when enthusiasm by the patient or the doctor for the new procedure is strong, when outcomes are based on the patient's self-assessment (e.g. quality of life studies), and when the treatment is primarily for symptoms. The placebo effect is less critical for objective outcomes like survival.
Blinding in surgical trials.
Surgical procedures are often difficult to completely blind. Nevertheless, Johnson (1997) suggests some partial steps at blinding that prevent some of the biases from creeping in.
If two surgical procedures use different types of incisions, identical blood or iodine stained opaque dressings could be used to keep the patients unaware of which operation was performed.
Also, although the surgeon cannot be blinded to the difference in surgery, those who evaluate the health of the patient after surgery could be kept unaware of the particular operation, so as to insure that their evaluation of the patient is unbiased.
Partial blinding in an observational study.
As noted earlier, it is impossible to completely blind an observational study. Gail (1996), however, describes an observational study where some level of blinding was achieved.
In a study of the relationship of smoking and cancer, the people asking questions about smoking and other risk factors were unaware of when they were interviewing lung cancer patients or controls. Thus, the interviewers could not subconsciously probe harder for smoking information among the lung cancer patients.
The problem with studies without blinding.
Two researchers have examined studies with and without blinding. These authors found that studies without blinding show an average bias of 11-17% (Schulz 1996; Colditz 1989). In other words, when an unblinded study was compared to a blinded study, the former study tended to estimate a treatment effect that was (on average) 11% to 17% higher than the latter.
Additional evidence of this problem appears in a meta-analysis of the effect of intermittent sunlight exposure and melanoma (Nelemans 1995). When nine studies without blinding were combined, they showed a odds ratio of 1.84 which was statistically significant (95% confidence interval 1.52 to 2.25). When the seven studies with blinding were combined, they showed a much smaller odds ratio (1.17, 95% confidence interval 0.98 to 1.39) which was not statistically significant. This is further evidence that unblinded studies are more likely to show statistical significance than blinded studies.
Problems with keeping a treatment blinded.
Even though the placebo may look the same, sometimes the doctor may infer which group a patient belongs to, perhaps through noting a characteristic set of side effects. In an anonymous survey, more than half of the doctors participating in research studies admitted to breaking a blinded allocation (Schulz 1996). If you are worried about this, ask the doctors to try to identify which treatment group they believe each patient belonged to. If the percentage of correct guesses is significantly larger than 50%, then the allocation scheme was not sufficiently blinded.
3.2 At the start of the study, did the patients know which group they were going to be in?
The randomization list should be blinded to those involved with recruiting subjects.
It is always possible to blind the randomization list, even when the treatment itself cannot be blinded. Check out all the exclusion criteria and if the subject qualifies, open a sealed envelope which identifies which group the patient belongs to. So, for example, it is impossible to use blinding when comparing a surgical to a non-surgical technique, but the selection of who gets the surgical technique could be hidden from both the patient and the surgeon until after all the selection and inclusion criteria are applied.
Knowledge of treatment order allows the doctors recruiting patients to consciously or unconsciously influence the composition of the groups. They can do this by applying exclusion criteria differentially or by delaying entry of a certain healthier (or unhealthier) subject so he/she gets into the "desirable" group. Unblinded allocation schemes show an average bias of 30-40% (Schulz 1996).
Problems with systematic allocation.
Systematic allocations can also cause biases. For convenience, some researchers will allocate in a systematic (non-random) fashion, such as alternating regularly between the two treatments. This is a bad idea. Patients may arrive in a systematic order. Systematic allocations allow the doctors to guess which group the next patient is going to be allocated to. Systematic assignment causes an average bias of 15% (Colditz 1989).
Summary - Who knew what when?
Knowledge of group membership, either before or during the data collection can bias the study.
3.1 During the study, did the patients know which group they were in? While this is not always possible, it is preferred to use a blinded approach to remove the possibility of the placebo effect.
3.2 At the start of the study, did the patients know which group they were going to be in? Even when blinding is impossible, you can always hide the randomization plan through the use of sealed envelopes. This will ensure that the health professional do not consciously or subconsciously influence group membership through the differential application of entry criteria.
3.3 Did the authors rely on retrospective data? Retrospective data are more likely to suffer from inaccuracy, incompleteness and bias.
