3.2.5 Measurements without established reliability
Reliability means different things in different fields, but the general concept is that a reliable measurement is one that would stay about the same if it were repeated under similar circumstances. Depending on the context, you would establish reliability differently. For example, one way to establish reliability is to have two people make independent assessments and show a good level of agreement. If you are measuring something that is stable over time, then you could take two measurements on different days or weeks and see how well they agree.
Be especially careful about measurements that have some level of subjectivity. If there is no establishment of reliability for these measures, then you have no assurance that the research is repeatable.
[Add two good examples.]
3.2.6 Post Hoc Changes
No research plan is perfect, and you should expect minor deviations from the plan in just about any research study. Major deviations, however, from the protocol can reduce the credibility of a study. Some examples of deviations from the plan include:
- Investigating end-points other than those originally specified
- Developing new exclusion criteria after the study has started
- Stopping the study unexpectedly or extending it beyond the planned sample size
You need to ask yourself if the authors deviated from the protocol in a conscious or subconscious effort to manipulate the results. Did the authors add other end-points in order to salvage a largely negative study? Were new exclusion criteria targeted to keep "troublesome" subjects out? It is impossible, of course, to discern the motives of the researchers. Nevertheless, for any deviation or modification to the protocol, you can ask whether this change would have made sense to include in the protocol if it had been thought of before data collection began.
Changes to the planned end of the study, either stopping the study early, or extending it beyond the planned sample size, can raise some serious problems (Ludbrook 2003). There are several reasons that you might want to stop a study early:
- early evidence that one of therapies is much better than the other (efficacy),
- early evidence that continuing the study would be unlikely to yield a significant result (futility),
- early evidence that one of the therapies is too dangerous (safety), and/or
- finishing the study would end up being far more expensive or time consuming than the original plan (economics).
[Insert a good example here.]
In order to maintain credibility, a study should have rules for stopping early that were specified prior to the start of data collection. Pre-determined rules are especially important when a study ends early for efficacy. If a study ends early for economic reasons, and the result is not statistically significant, you need some assurance that the truncated sample size still provided a reasonable level of precision. In this situation, the width of the confidence intervals would indicate clearly if the sample size was still adequate.
Extending a study beyond the original end date can also be problematic. Extensions for economic reasons (the budget went further than expected or an extra funding source appeared) is probably not a serious problem, but be very careful if the study gets extended because of a failure to achieve statistical significance at the planned sample size. The provisions for such an extension must be specified prior to the start of data collection.
Detecting a deliberate and fraudulent change in a research study is extremely difficult for anyone, but especially difficult for the reader. A thorough peer review provides a limited level of protection from fraud. Another suggested remedy is a proposed requirement that journals should see the original protocols for research studies as part of the peer review process (Hawkey 2001). Sometimes a careful review of the numbers in a study can highlight the possibility of fraud. If a study used randomization, for example, watch out if there is an unexpected and unexplained deviation from a 50-50 split between treatment and control. Replication of research findings is also a good protection against fraud.
Example: An interesting deviation from the research plan occurs in a randomized double blind control trial for the use of selenium supplements (Clark 1996). The study was initiated in 1983 with basal skin carcinoma and squamous skin carcinoma as the primary end points. The researchers also looked for signs of selenium toxicity. In 1990, funding was obtained to look at additional secondary end points (total mortality, cancer mortality, and incidence of lung, colorectal, and prostate cancers). While it was relatively easy to add extra endpoints in the middle of the study, the authors acknowledged that this represented a deviation from the protocol. Another deviation from the protocol occurred when the study was terminated early (January 1996). No statistical changes were found in the primary endpoints, nor was any evidence of selenium toxicity found. Among the secondary endpoints, however, the authors found statistically significant declines in total cancer mortality and lung cancer mortality. The authors also found statistically significant declines in the incidence of prostate cancer, colorectal cancer, lung cancer and total carcinomas. There was also a decline in overall mortality, though it did not achieve statistical significance. There were no significant changes in the incidence of nine other types of cancer, including breast cancer, bladder cancer, and leukemia. Because the significant results occurred in areas that were not originally planned for study, the authors acknowledge that any results have to be considered preliminary. Furthermore, it is unclear what impact the early termination of the study had on the statistics. Early termination of a study can cause serious biases, unless specific rules for early termination are established at the start of the study.
3.2.7 Measuring the outcome well--what to look for
When you are looking at how the outcome was measured, ask yourself the following questions:
- Was the outcome dependent on the memory of the patients?
- Did the outcome have established validity and reliability?
- Were there post hoc changes in the protocol?
3.3 Were the changes clinically important?
[Add material to this section]
Examples
Absolute Risk
Particularizing
3.4 Summary - Mountain or molehill?
Look carefully at how the researchers measured the outcome in their study.
Did they measure the right thing? You would like to see an outcome of direct interest to your patients.
Did they measure it well? You want an outcome that is valid and reliable and not subject to changes are the start of data collection.
Were the changes clinically important? You want a change that is large enough to have a practical impact in a clinical setting.
