Despite being the gold standard, a recent publication in Annals of Medicine suggests that randomized controlled trials may not be as unbiased as scientists believe.
Randomization is an important element of modern clinical trials. It refers to the process of randomly allocating participants to either the experimental or the control condition. The purpose of randomizing participants is to eliminate bias. However, Alexander Krauss, a fellow in scientific methods at the London School of Economics has recently published an article that investigated why this is potentially not enough.
Through an analysis of the ten most commonly cited randomized controlled trials, where all trials had at least 6,500 citations or more, he suggests these trials may have potential biases that have not been recognized or reported. His findings were recently published in the Annals of Medicine and summarized in a research news report in the BMJ.
Findings may be reported as more significant than they should be
One example Krauss gives is from a randomized controlled trial in 1995. The trial showed that a clot-busting drug was effective for the treatment of ischemic stroke. After randomization, the researchers observed that the treatment group was 3% less likely to have congestive heart failure compared to the control group, 14% more likely to have taken aspirin, and 3% more likely to be white than black. However, the effect of the drug was also only a small difference between the two groups, with only a 4% difference in the chance of dying in the three months following the beginning of the trial.
To have confidence that the differences between the groups are due to the treatment itself, and not due to chance, the process of randomization is meant to create a group with evenly distributed properties. This way, scientists can confidently attribute the treatment effect to the treatment itself and not coincidence. The natural differences found between the two groups, however, shake that foundation.
The argument by Krauss is that if the trial was re-randomized multiple times to more evenly distribute the varying factors, the trial outcomes would likely not be identical and therefore this trial outcome is simply the result of only undergoing randomization once.
This trial included 624 participants, of which 54 patients in the treatment group died within three months, compared to 64 patients in the placebo group. He likens this primary outcome to flipping a coin 624 times and getting heads over tails ten more times, suggesting that many trial findings are reported as more significant than they should be, especially given their size.
Potential biases in the trials
Other potential biases that may exist in randomized controlled trials include sample selection bias. Sample selection bias may arise due to many trials being recruited either from only a small geographical area or from many international centres in an unspecified way (there is limited or no information on the centres, the patients they treat, and by whom).
Another example is the lack of blinding bias, which is demonstrated by a trial that was looking at drugs to lower cholesterol. Despite being randomly assigned either treatment or a placebo, the patients discovered their cholesterol levels on their own and decided to start taking a treatment in place of the placebo.
Another case of where bias may exist in randomized controlled trials is the quantitative variable (a variable that is measured on a numeric scale such as a country’s population or a car’s speed) limitation they impose. This is the requirement for the primary outcome to be easy to compare between the treatment and placebo groups, such as mortality.
The example given is from a colorectal cancer trial, where even though the treatment group survived an average of 4.5 months longer, their quality of life may have been significantly less. The treatment group was 11% more likely to experience a serious accidental reaction or harmful medical occurrence and 14% more likely to have high blood pressure. Hence, Krauss argues that despite the fact that the treatment group patients living longer, due to their decrease in quality of life, they may have actually preferred to not have received treatment.
Updates needed in the CONSORT Guidelines
Despite suggesting that randomized controlled trials all potentially produce unrecognized and unreported biases, Krauss does not discount the relevance of the trials and their importance to inform physicians and scientists in decision making. However, he warns that no sole study should be an authoritative source.n His suggestions to reduce potential bias in future randomized controlled trials include adding some extra recommendations to the Consolidated Standards of Reporting Trials (CONSORT) guidelines.
One of his recommendations is to include patient data at the end of the study, as most trials generally only report the participants’ data at the start of the study. This data is usually assumed to remain unchanged throughout the course of the trial. Krauss also suggests that journals should require a standalone section written by the researchers on their research assumptions, biases, and limitations.
Written by Lacey Hizartzidis, PhD
Reference: Hawkes, Nigel. Randomised controlled trials may have many unrecognized potential biases. The BMJ website https://www.bmj.com/content/361/bmj.k1561. Accessed April 18th, 2018.
