The effectiveness of the nasal spray flu vaccine illustrates of the limitations of randomized controlled trials. Randomized controlled trials have long been held up as the “gold standard” of clinical research. There’s no doubt that well-designed trials are effective tools for testing a new drug, device, or other intervention. Yet much of modern medical care — perhaps most of it — is not based on randomized controlled trials and likely never will be. In this “dark matter” of clinical medicine, past practices and anecdotes all too often rule. We need to look beyond trials to improve medical care in these areas. In a randomized controlled trial (RCT), participants are randomly assigned to receive either the treatment under investigation or, as a control, a placebo or the current standard treatment. The randomization process helps ensure that the various groups in the study are virtually identical in age, gender, socioeconomic status, and other variables. This minimizes the potential for bias and the influence of confounding factors. Despite their strengths, RCTs have substantial limitations. They can be very expensive to run. They can take many years to complete, and even then may not last long enough to assess the long-term effect of an intervention such as vaccine immunity, or to detect rare or long-term adverse effects. Findings from RCTs may not be valid beyond the study population — a trial that included a high-risk population in order to maximize the possibility of detecting an effect, for example, may not be relevant to a low-risk population. RCTs may not be practical for population-wide interventions and often aren’t relevant for urgent health issues such as infectious disease outbreaks, for which public health decisions must be made quickly. As I write this week in the New England Journal of Medicine, several other study types can generate data that are at least as effective as RCTs, or may be even more effective, at generating evidence for action, especially related to population-wide interventions. The effectiveness of the nasal spray flu vaccine (also called the live attenuated vaccine) is a dramatic illustration of the limitations of RCTs. Trials suggested that the nasal spray vaccine was superior to flu shots, at least for some populations. In subsequent years, however, observational studies, including case-control studies, documented that, for reasons which are still unclear, the nasal spray wasn’teffective against the flu. That led the Advisory Committee on Immunization Practices to recommend, and the CDC to accept the recommendation, that the nasal spray flu vaccine not be used in the 2016-2017 flu season. For some public health issues, it isn’t ethical to conduct an RCT. Take sudden infant death syndrome (SIDS). Early case-control studies suggested, but didn’t prove, that babies who sleep on their stomachs are more likely to die of SIDS than babies who sleep on their backs. It wouldn’t have been ethical to randomize some babies to stomach sleeping. A public program to implement putting children to sleep on their backs proved that this measure reduced the incidence of SIDS. It would be difficult, if not impossible, to do an RCT of community-wide tobacco control measures. But analyses of the results of implementing tobacco control policies, such as taxes, smoke-free laws, and advertising bans, have generated robust evidence of effectiveness that could not have been accomplished through an RCT-style study. For the several thousand rare diseases, RCTs are unlikely to be conducted due to the small number of people who have them and other logistical constraints. Detailed case studies, registries that collect information about specific conditions and diseases, and other study types can enhance understanding of a particular disease and its treatment to improve the health of affected patients. The emerging use of “big data,” including information from electronic health records and expanded patient registries, presents new opportunities to conduct large-scale studies with many of the benefits of RCTs but without the expense. One such study used data from the Veterans Health Administration and Medicare to examine outcomes of treatment for type 2 diabetes. This study was many times larger, with much longer follow-up and lower cost, than previous RCTs comparing the effectiveness of different diabetes drugs. It clearly showed that one class of drug, the thiazolidinediones, was much more effective than another class, the sulfonylureas, in reducing hospitalization and death. Clinical and public health decisions are almost always made with imperfect data. There is no single, best approach to obtain better information about health interventions. Evidence grading systems, policy makers, and researchers must embrace other study types in addition to RCTs. Essential steps in interpreting findings and identifying data for action include promoting transparency in study methods, ensuring standardized data collection for key outcomes, and using new approaches to improve data synthesis. Despite the global evidence base, around the world there are often claims that “there is no evidence tobacco harms health here” or that “soda isn’t proven to drive obesity in this country.” In part, such claims can be made because some formal systems of analyzing evidence give undue weight to RCTs and inappropriately discount other types of rigorously developed evidence. A valid ideal is “evidence-based practice,” which means implementing in clinical care and public policy interventions that are proven to work. But it’s also important, and perhaps more so, to develop “practice-based evidence,” — that is, to implement programs and rigorously document whether or not they work. That would both save lives and expand the evidence base of effective interventions. There will always be an argument for more research and for better data. But waiting for more data is often an implicit decision not to act, or to act on the basis of past practice rather than on the best available evidence. Glorifying RCTs above other approaches, even when these other approaches may be either superior or the only practical way to get an answer, relegates patients to receiving treatments that aren’t based on the best available evidence. An approach that uses all appropriate evidence types and builds on the existing evidence base using proven best practices is the one most likely to result in clinical and public health action that will save lives. Source
