Study Finds That Many Published Psychology Experiments Lack Evidence Of Validity

An examination of nearly 350 published psychological experiments found that nearly half failed to show that they were based on a valid foundation of empirical evidence, suggesting that a wide swath of psychological science is based on an "untested foundation."
The study—conducted by David Chester, PhD, a psychology professor at Virginia Commonwealth University, and Emily Lasko, a psychology doctoral student at VCU—focuses on the practice of experimental manipulations, in which psychologists induce specific mental states, such as giving research participants insulting or complimentary feedback to manipulate how angry they feel.

To conduct these experimental manipulations in a scientifically valid way, researchers must first establish whether their manipulations actually affect the intended psychological variable (for example: make people feel angry) and not other closely related variables (for example: make people feel sad). However, the extent to which psychologists actually examine the validity of their manipulations remains unknown.

Chester and Lasko investigated 348 psychological manipulations included in peer-reviewed studies. They found that roughly 42% of the experiments were paired with no validity evidence, and that the remaining psychological manipulations were validated in ways that were extremely limited.

"These findings call into question the accuracy of one of psychology's most common practices and suggest that the field needs to strongly improve its practices in this methodological domain," said Chester, an assistant professor in the Department of Psychology in the College of Humanities and Sciences.

The forthcoming study, "Construct Validation of Experimental Manipulations in Social Psychology: Current Practices and Recommendations for the Future," will be published in the journal Perspectives on Psychological Science.

The study is the first of its kind to systematically document the extent to which psychology's experiments are based on a valid foundation of empirical evidence.

Importantly, Chester said, the study's findings do not suggest that the experimental psychologists' findings were necessarily wrong or invalid.

"We do not find such experiments are invalid, instead we simply don't have the evidence to know one way or another how valid they are," he said. "Almost all of the manipulations we examined failed to provide the necessary evidence that they were valid, which does not mean they are invalid—their validity is just unknown."

As a result, he said, the study suggests that "the findings of experimental psychology likely rest on an untested foundation."

"This framework might be weak, it might be strong, it is more likely both of these things depending on many factors," he said. "We have outlined a prescribed series of recommendations for experimenters to ensure that this is not the case going forward—that the validity of each experimental manipulation is tested in a systematic and accurate way."

Chester added that he and Lasko hope their findings encourage experimental psychologists to include validity evidence in future research.

"We hope our paper makes experimenters aware of this untested aspect of their research, motivates them to change their practices, and provides a road map of precisely what to do to make such changes," he said.

Source