An analysis of 30 years of educational research by scholars at Johns Hopkins University found that when a maker of an educational intervention conducted its own research or paid someone to do the research, the results commonly showed greater benefits for students than when the research was independent. On average, the developer research showed benefits — usually improvements in test scores — that were 70 percent greater than what independent studies found.
The study, “Do Developer-Commissioned Evaluations Inflate Effect Sizes?” was presented at a March 2019 conference session of the Society for Research on Educational Effectiveness (SREE) in Washington, D.C.
https://hechingerreport.org/wp-content/uploads/2019/03/developer-abstract.pdf
A second dark side: lack of replication in education research
Carl Sagan (1997) wrote:
‘At the heart of science is an essential balance between two seemingly contradictory attitudes—an openness to new ideas, no matter how bizarre or counterintuitive they may be, and the most ruthless skeptical scrutiny of all ideas, old and new. This is how deep truths are winnowed from deep nonsense.
Makel, M. C., & Plucker, J. A. (2014). Facts Are More Important Than Novelty: Replication in the Education Sciences. Educational Researcher, 43(6), 304–316. https://doi.org/10.3102/0013189X14545513
Conclusions
Like Campbell and Stanley (1963) noted a half century ago about experimental design, replication is not a panacea (Makel et al., 2012; Pashler & Wagenmakers, 2012). It will not resolve all issues and concerns about rigor, reliability, precision, and validity of education research. However, implicitly or explicitly dismissing replication indicates a value of novelty over truth (Nosek, Spies, & Motyl, 2012) and a serious misunderstanding of both science and creativity. If education research is to be relied upon to develop sound policy and practice, then conducting replications on important findings is essential to moving toward a more reliable and trustworthy understanding of educational environments. Although potentially beneficial for the individual researcher, an overreliance on large effects from single studies drastically weakens the field as well as the likelihood of effective, evidence-based policy. By helping, as Carl Sagan (1997) noted, winnow deep truths from deep nonsense, direct replication of important educational findings will lead to stronger policy recommendations while also making such recommendations more likely to improve education practice and, ultimately, the lives of children.