Psychology advances knowledge by testing statistical hypotheses using empirical observations and data. The expectation is that most statistically significant findings can be replicated in new data and in new laboratories, but in practice many findings have replicated less often than expected, leading to claims of a replication crisis. We review recent methodological literature on questionable research practices, meta-analysis, and power analysis to explain the apparently high rates of failure to replicate. Psychologists can improve research practices to advance knowledge in ways that improve replicability. We recommend that researchers adopt open science conventions of preregi-stration and full disclosure and that replication efforts be based on multiple studies rather than on a single replication attempt. We call for more sophisticated power analyses, careful consideration of the various influences on effect sizes, and more complete disclosure of nonsignificant as well as statistically significant findings.

Psychology faces a replication crisis. The Reproducibility Project: Psychology sought to replicate the effects of 100 psychology studies. Though 97% of the original studies produced statistically significant results, only 36% of the replication studies did so (Open Science Collaboration, 2015). This inability to replicate previously published results, however, is not limited to psychology (Ioannidis, 2005). Replication projects in medicine (Prinz et al., 2011) and behavioral economics (Camerer et al., 2016) resulted in replication rates of 25 and 61%, respectively, and analyses in genetics (Munafò, 2009) and neuroscience (Button et al., 2013) question the validity of studies in those fields. Science, in general, is reckoning with challenges in one of its basic tenets: replication. Comparative psychology also faces the grand challenge of producing replicable research. Though social psychology has born the brunt of most of the critique regarding failed replications, comparative psychology suffers from some of the same problems faced by social psychology (e.g., small sample sizes). Yet, comparative psychology follows the methods of cognitive psychology by often using within-subjects designs, which may buffer it from replicability problems (Open Science Collaboration, 2015). In this Grand Challenge article, I explore the shared and unique challenges of and potential solutions for replication and reproducibility in comparative psychology.

A Bayesian perspective on Ioannidis's (2005) memorable statement that "Most Published Research Findings Are False" suggests a seemingly inescapable trade-off: It appears as if research hypotheses are based either on safe ground (high prior odds), yielding valid but unsurprising results, or on unexpected and novel ideas (low prior odds), inspiring risky and surprising findings that are inevitably often wrong. Indeed, research of two prominent types, sexy hypothesis testing and model testing, is often characterized by low priors (due to astounding hypotheses and conjunctive models) as well as low-likelihood ratios (due to nondiagnostic predictions of the yin-or-yang type). However, the trade-off is not inescapable: An alternative research approach, theory-driven cumulative science, aims at maximizing both prior odds and diagnostic hypothesis testing. The final discussion emphasizes the value of pluralistic science, within which exploratory phenomenon-driven research can play a similarly strong part as strict theory-testing science.

Psychological researchers have long sought to make universal claims about behavior and mental processes. The various crises in psychology—reproducibility, replication, measurement, theory, generalizability—have all demonstrated that such claims are premature, and perhaps impossible using mainstream theoretical and methodological approaches. Both the lack of diversity of samples and simplistic conceptualizations of diversity (e.g., WEIRD, individualism/collectivism) have contributed to an “inference crisis,” in which researchers are ill equipped to make sense of group variation in psychological phenomena, particularly with respect to race/ethnicity. This talk will highlight how the lack of sophisticated frameworks for understanding racial/ethnic differences is a major barrier to developing a reproducible, cumulative psychology.

This article aims to reconceptualize the replication crisis as not merely a problem of flawed methods, lack of scientific rigor, or questionable researcher conduct, but as a fundamentally epistemological and philosophical issue. While improved methodologies and scientific practices are necessary, they must be considered through the lens of the underlying epistemologies. Toward this end, a new paradigm for psychological research and practice, grounded in second-order cybernetics and transactional causality, is proposed as instrumental. Second-order cybernetics, as introduced by Heinz von Foerster, challenges traditional scientific methodologies that assume a strict separation between the observer and the observed. The core idea is that the observer, through the very act of observing, inevitably becomes part of the system they study, leading to a shift from linear to transactional causality. This epistemological shift has profound implications for the research practice and the responsibility of the psychology practitioner. Foerster’s ethical imperative –act always so as to increase the number of choices– combined with the aesthetic imperative –If you desire to see, learn how to act– illuminates an alternative methodological landscape for the clinical practice. The replication crisis in psychology is examined in light of these theoretical shifts, allowing for a new constructive vision which integrates basic research with applied psychological practice. Second-order cybernetics encourages a participatory approach to research, emphasizing the catalyzing role of the observing practitioner. The article concludes by advocating for an epistemological superposition, where psychologists navigate multiple perspectives to enhance the integrity and applicability of their findings in the real world.

Since 2011, the academic field of social psychology has been undergoing a crisis of confidence in its results. This so‐called “replication crisis” has inspired a widespread examination of the practices that can lead to less‐than credible results, the incentives that inspire such practices, and the practices that could protect or enhance research credibility. We believe that the credibility of the field of impact evaluation may be similarly threatened by incentives to use poor research practices. We assess the history of both social psychology and impact evaluation and conclude that, although the source of the incentives for impact evaluators are different in kind than those for social psychologists, impact evaluators may be similarly incentivized to make their results look better than they really are. We review three parallels in the history of the two fields, including the use of research as sales, the pressurized, competitive environments in which the work takes place, and the use of research methods as rhetorical devices. Impact evaluators may be able to learn from the credibility‐enhancing solutions that worked for social psychology, though the field can consider new tools, such as standards, investment mechanisms, administrative data labs, and innovations in evidence‐sharing and synthesis. Rather than sleepwalking through a crisis we cannot see, impact evaluators should consider that their field may be in crisis, a crisis that may require widespread and collective efforts—a credibility renaissance—to discover and deploy methods to protect its legitimacy.

research. The field suffers from high bias, low reproducibility, and widespread misuse of statistics. The replication crisis affects psychology more strongly Metascience, also known as science of science, is the systematic study of science itself. It analyzes the practices, structures, and outcomes of scientific research, including research design, peer review, publication bias, replication, and incentive structures, with the aim of improving the reliability, transparency, and impact of scientific knowledge production. Emerging at the intersection of t The replication crisis is an ongoing methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate. While the crisis has its roots in the meta-research of the mid- to late 20th century, the phrase "replication crisis" was not coined until the early 2010s as part of a growing awareness of the problem. The replication crisis has been closely studied in psychology (especially social psychology) and medicine, including cancer research. Replication is an essential part of the scientific process, and the widespread failure of replication puts into question the reliability of affected fields. Moreover, replication of research (or failure to replicate) is considered less influential than original research, and is less likely to be published in many fields. This discourages the reporting of, and even attempts to replicate, studies. Metascience has revealed significant problems in psychological research. The field suffers from high bias, low reproducibility, and widespread misuse of statistics. The replication crisis affects psychology more strongly than any other field; as many as two-thirds of highly publicized findings may be impossible to replicate. Meta-research finds that 80-95% of psychological studies support their initial hypotheses, which strongly implies the existence of publication bias. The replication crisis has led to renewed efforts to re-test important findings. In response to concerns about publication bias and p-hacking, more than 140 psychology journals have adopted result-blind peer review, in which studies are pre-registered and published without regard for their outcome. An analysis of these reforms estimated that 61 percent of result-blind studies produce null results, in contrast with 5 to 20 percent in earlier research. This analysis shows that result-blind peer review substantially reduces publication bias. Psychologists routinely confuse statistical significance with practical importance, enthusiastically reporting great certainty in unimportant facts. Some psychologists have responded with an increased use of effect size statistics, rather than sole reliance on the p values.

Broadly defined as the study of language, linguistics is a diverse field spanning many disciplines. Recent studies on the prevalence of Questionable Research Practices (QRPs) in linguistics (e.g. Isbell et al. 2022) suggest that it suffers from many of the same issues that triggered the replication crisis in psychology (see e.g. Sönning and Werner 2021). While surveys have indicated that linguists are generally in favour of Open Science/Scholarship (OS), there appears to be a “a misalignment between the attitude to and the adoption of OS practices” (Liu and de Cat 2024, 64). The present study aims to gain insights into this misalignment by exploring linguists’ understanding of what constitutes OS and of the specificities of linguistic research that (can) affect its applicability to (subdisciplines of) linguistics. To this end, the study draws on the results of an anonymous, small-scale survey and the qualitative analysis of semi-structured interviews conducted with 26 linguists based in Northern Europe, representing all career stages, and a wide range of subdisciplines within linguistics. The results reveal diverse understandings of OS among linguists. While some focus on the accessibility of research (for both academics and the wider public), others prioritise the sharing of data, materials, and code to promote transparency, reproducibility, and replicability. The latter group also emphasises the importance of OS principles and values like rigour, fairness, and collaboration

The replication crisis, also known as the reproducibility or replicability crisis, refers to widespread failures to reproduce published scientific results The replication crisis, also known as the reproducibility or replicability crisis, refers to widespread failures to reproduce published scientific results. Because the reproducibility of empirical results is the cornerstone of the scientific method, such failures undermine the credibility of theories and challenge substantial parts of scientific knowledge. Psychology and medicine have been focal p The…