Large-scale study finds conclusions often diverge when hundreds of researchers reanalyze the same data

When nearly 500 independent analysts reexamine the same data from 100 studies, they frequently reach different conclusions—revealing how much methodological choices shape scientific findings.

A new study published in Nature, “Investigating the analytical robustness of the social and behavioral sciences,” finds that scientific conclusions can shift dramatically depending on who conducts the analysis.

The results come from a large-scale international collaboration. A team of 457 independent analysts from institutions around the world conducted 504 reanalyses of data from 100 previously published studies across the social and behavioral sciences. All analysts received the same dataset and the same key research question, but were given freedom in how to conduct the analysis based on their informed judgment.

Analyses of the exact same question using the same data turned out to yield divergent results. Although most reanalyses broadly supported the main claims of the original studies, effect sizes, statistical estimates, and levels of uncertainty often differed meaningfully. All analysts reached the same conclusion as the original authors in about one third of cases.

Importantly, these discrepancies were not due to a lack of expertise. Experienced researchers with strong statistical backgrounds were just as likely to arrive at divergent results as others. At the same time, observational studies proved less robust than experimental ones, suggesting that more complex data structures allow greater analytical flexibility—and thus greater uncertainty.

Don van Ravenzwaaij, affiliated with the Faculty of Behavioural and Social Sciences: “The results suggest that we may need to view unsuccessful replications with more nuance: if there is already so much variation within a single study due to the analyst or analytical strategy, then variation between studies is both logical and to be expected.”