Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis

Starns, J. J., Cataldo, A. M., Rotello, C. M., Annis, J., Aschenbrenner, A., Broder, A., Cox, G., Criss, A., Curl, R. A., Dobbins, I. G., Dunn, J., Enam, T., Evans, N., Farrell, S., Fraundorf, S. H., Gronlund, S. D., Heathcote, A., Heck, D. W., Hicks, J. L., Huff, M. J., Kellen, D., Key, K. N., Kilic, A., Klauer, C., Kraemer, K. R., Leite, F. P., Lloyd, M. E., Malejka, S., Mason, A., McAdoo, R. M., McDonough, I. M., Michael, R., Mickes, L., Mizrak, E., Morgan, D. P., Mueller, S. T., Osth, A., Reynolds, A., Seale-Carlisle, T. M., Singmann, H., Sloane, J. F., Smith, A., Tillman, G., van Ravenzwaaij, D., Weidemann, C. T., Wells, G., White, C. N. & Wilson, J., 1-Dec-2019, In : Advances in Methods and Practices in Psychological Science. 2, p. 335-349 15 p.

Research output: Contribution to journalArticleAcademicpeer-review

Copy link to clipboard


  • Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis

    Final publisher's version, 588 KB, PDF document

    Request copy



  • J. J. Starns
  • A. M. Cataldo
  • C. M. Rotello
  • J. Annis
  • A. Aschenbrenner
  • A. Broder
  • G. Cox
  • A. Criss
  • R. A. Curl
  • I. G. Dobbins
  • J. Dunn
  • T. Enam
  • N. Evans
  • S. Farrell
  • S. H. Fraundorf
  • S. D. Gronlund
  • A. Heathcote
  • D. W. Heck
  • J. L. Hicks
  • M. J. Huff
  • D. Kellen
  • K. N. Key
  • A. Kilic
  • C. Klauer
  • K. R. Kraemer
  • F. P. Leite
  • M. E. Lloyd
  • S. Malejka
  • A. Mason
  • R. M. McAdoo
  • I. M. McDonough
  • R. Michael
  • L. Mickes
  • E. Mizrak
  • D. P. Morgan
  • S. T. Mueller
  • A. Osth
  • A. Reynolds
  • T. M. Seale-Carlisle
  • H. Singmann
  • J. F. Sloane
  • A. Smith
  • G. Tillman
  • Don van Ravenzwaaij
  • C. T. Weidemann
  • G. Wells
  • C. N. White
  • J. Wilson
Scientific advances across a range of disciplines hinge on the ability to make inferences about unobservable theoretical entities on the basis of empirical data patterns. Accurate inferences rely on both discovering valid, replicable data patterns and accurately interpreting those patterns in terms of their implications for theoretical constructs. The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted to the validity of inferences based on those findings. Using an example from cognitive psychology, we demonstrate a blinded-inference paradigm for assessing the quality of theoretical inferences from data. Our results reveal substantial variability in experts’ judgments on the very same data, hinting at a possible inference crisis.
Original languageEnglish
Pages (from-to)335-349
Number of pages15
JournalAdvances in Methods and Practices in Psychological Science
Publication statusPublished - 1-Dec-2019

ID: 112163172