Benefits of and doubts about the preregistration procedure
|Date:||09 October 2020|
Introduction and research context
This case study "Publishing a preregistered study in an open access journal. Benefits of and doubts about the preregistration procedure" describes the publication process of a preregistered study in an open access journal, using preexisting data from a larger cohort study. The complete process, from the preparation of the preregistration to acceptance of the manuscript took about 18 months.
The preregistration was prepared by the core research team involved in my PhD trajectory and informed by other colleagues who had experience with preparing preregistrations. A series of talks by early open science adaptors in our department (i.e., the Interdisciplinary Center Psychopathology and Emotion regulation) inspired us to also apply these principles in our own work. In response to a growing discomfort about the lack of reproducible findings in psychology, and the wish to make our analytical procedures more transparent we decided to preregister the background, research questions, hypotheses and analyses for our study.
Challenges and benefits
The biggest challenge of preregistering, for us, was to consider all possible scenarios that could occur and describe which steps we would take in case we would run into any problems. Although this helped in refining our ideas, and forced us to specify several sensitivity analyses that we would run in addition to our main analysis, it could not prevent that we had to deviate from our preregistered plan. Subsequently, the challenge became to describe the deviations in the manuscript, without confusing the reader about what the final analytical procedure had been. We chose to mention the deviations as footnotes to the paper. As such, we could provide clarity about the final procedures, while also being open about what our initial model had been and what forced us to make different choices. Another challenge that we identified while incorporating the preregistration in our manuscript was that there are currently no clear guidelines and there is no room in the submission process to do this properly. We thought this was interesting especially as we submitted the paper to an open access journal that explicitly encouraged researchers to also submit open data. Having a template, or a submission portal in which you can provide more details about a preregistration that accompanies the manuscript would benefit both reviewers during the submission process and readers (eventually).
Because the benefits of a preregistration for the wider scientific community are evident, we would like to focus on what benefits were realized for us as a team. A first benefit is that the preparation of a preregistration enabled early discussions about possible difficulties that we could encounter during the analyses, which helped to identify possible solutions for these difficulties. The latter accelerated the process of decision making when we were faced with model that did not converge using our initial specifications. A second benefit is that (we believe) the preregistration helped us in publishing our null-results. For editors and reviewers it was clear that we had made certain analytical choices prior to our analyses, which we think increased their confidence in the relevance of our findings. Two out of the three reviewers explicitly mentioned that they appreciated the preregistration and thought it showed our methodological rigor.
After overcoming initial difficulties with incorporating deviations from our initial plan in the manuscript, we realized preregistrations have more benefits than doubts. Discussions about analytical choices become more elaborate due to having to conduct them prior to running any analyses, which we believe further improved the quality of our analyses. A preregistration of an existing dataset is feasible, and will improve both the process of conducting the study as a team and may also help you to share your results in a journal, despite those results not providing support for your hypothesis.
References and further information:
- https://osf.io/jwuz9 (preregistration); manuscript is currently in press