Is this a good idea to preregister a complex, partly explorative study with an innovative analytical technique?

Open Research objectives/practices

In our project, we adhere to the following Open Research objectives and practices:

• Using online tools and services to increase the transparency of research processes and methodologies (i.e., OSF).
• Making scientific research more reproducible by increasing the amount and quality of information placed on the public record.
• Creating a public pre-registration of a study design.

Introduction

Preregistering a study prior to conducting it is vital for truthful reporting of the results and avoiding (un)intentional deviation from the study aims as a post-hoc response to the obtained results. It helps formulate clear hypotheses and plan appropriate analytic methods to confirm or reject those hypotheses. This practice can also be extremely valuable for other researchers who might plan to replicate our study or use our proposed methodology.

Motivation

My experience with open science practices started a few years ago with publishing my articles in open-access journals and going through an open peer-review process (being a reviewer as well as a reviewee), including publishing a preprint of my research before its publication. However, my team and I recently decided to take a step forward and explore another open science practice of preregistering a study (including hypotheses and analytical methods) before conducting a study. In the project that we decided to preregister, we planned to address highly innovative research questions that, due to their novelty, lacked established hypotheses supported by the literature. Hence, we had to base our hypotheses on available but slightly distant research. We also chose to use a novel statistical method that has previously only been used on simulated data and not real data from patients. However, we believed this method would be theoretically most likely to work. Due to our study’s complex nature and a mix of exploratory and confirmatory research questions, we saw the preregistration as an opportunity to carefully think through different stages of our study and analysis steps in particular.

Lessons learned

On the positive side, this preregistration helped us have a clear analytical plan with planned steps, a backup plan, established hypotheses (although still somewhat exploratory), and outcome variables. This prompted us to think about certain aspects of the study, such as backup plans, that we would not have thought about prior to conducting a study.

However, there were a few worth-mentioning challenges. First, although the available preregistration forms were clear and straightforward, initially, it was challenging to find an appropriate preregistration form on the OSF website that contained all relevant fields and was suitable for our study design. Second, it took a surprisingly long time to address all questions in the preregistration form since the data were already collected by other researchers but have not been used before. Third challenge arose when we discovered that our proposed method could not answer two out of five planned research questions due to the nature of the data. Since the whole study was meticulously preregistered, we were very restricted in how we could overcome this situation. Although we had a backup statistical analysis method, it concerned the later stages of the analysis. As a result, we had to prepare a document detailing all occurred deviations from preregistration and upload it on the OSF website along with our initial preregistration. This hampered the study due to its unanticipated additional workload to solve this situation. Despite this challenge, I believe preregistration has added value to our project and increased its transparency and reproducibility.

In conclusion, even though extra-challenging at times, my experience with open research practices was primarily positive. I still believe in the importance of open practices and plan to continue adhering to them.

URLs, references and further information

https://osf.io/f45qg - Preregistration
https://osf.io/xr24j - Deviations from the preregistration