PhD on Track » Open Science » Preregistration

Preregistration

picture - research data

Preregistration involves specifying your hypotheses, study design and data analyses before writing up your final report. Sometimes, preregistration takes place before any data are collected, while in other cases (when using pre-existing data), it takes place before the data are analysed. Preregistration is typically done in a time-stamped, non-editable file, which is then deposited in a secure online archive. While not yet equally relevant in all disciplines or to all types of study, the practice of preregistration is currently expanding.

” (…) preregistration combats the natural human inclination to craft a post hoc story (…)” (Gehlbach & Robinson, 2018, p. 303).

On this page you will find information about

  • why preregistration may be a good idea
  • the two types of preregistration
  • what preregistration involves
  • where to submit your preregistration

Why preregister studies?

Preregistering studies, like other open science practices, contributes to the transparency of science and to the reliability and reproducibility of scientific results. Preregistration helps prevent certain research practices that threaten the integrity of the scientific process. Such practices include presenting flexible, post-hoc analyses as if they had been planned (Lindsay, Simons & Lilienfeld, 2016), selective outcome reporting or “outcome switching”, selective study reporting (contributing to publication bias), and hypothesising after results are known (“HARKing”). Such questionable research practices are often engaged in without malicious intent, and are partly a result of the influence of unhealthy incentive structures (Munafó et al., 2017), our fallible memories, and the mostly unconscious operation of universal psychological biases.

Pre-registration may also improve study design and data analyses. The very act of writing a detailed protocol or answering the questions involved in a preregistration may lead to the discovery of flaws and thus opportunities to improve the study. This may help prevent mistakes that would be costly or hard to correct.

While preregistration already has a decades-long history in medical research, it is still a phenomenon that is just starting to gain traction in other disciplines. At present, you should carefully consider preregistering if you are doing research in the medical and health sciences, neuroscience or psychology, or other life sciences. We predict, however, that preregistration may become more common throughout the sciences in the years to come, and in principle there is no reason not to preregister your study, even if you work in a field where this practice is not yet common. Further, while the benefits of preregistration are probably most notable for studies that purport to test hypotheses with some degree of methodological rigour, it may still be a good idea to preregister a well-planned exploratory study. Note also that not all parts of a study need to be be specified in advance. Sometimes, parts of a study may be exploratory (as opposed to testing pre-specified hypotheses).

Questionable research practices

Below is a selected list of questionable research practices and links to more in-depth treatments of each:

  • HARKing (also known as “hypothesis accommodation”). Altering hypotheses or generating new ones after analysing data, and reporting as if the study was designed to test those hypotheses (Rubin, 2017).
  • p-hacking (also known as “data dredging”). Adjusting statistical inference analyses (e.g. trying many different unplanned analyses), increasing the likelihood of reporting statistically significant results (Head, Holman, Lanfear, Kahn & Jennions, 2015).
  • Outcome switching: Not reporting pre-specified outcomes or introducing new outcomes that were not pre-specified (Altman, Moher & Schultz, 2017).
  • Publication bias (also known as the “file-drawer problem”). Not a practice as such, but an unfortunate consequence of neglecting to publish studies that do not produce positive or novel results (see e.g Franco, Malhotra & Simonovits, 2014).


Arguably, another very important benefit of preregistering is that it allows you to claim credit for having thought up the idea of the study you described in your plan, even if someone beats you to it, as it were, and publishes a final report on a similar study before you have had a chance to submit your own. Note, however, that preregistering confers no formal rights to exclusivity.

Conversely, when planning a study, you may find it useful to search protocol registeres (see below) to inform yourself about similar planned or ongoing studies.

How to preregister your study

The specification involved in a preregistration can be more or less detailed, depending on the intended audience, the service or site in which the protocol is submitted, and the amount of actual preplanning. If there is already a project description or a protocol from an application for funding or for approval by an ethics committee, it may already contain much of the information you need for preregistration. Similarly, the work that goes into writing a data management plan will probably yield many of the required details. Importantly, preregistration may be either reviewed or unreviewed.

Unreviewed preregistration

You can preregister your study plan without involving anyone but your supervisors or you co-researchers. This would then be an unreviewed preregistration. It serves several related purposes. First, readers of your final study report can confirm that your study was conducted according to a plan conceived a priori. Second, they can judge for themselves whether any deviations from the original plan are acceptable. Third, you can refer back to your original plan without ever having to doubt that it was in fact the original plan. Last, but not least, consumers of research in your field, including teams compiling evidence for systematic reviews, will know that there is a study being conducted designed to answer a question they may be seeking answers to. This arguably encourages the dissemination of all study results, even those that may not be favourable to the hypotheses being tested.

In fact, preregistration of clinical trials of interventions and treatments in medical research is now mandatory in the EU (Clinical trials regulation, 2014) precisely in order to help ensure that all findings that may bear on a decision to approve or adopt a treatment are available to decision makers, health professionals, and patients. If you are contributing to the implementation of a clinical trial as part of your PhD, the trial’s principal investigator will be responsible for submitting a study protocol to an approved trials register (see below). Registering study protocols in trials registers is usually considered a form of unreviewed registration, even though protocols for such studies are carefully scrutinized by the bodies assigned responsibility to approve trials (in Norway these are the Regional Committees for Medical and Health Research Ethics).

Note that a preregistered protocol does not always have to be made public at the outset. You can embargo it to prevent anyone from knowing the details of your study before you have written it up. You can still prove by the timestamp that you were first to have pre-registered it, and other researchers can still compare your final study to your preregistered plan, once you lift the embargo.

Reviewed preregistration

Reviewed preregistration confers all the benefits of unreviewed preregistration, but also allows for expert feedback on the study protocol before it is finalized. This form of preregistration is offered by journals, and is commonly referred to as registered reports. Submitting a registered report is a lengthy process, often requiring several rounds of revision. If the study plan passes review, the journal editor promises to publish the final report, regardless of the outcome of the study, provided the study is conducted according to the reviewed study plan. While such provisional or “in principle acceptance” is clearly advantageous, the lengthy review process and the amount of work it entails should be carefully considered when planning your PhD. You can read more about registered reports at the Open Science Framework.

Where to preregister?

Here is a selection of sites or journals that allow you to preregister your study by submitting a time-stamped research plan, or that offer registered reports.

  • The Open Science Framework by the Center for Open Science offers what is perhaps best described as a suite of project management tools, supporting collaboration and open science practices, including preregistration.
  • AsPredicted offers what is perhaps the easiest way to achieve preregistration, by filling in an online form with nine questions for the researcher. Of course, the value of such preregistration depends on the quality of the planning that lies behind those answers. All types of studies from all disciplines are welcome to register at AsPredicted.
  • Prospero is a database of protocols for systematic reviews of studies with health related outcomes. It offers easy registration via an online form, and also allows you to search already registered protocols. Prospero is operated by the University of York Centre for Reviews and Dissemination.
  • On the Center for Open Science’s Registered Reports Wiki, you will find a continuously updated list of journals offering this form of reviewed preregistration.

Clinical trials registers

While you are unlikely to be responsible for registering a clinical trial, such trials registers may still be of interest. Below is a selection.

Deviating from preregistered plans

While preregistration is partly intended to reduce unbridled flexibility in study implementation, it is not meant to irrevocably cement or lock all aspects of a planned study. Even the best laid plans may turn out to be harder to implement than predicted, and it may become necessary to deviate from your preregistered study plan. This does not in itself invalidate your results or conclusion, but it is important to be explicit about how your final study implementation differs from your protocol. This will allow reviewers and other readers to evaluate whether your changes are acceptable or introduce bias. Arguably, preregistration leads to more thorough and principled considerations of whether or not deviations are strictly necessary, and to better justifications for any changes decided on.

References

Altman, D. G., Moher, D., & Schulz, K. F. (2017). Harms of outcome switching in reports of randomised trials: CONSORT perspective. British Medical Journal, 356, 1-4. https://doi.org/10.1136/bmj.j396

Clinical Trials Regulation. (2014). Regulation (EU) No 536/2014 of the European Parliament and of the Council on clinical trials on medicinal products for human use. Retrieved from https://ec.europa.eu/health/human-use/clinical-trials/regulation_en

Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345, 1502-1505. https://doi.org/10.1126/science.1255484

Gehlbach, H., & Robinson, C. D. (2018). Mitigating illusory results through preregistration in education. Journal of Research on Educational Effectiveness, 11, 296-315. https://doi.org/10.1080/19345747.2017.1387950

Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13(3), 1-15. https://doi.org/10.1371/journal.pbio.1002106

Lindsay, D. S., Simons, D. J., & Lilienfeld, S. O. (2016). Research preregistration 101. APS Observer, 29(10), 14-17.

Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., . . . Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 1-9. https://doi.org/10.1038/s41562-016-0021

Rubin, M. (2017). When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress. Review of General Psychology, 21, 308-320. https://doi.org/10.1037/gpr0000128

van ‘t Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology: A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2-12. https://doi.org/10.1016/j.jesp.2016.03.004

A s k -u s