How to crack pre-registration: There are methods for camouflaging a registered study as successful

How to crack pre-registration: Toward transparent and open science. Yuki Yamada. Front. Psychol. | doi: 10.3389/fpsyg.2018.01831

The reproducibility problem that exists in various academic fields has been discussed in recent years, and it has been revealed that scientists discreetly engage in several questionable research practices (QRPs). For example, the practice of hypothesizing after the results are known (HARKing) involves the reconstruction of hypotheses and stories after results have been obtained (Kerr, 1998) and thereby promotes the retrospective fabrication of favorable hypotheses (cf. Bem, 2004). P-hacking encompasses various untruthful manipulations for obtaining p-values less than 0.05 (Simmons, Nelson, & Simonsohn, 2011). Such unethical practices dramatically increase the number of false positive findings and thereby encourage the intentional fabrication of evidence as the basis of scientific knowledge and theory, which leads to individual profits for researchers.

Misuse of pre-registration

The preceding paragraphs provide a narrative about QRPs that can be effectively discouraged by pre-registration. However, a detailed examination of the current pre-registration system also reveals problems that this system cannot address. As mentioned, recognition of the value of pre-registration with respect to being able to confer reliability on research findings is becoming increasingly widespread. In terms of reputation management, researchers are motivated to improve their reputation regarding the credibility of their research (and themselves). A subset of researchers may attempt to misuse the pre-registration process to enhance their reputation even if their personality characteristics are not associated with readily engaging in data fabrication or falsification. Alternatively, certain situations may cause normal researchers to misuse this process on a momentary impulse (Motyl et al., 2017; Schoenherr, 2015). Their goals are to enhance the credibility of their research by pre-registering and to show the excellence of their hypothesis by presenting data that support that hypothesis.

There are methods for camouflaging a registered study as successful (van 't Veer & Giner-Sorolla, 2016). One such method is selective reporting, which is a type of data fabrication in which data that do not support the hypothesis are not reported (Goodman, Fanelli, & Ioannidis, 2016). Similarly, in the case of infinite re-experimenting, malicious researchers repeatedly perform the same experiment multiple times until the desired data to support the hypothesis are obtained and then report these data. Such QRPs cannot be completely prevented unless third parties can manage all of the data from experiments performed by researchers following registration. There is also a method that I call overissuing. Researchers who engage in overissuing pre-register a large number of experiments with extremely similar conditions and ultimately report only successful studies. This practice is difficult to discover by reviewers and editors who do not know a researcher’s overall registration status; to date, this approach has not been explicitly identified as a QRP.

Another method is an approach that I call pre-registering after the results are known (PARKing). Researchers engaging in this practice complete an experiment (possibly with infinite re-experimentation) before pre-registering and write an introduction that conforms to their previously obtained results. Because such researchers apparently get attractive results and misrepresent those results as having been obtained under pre-registration, the research can readily acquire false credibility and impact. Rigorous initial peer-reviews that require revision of protocols may be able to reduce PARKing to some extent, but it is not effective if the malicious researchers involved engage in overissuing or target journals with poor peer-review practices. Furthermore, even if all unprocessed data are shared in a repository, the time stamps of uploaded data files can easily be forged or tampered with in various ways, such as by changing the system date for the operating system that is handling the data file. Therefore, there is currently no method for journals or reviewers to detect PARKing. Because many research resources would be required to implement the unethical methods described above, given the discarding of data that do not fit researchers’ hypotheses, such methods can most easily be implemented by laboratories with abundant funds. If the aforementioned QRPs become rampant, their use could not only avoid decreases in false positives (which is a substantial advantage of pre-registration) but also accelerate the Matthew effect of rich people becoming richer (Merton, 1968).

It is easier to fabricate data and falsify results than to engage in cracking pre-registration; therefore, why should researchers attempt to crack pre-registration at all? The answer depends on the associated risk. Because data fabrication is a clear case of research misconduct and is subject to punishment, the risk associated with revelation is large. On the other hand, many of the cracking methods introduced here can be performed by simply extending general research practices. For example, suppose that a researcher conducted a paper-based questionnaire survey in the typical manner (without pre-registration) and had obtained significant results that supported his/her hypothesis and written a manuscript about this research. In this case, barriers to PARKing by using the introduction and method sections of the manuscript and subsequently publishing the full article appear to be low. Excel files for data aggregation can be recreated after pre-registration. If such cracking techniques have benefits that outweigh the difficulties and can be used with little risk, researchers who engage in these techniques will readily emerge.