Abstract
Prereg posters are conference posters that present planned scientific projects. We provide empirical evidence for their value, and establish prereg posters as means to receive valuable feedback, promote Open Science, and support Early Career Researchers.
Introduction
In recent years, it has been repeatedly shown that the results of many scientific studies fail to replicate (Baker, 2015; Munafo, 2017a; Munafo et al., 2017b). The growing awareness of this problem has prompted several attempts to tackle it. Foremost among these is study preregistration (Nosek et al., 2017), in which researchers commit to specific data collection and analysis procedures prior to knowledge of the results. Preregistration helps to prevent the statistical biases that can arise from post hoc inferences, while also providing room for transparent exploratory analyses.
Researchers can preregister their hypotheses, study design and analyses on various online platforms such as the Open Science Framework (https://osf.io/prereg/), though these are not normally peer-reviewed. An alternative is to submit a registered report to a journal, which is peer-reviewed and has the opportunity to be revised, before any data are collected (Chambers, 2013; Chambers et al., 2015; Munafo, 2017; COS Team, 2013). Not only does the peer-review potentially improve the study design and analyses pipeline, but if accepted, the journal is committed to publishing the results, whatever the outcome, which reduces the “file-drawer” problem in which many null results are never published (Head et al., 2015). Registered reports, first offered by the journal Cortex, are now supported by over 200 peer-reviewed journals, including Nature Human Behaviour.
Recently, we proposed a third possible avenue for preregistration, namely posters at academic conferences (Tibon et al., 2018) – so-called “prereg posters”. We suggested that conferences allow prereg posters that present planned work as means to promote preregistration and stimulate discussion throughout the entire lifecycle of the scientific work. We argued that this would allow researchers to receive feedback on early stages of their projects, when such feedback is arguably most valuable (that is, before the protocol is finalised and data are collected). Moreover, colleagues with shared scientific interests would become aware of the study early on, which can open the door to collaborations. We further argued that authors who submit prereg posters would be more likely to use these as the basis of a registered report (amended following conference feedback).
Prereg posters were recently adopted by the BNA2019 Festival of Neuroscience (https://meetings.bna.org.uk/bna2019/), a biannual event organised by the British Neuroscience Association (BNA). Such posters were also allowed in at least two other conferences in 2019: the FLUX meeting, https://fluxsociety.org/2019-new-york/, and BACN meeting, https://www.bacn.co.uk/conferences. The BNA2019 Festival organisers additionally collected survey data about prereg posters, which we report here as empirical evidence for their value. Nearly a fifth (100/491) of all submitted posters at the BNA2019 Festival conformed to the new prereg format, covering a diverse range of neuroscience topics and disciplines, and the overall impression was that they were enthusiastically welcomed. The survey data were collected at two time points (before and after the event), via two different surveys. The full set of questions and response data are available here: https://bit.ly/36CkfLG. Below, we analyse data from the questionnaire items that address three main themes: prereg posters as means to (1) receive valuable feedback, (2) promote Open Science, and (3) support Early Career Researchers (ECRs).
Methods
The first survey (pre-conference survey) was administered three months prior to the event, and offered to all 445 participants whose poster was accepted. Of the 200 who responded, 151 were presenters of traditional posters and 49 of prereg posters. The second survey (post-conference survey) was administered one month after the event, and was completed by 97 participants, 66 of whom were presenters of traditional posters, and 29 of prereg posters. For analysis of ordinal data, we used a two sided two-sample Wilcoxon Rank Sum test (where W refers to the sum of ranks for a sample) and a significance (alpha) level of .05.
Results and Discussion
Theme 1: Prereg posters as means for getting valuable feedback
As a presenter, the motivation to present a prereg poster is clear: it allows one to get feedback on work at early stages. Indeed, 33% of the 123 responses to the pre-conference question “why did you submit a prereg abstract instead of a traditional abstract” indicated that this choice was made in order to get feedback (either in general or on specific analyses) before completing the research. As shown in Figure 1, the groups differed in the type of feedback that they subsequently received at the conference. Not surprisingly, presenters of traditional posters mostly received feedback regarding future projects, whereas presenters of prereg posters mostly received feedback regarding methods and experimental design.
Distribution of responses to the post-conference survey question “what kind of feedback did you receive”, among presenters of prereg posters (left) and traditional posters (right). N represents the total number of responses.
Interestingly, the groups did not differ significantly in their estimation of the likelihood of the presented research being published in a peer-reviewed journal upon completion (prereg: Median = 6 [scale: 1 – “highly unlikely” to 7 – “highly likely”], IQR = 2, W = 944; traditional: Median = 6, IQR = 2, W = 970; p = .91). This is even though projects presented as prereg posters were at their early stages, and in most cases were still at (or prior to) the data collection phase. This might reflect the presenters’ trust in the preregistration system, where publication does not depend on results. But it might also suggest that presenters of prereg posters felt that they have received valuable feedback that would promote their ability to publish their work later on.
One potential objection to prereg posters is that conference attendees may not bother to discuss “half-baked” work and therefore there is no point in “wasting” conference space on posters that would not attract visitors. To test this, we analysed responses to two questions in the post-conference survey: (1) did you receive feedback on your work during the poster session, and (2) how many people talked to you about your poster during the poster session. There was no evidence that prereg posters received less feedback than traditional posters (prereg: Feedback = 75%; W = 1030.5; traditional: Feedback = 68%, W = 883.5; p = .446), or fewer visitors (prereg: Median category = 2 [5-8 people], IQR = 1, W = 958.5; traditional: Median Category = 2 [5-8 people], IQR = 1, W = 955.5; p = .99). These findings provide no evidence that conference attendee avoided discussion of prereg posters, despite the fact that these posters presented planned or preliminary work.
Theme 2: Prereg posters as means to promote Open Science
Our second theme concerns attitudes among the attendees towards Open Science in general, and preregistration in particular. For this purpose, we analysed two questions from the post-conference survey: (1) to what degree would you say that preregistration of work is necessary in today’s neuroscience community, and (2) to what degree did BNA2019 increase your awareness of preregistration. Presenters of both prereg and traditional posters agreed that preregistration is necessary (prereg: Median = 4 [scale: 1 - “not at all” to 5 – “incredibly necessary”], IQR = 1, W = 1199.5; traditional: Median = 4, IQR = 2, W = 714.5), with higher rating by presenters of prereg posters compared to presenters of traditional posters, p = .04. Moreover, participants in both groups indicated that the event increased their awareness of preregistration (prereg: Median = 6 [scale: 1 - “not at all” to 7 – “completely”], IQR = 2, W = 1225; traditional: Median = 5, IQR = 2.75, W = 689), and the increment was greater for prereg poster presenters, p = .03. These results indicate that the attendees of BNA2019 acknowledged the importance of pre-registration and that the event was helpful in promoting this understanding further, particularly for those presenting prereg posters.
Theme 3: Prereg posters as mean to support ECRs
Although all scientists can benefit from their colleagues’ comments, these can be particularly beneficial to ECRs who are still developing their networks and expertise in the field. Because on most occasions (especially for ECRs) travel funding is only available when work is presented, and given that ECRs often have fewer existing datasets to present compared to more senior colleagues, prereg posters can provide ECRs additional opportunities to attend academic events. Therefore, the opportunity to present a prereg poster may be especially compelling for ECRs. To test this, we analysed questions about the presenters’ career stage. In both surveys, participants were asked to indicate (1) how many years of research experience they have, and (2) how many posters they have submitted to conferences before. For both pre- and post-conference survey, research experience for presenters of prereg posters (pre-conference: Median = 1 [1-2 years of research experience], W = 2641.5, IQR = 5; post-conference: Median = 2 [2-3 years], IQR = 3, W = 678.5) was lower than that of traditional poster presenters (pre-conference: Median = 4 [3-4 years], IQR = 3, W = 4757.5; post-conference: Median = 4, IQR = 3, W = 1235.5), pre-conference p = .002; post-conference p = .02. Additionally, analysis of the pre-conference survey revealed that presenters of prereg posters had presented fewer posters throughout their career (Median = 1 [1-3 posters], IQR = 2, W = 1066.5) than traditional poster presenters (Median = 2 [4-7 posters], IQR = 2), W = 4767.5, p = .0017. This effect was numerically similar, though not significant, for the post-conference survey (p = .37).
Distribution (around the median) of the years of research experience (left) and number of presented posters in academic conferences (right) among the presenters of prereg posters (cyan) and traditional posters (amber). Within each plot, boxplots on the left represent data obtained from the pre-conference survey (N=200) and boxplots on the right represent data obtained from the post-conference survey (N=97).
Conclusions
We provide empirical evidence that prereg posters can be a useful tool in promoting academic discussion of planned and on-going research, supporting Open Science and benefiting Early Career Researchers. We hope this encourages their adoption by an increasing number of future scientific conferences. As discussed in Tibon et al (2018), and as with any new initiative, we believe it is vital to instruct all the people involved in conferences (organisers, presenters, poster reviewers and attendees) about the aim and value of these posters, otherwise the initiative might flounder (e.g, if reviewers score prereg posters lower owing to their lack of results, because the reviewers were not properly informed of the aims). Additionally, at least until they become more customary, we recommend highlighting prereg posters in the conference program and at the stand (e.g., by Open Science badges, as done at BNA2019). Further practical advice and useful tips can be found on the BNA website: https://www.bna.org.uk/mediacentre/news/pre-reg-posters/. Importantly, our results demonstrate that conferences and academic events should not only support specific scientific topics, but also act as venues that increase the quality of scientific research.
