Asking questions

Research on research is based on methods that can be qualitative or quantitative, observational (meta-epidemiological studies) or interventional, but can also use simulation or modelling methods.

Let us take research reproducibility as an example. In a first approach, qualitative or quantitative studies can be used to assess perceptions about data-sharing, which is necessary for the reproducibility of analyses. Another approach is to review studies published in several journals to assess how many of the study’s signatories would agree to share their data. A randomized study could then seek to assess whether a particular incentive increases the rate of data-sharing.

Of course, sharing data is not enough to ensure reproducibility. Another approach is to re-analyze studies that share their data in order to reproduce the results. We can also take the same set of data and have it analyzed by different statisticians to see to what extent their conclusions differ from one analyst to another. You can also vary the different parameters of the statistical analysis to explore how this could affect the result. It is even possible to reproduce certain experiments identically in order to explore whether or not they yield identical results to the original study. This has been done, for example, in the field of cancer research. We can also seek to gain a better understanding of the phenomenon using a modelling approach, for example to explore the links between pressure to publish (the “publish or perish” system) and reproducibility problems. Meta-epidemiological studies can also provide a better understanding of certain biases that limit reproducibility, such as publication bias. It is by triangulating the results of these different approaches that we can finally identify what makes a research result reproducible.

Florian Naudet