“I could’ve told you that” might have a useful role to play in science, Ars Technica

           

Last year, a huge group of researchers collaborated to try to replicate the results of somevery famous social science research. They determined that only 62% of the studies found similar results when they were repeated. But the researchers found something else intriguing: other scientists were astonishingly good at guessing which of the results would replicate.

Does that mean we can just ask scientists for their hunch on what research is robust? It’s a lot more complicated than that, but predictions could have a useful role to play in science, and new projects are springing up to make use of them.

Obvious or not?

The success of scientists’ predictions doesn’t mean that every individual scientist (or non-scientist) can just rely on their gut to guess which scientific results are true. We have a term for that: “confirmation bias.” But that doesn’t mean that scientists can’t have informed, well-founded suspicions about some research. Predictions taken from across a whole group ofscientistscould average out the bias and highlight the well-informed doubts and thus be a very useful contribution to scientific research.

Writing inSciencethis week, economists Stefano DellaVigna, Devin Pope, and Eva Vivalt point out some of the benefits that gathering predictions could have for science.

Right now, we have very little sense of what people think about a topic before a study is conducted and whether they update their beliefs afterward. Everyone is susceptible to hindsight bias, which means that a new study, even if it’s surprising, can easily induce a reaction of “Pfft, I knew that already.” But is that actually true?

Without getting people’s thoughts down before the results are known, it’s difficult to determine whether the results of a study are truly what was expected or whether people have just nudged their beliefs to match up with the new data. Of course, the opposite can be true: people can dig their heels in and refuse to change their beliefs.

Everyone — including scientists — is susceptible to foibles like hindsight bias and confirmation bias. So having data on how predictions and belief updating actually work could be really useful to help researchers understand how the people they most need to convince — like policy-makers and fellow scientists — respond to evidence. And on a truly grand scale, predictions could make it possible to understand how scientific fields shift their consensus over time.

Figuring out which studies are genuinely surprising would also help to put scientific results in their proper context. If they’re properly astonishing, maybe they need a lot more replication and sanity checking. And cold, hard evidence of surprising results could help with publication bias, suggest DellaVigna, Pope, and Vivalt. Journals often refuse to publish results that seem boring and not “surprising” enough — having the forecasts to show how surprising results really are could help researchers jump through this hoop. (The authors don’t question the wisdom of having that hoop in the first place.)

What makes predictions tick

As a range of different forecasting projects pop up, different ones are figuring out distinct ways to get people to provide them and how to make use of predictions .Some projectsuse betting markets, in which people can see other participants ‘guesses and update their own accordingly. DellaVigna and Vivalt’s forecastingplatformkeeps people’s responses private to avoid them Being biased by other people’s opinions.

Getting predictions about research could look very different from one project to the next. Target audiences could differ (scientists, policy-makers, the general public). Researchers might want to ask people for more general predictions, like “Do you think this study will have a significant result?” – or for very specific predictions, like “What do you think the average result will be?”

As more projects start to use forecasting, it should become possible to understand how to make predictions of scientific research more accurate. This could help scientists winnow down their options when they’re figuring out which research avenue could be the most promising.

It could also help researchers figure out why betting markets are good at predicting which work will be robust. A huge prediction project at the University of Melbourne,repliCATS, is doing just that: figuring out how to tap into the instincts of scientists about their peers’ work, establishing what makes scientists trust each other’s studies, and how well-reasoned predictions from large groups of scientists could accurately estimate which research is likely to replicate.

The use of predictions in science is very much in its infancy, but as it grows, its data could become increasingly useful — like a second, meta -dataset on top of the actual research itself.

Science, 2018. DOI:10. 1126 / science.aaz 1704(About DOIs).

        

Listing image byflickr user: anlopelope