The Paper That Broke Psychology Was Done Correctly. That Was the Problem.

TL;DR: In 2011 a Cornell psychologist published nine experiments and more than a thousand participants in the Journal of Personality and Social Psychology, one of the most respected journals in the field, and reported that people could feel the future. The headline number was 52.6 percent against a 50 percent chance baseline. By September 2013 there were 69 replication attempts from 33 laboratories in 14 countries. The result did not hold. What happened next was not a fight about psychics.

Higgypop went back to the Bem experiments this weekend, and it is worth going with them, because almost everybody who tells this story tells the wrong half of it.

Daryl Bem was not a fringe figure. He was a Cornell social psychologist with decades of standing, and what he did in Feeling the Future was take nine standard psychology protocols and run them backwards. The famous one reverses a mere-exposure test: a participant guesses which of two curtains hides an image, and the computer only decides where to put the image after the guess is locked in. Getting it right more often than chance means the guess was informed by something that had not happened yet.

He reported 52.6 percent on the negative stimuli. Chance is 50.

That is the whole edge. Two and a half points, spread across more than a thousand people. Nobody was levitating anything.

Then came the part nobody expected. It got through peer review at JPSP, which is not a place that publishes anything on a whim, and the reviewers were not asleep. They read a paper that used accepted methods, reported accepted statistics, and cleared the accepted significance bar. There was no obvious hole. That was the alarming part.

Richard Wiseman and colleagues ran it again and got nothing. They sent the failure to the same journal. The editor declined to send it out for review at all, on the standing logic that the journal did not publish replications. So the impossible finding was in the literature and the correction was not.

By late 2013 the count was 69 attempts, 33 labs, 14 countries. The effect did not survive contact with them.

Here is why I keep this one in the file.

Every argument that broke out afterward was about the machinery, not the ghost. Critics went at optional stopping, which is peeking at your data as it comes in and deciding when to quit collecting. They went at trial splitting, at analysis choices made after the numbers were already visible, at the way a study with enough small forks in it can find significance in almost anything. Wagenmakers and colleagues came at the statistics directly. James Alcock argued the whole result was procedure and not perception.

Not one of those criticisms is about precognition. Every one of them applies with equal force to a paper about persuasion, or memory, or whether standing in a certain posture changes your hormones.

That is what Bem actually demonstrated. He took the field’s standard toolkit, pointed it at something the field was certain could not exist, and the toolkit produced a publishable positive result anyway. The tools were the finding. Whether or not anybody can feel the future, the machine that decides what psychology knows had just been shown returning a confident yes on demand.

The replication crisis has a lot of parents. Bem is the one who put the problem in a form nobody could file under someone else’s mistake, because the mistake was procedure, and everybody was using it.

Two things I hold on to from this.

The first is that being wrong in an ordinary way is more dangerous than being wrong in a spectacular one. A crank with a bent spoon gets laughed at and forgotten inside a week. A careful scientist with a clean protocol and a 2.6 point edge takes a decade out of a discipline, and the discipline comes out of it different.

The second is about where the door swings. The people who most wanted Bem to be wrong ended up doing the work that showed the whole field was standing on the same soft ground he was. They were not defending psychology from psychics. They ended up finding out what their own instruments did when you asked them a question with a known answer.

The instruments said yes. That is the part I cannot put down.

Sources

• Higgypop: Dr. Daryl Bem’s ESP experiment
• Psi Encyclopedia: Feeling the Future precognition experiments
• Wagenmakers et al., on Bem’s Feeling the Future