A theory with an emergency exit

A convincing theory is especially pleasant for a society: it explains everything and leaves little room for doubt. But being able to explain what has already happened does not yet make a theory a good one — with enough ingenuity a justification can be found for almost any outcome.

In Peter Wason’s experiment participants were shown the sequence 2–4–6 and asked to discover the rule by testing triples of numbers of their own. People usually formulated a narrow hypothesis — “each next number is two greater”, for instance — and offered examples that confirmed it. Far more rarely did they try to find a sequence capable of refuting it.

The experimenter’s rule was considerably broader. The task became a classic illustration of how naturally we look for the world to agree with a theory we have already chosen. This is exactly why a good theory needs an emergency exit: you have to know in advance which observation will make you leave it.

The stronger model is the one that states in advance what to expect and under which observations it will have to be abandoned.

For an empirical claim it helps to ask:

What could show that it is wrong?

If no possible event counts as a refutation, the theory is protected from error, but along with error it is protected from testing.

A consultant, for instance, predicts that the market will grow. The market grows — the forecast was confirmed. The market falls — the consultant explains that the hidden tension was so great that it manifested in the opposite direction. The market does not move — that is the accumulation phase. Such a model is always right and therefore almost useless.

Not every meaningful statement is a scientific hypothesis, though. Moral principles, aesthetic judgements and metaphysical convictions can matter even though they cannot be tested by experiment. The error begins when an untestable claim is used as a reliable forecast about the physical world.

A useful theory leaves an emergency exit: an observation after which we admit that the map needs replacing.

Without one, a model turns into a belief in one’s own infallibility, sometimes presented with diagrams.

The domestic version of an untestable theory is a course of vitamins. You feel better — they helped. You do not feel better — you started late or the dose was too small. You fall ill — without the vitamins it would have been worse. Any outcome fits inside, and so the course can never end in failure. A parenting method, a management style and any stock trading system in which every loss is explained by insufficiently strict adherence to the system are built exactly the same way. The construction is always the same: the theory has an explanation for success and a separate explanation for failure, and both are ready in advance.

It helps to ask in advance and out loud: at what result will I say that this did not work. If there is no answer, the experiment is already over. It is just still running.