We’ve talked many times before, and at greater length in Uncertanity, about the concept of falsifiability. It has come up again lately.
The term shouldn’t be but is equivocal. I mean it is used in an equivocal sense, which is always a danger, because it happens that an argument is started with one sense of the word, but finished with another, invalidating the argument.
The first, and what I say should be the only, is the logical interpretation. To falsify is to prove something false. To prove is to demonstrate by a sound, valid argument. Thus if a man holds the proposition “7 is less than 4”, we can falsify that proposition with a simple proof which takes as its premises basic arithmetic. Indeed, for every proof we have (in math, logic, philosophy) we have also proved a contrary is false. Falsifications are thus not rare—but they are restricted, as we shall see.
Theories make predictions, or rather it is possible to infer predictions from theories. If a theory says, “X is impossible” and X is observed, then the theory has been falsified, i.e. proved false. We here take impossible in its strictest sense.
But if the theory merely says, “X is very unlikely” and X happens, then the theory has not been falsified. Not according to the logical definition of the word. Rare things will happen. The theory admitted this rare thing was not an impossibility; it admitted that the rare thing was possible. The rare thing did happen. Therefore, if anything, the theory has been given support!
Falsification was, as we all know, made popular by Karl Popper. He relied upon it for both philosophical and practical reasons. About the latter, he was frustrated by certain ridiculous theories that could not, by the use of the common tools of the day, be shown to be false. Every observation, said the believers in these queer theories, was consistent with their cherished models.
Take the theory that UFOs have visited earth. The lack of clear or unambiguous photographs means that the UFOs are clever in evading cameras. That NASA denies the existence of UFOs shows the government is in on the conspiracy. And so on. There is no way to falsify the theory that UFOs are real.
Indeed, there does not exist a definitive proof, in the logical sense. Such a proof would have to demonstrate both why other spaceship-building lifeforms could not exist or that even if they could, they could not have got to earth. These demonstrations cannot say such feats are unlikely; they have to say they are impossible. And that’s not possible. UFOs therefore cannot be falsified.
True theories also cannot be falsified; and they have the happy bonus of being right about everything. That is, every observation also supports true theories. No observation anywhere falsifies the theory “7 > 4”.
Enter the second, and incorrect, use of falsified. People will say that the UFO theory has been “practically falsified”, because no evidence of little green men has ever withstood scrutiny. These people are right about about the lack of evidence, and even right according to my lights about their judgement not to take UFOs seriously. But they are wrong to say that the theory has been falsified, practically or not.
Practically falsified is to falsified as practically a virgin is to virgin. They are not in the same philosophical ballpark.
Back to our rare X. A theory has said X is rare, but not impossible. X is seen. Depending on the rarity, for rarity is a loose word with many interpretations, people will incorrectly say the theory has been “practically falsified”. Suppose it was judged Pr(X|theory) = ε > 0, and X is observed. If ε is “small enough”, the criterion for “practical falsification” has been met. Yet once people allow themselves the phrase “practically falsified”, they generously given themselves permission to strip off the qualifier and say just plain “falsified”.
If this doesn’t sound familiar, then you have not been paying attention, dear reader. For this is the basis of the use of p-values — where ε stands in for the magic number. As some of us recall, it was Fisher’s intent to follow Popper’s lead here.
Now it may be the case that every other theory that we know also says X is rare; in notation Pr(X|other theories) = rare. Or we may not know of any other theory; or the theories we know do not allow us to compute a numerical probability.
In the logical sense, all we can say about X is that, perhaps, some theories give it higher probabilities than others. It is natural also to think that the theory which said X was more likely is itself more useful, which it might very well be. But that depends on the use of theory, and how the probabilities affect our decisions. This is why I said I agree about the decision to treat UFOs as unimportant (mistakes or hoaxes etc.), and why I did not say UFOs are false.
We are on the familiar ground of the predictive method, where we recognize that probability is not decision, and understanding how a theory can be useful to one man, but useless to a second, and so forth. See the on-going class for details.
This leads to the conclusion that “practical falsification” is an act, a decision, and not a demonstration. Nothing has been proved with “practical falsification”; something has been judged. “Practical falsification” is in this sense an opinion. In statistics, p-values are a one-size-fits-all decision, because the magic number is, well, magic. One theory is judged false (the “null”), while another has been judged true (the “alternate”).
Wait. Not judged true. Judged “not falsified”. Popper could never bring himself to believe anything; all theories were to him temporary and waiting to be supplanted. That accounts for the screwy “not falsified”, which isn’t wrong, but it is odd. See Uncertainty for more details on this.