Everything is Predictable
A review of Tom Chivers’ book about Bayesianism
One of my biggest pet peeves is the use of a single number to describe the accuracy of a diagnostic test. It’s easy to find statements like “Rapid COVID tests are 90% accurate”, or “Mammograms are 85% accurate”.
I have no idea how to interpret numbers like that.
The most obvious way to interpret it is something like “This is how often the test gives the correct answer”. But even if that were true (it isn’t, it can’t be), that wouldn’t tell you anything about how well the test actually works. To take a trivial example, imagine you have a disease infecting 1% of the population of the USA. I will happily provide a very low-cost test that will be 99% accurate if used across the whole population: for each person, just say “Negative”. It will be right 99% of the time!
A test has two jobs: give a positive when there’s a disease, and a negative when there isn’t. You can separate these out and see how good a test is at each, but that requires two numbers (sensitivity and specificity, which you’ll often find in the fine-print). But even then, these numbers don’t tell us what we really want to know: given a test result, what is the chance I actually have the disease?
That’s what’s called a conditional probability—conditioned on a positive result, what is the probability of the disease? We have a well-known equation for that called Bayes equation, and this little equation is what Tom Chivers’ book is all about.
Bayes’ equation isn’t complicated. It just requires putting together a few numbers to calculate, and it’s intuitive once you understand it. But despite its simplicity, Bayes’ equation is at the heart of discussions of rationality, good scientific practice, and has even been suggested as the primary thing the brain does.
If we have an equation, why is it so hard to tell what your chances of having a disease are, given the result of a test? That’s because you need to know the prior. Imagine that, for fun, you test yourself for the bubonic plague. It comes back positive! What’s more likely: you somehow contracted an incredibly rare disease and are showing no symptoms, or the test is giving a false-positive? This is the case of having a low prior—the chance that you have the disease was low prior to taking the test. A positive test might make it more likely that you do, but on balance it might still be very unlikely.
On the other hand, if you’ve lost your sense of smell, have a headache, fever, trouble breathing, and recently went to a crowded party during the peak of COVID, a negative COVID test is likely to be a false-negative. This would be a high prior.
Bayes’ formula is a way of quantifying this intuition. If you can quantify how likely it is to have the disease prior to the test, you can get a quantification of how you should update your belief after the fact.
Bayesian reasoning is incredibly powerful. It tells us the right way to update probabilities given new evidence. If you see our beliefs as probabilities about the world, then everything becomes a bit like the diagnostic test case, and Bayes is the rational way to update your beliefs.
It’s no wonder, then, that it’s held in such high regard among those that consider themselves rational.
Take, for example, Scott Alexander’s description of his (excellent) publication: it’s Bayes’ equation, followed by “ALL THE REST IS COMMENTARY”. His publication is one of the key venues that gave rise to the Rationalist movement.
I applaud Chivers writing a book trying to make the importance of Bayesian thinking more widely appreciated. I think the vast majority of people would be better off if they understood Bayes better and looked at the world through that lens more.
That said, I found myself a bit peeved at Chivers overstating the case in a few places, calling certain ideas Bayesian because they were kind-of sort-of related. This will probably read as pedantic, and I doubt I would criticize a book for this kind of stuff unless I agreed with most of the rest of it.
For example, there’s a section called “Bayesian Epistemology” where Chivers is making the case that “a Bayesian view on the world [...] dissolves a large number of philosophical conundrums”. What follows is Chivers adopting certain philosophical ideas and then slapping the label Bayesian on them.
Wittgenstein makes a point that “games” don’t have a shared essence, and instead have a family resemblance. Chivers says this is “clearly” Bayesian (page 266). But it isn’t—it’s a claim about how a category is structured, which can be formalized in different ways. Bayesian approaches are one among many options. More generally, Chivers slides from epistemology (our beliefs are graded) to ontology (categories in the world are graded). Plenty of realists are happily Bayesian, holding graded beliefs about sharply defined facts. Once you separate the epistemic from the ontological, all of the philosophical questions Chivers claims can be brushed aside by Bayes come right back.
Similarly, in the discussion of machine learning and AI, Chivers claims machine learning algorithms are “doing something Bayesian. They are using their training data to produce prior probabilities, which they then use to predict future data.”
This isn’t right. Some models are explicitly Bayesian. But many—I would guess most machine learning models being used in the world—are not. Discriminative models, which are extremely common, learn the conditional probability directly and never calculate a prior. Not everything that outputs a probability is Bayesian.
These are relatively small points in the book. I agree with the overall outlook of how important and ubiquitous Bayesian reasoning is, and think Chivers gives this one of the best discussions I’ve seen of how this comes into play in science.
Bayes and Science
In 2011, Cornell psychologist Daryl Bem published a paper claiming that he had found evidence for precognition. He claimed that people could be influenced by a stimulus shown after they had already responded.
Most psychologists didn’t accept Bem’s conclusions. But his methods were very similar to methods used in other areas of psychology, his statistical tests showed significance, and there was nothing obviously wrong with the study.
This event raises two questions: If the standard methods of psychology are leading to wacky conclusions, does that mean there’s an issue with the methods? The decade following the publication of Bem’s study has basically led to an affirmative answer: psychology has been reckoning with the replication crisis. The bulk of the priming literature that Bem was drawing his methods from has been called a train-wreck.
The second question is the one that concerns Bayes: why did scientists happily believe the statistically significant results of other studies, but not this one?
If you’re unfamiliar with statistics, this might be surprising, but the traditional statistical methods used in scientific studies are not aimed at establishing the probability that an effect is real or even probable.
Generally in science, frequentist statistical methods are used, and the explicit test they are doing is: is this different from what I would expect given there being no relationship?
If you are developing a drug to treat an illness, you give a drug to one group, and a sugar pill to another group, and then at the end see that 45% of the drug group and 42% of the placebo group have gotten better after a week. Does the drug work? Hard to say—so you do a statistical test to see if the difference between the groups is larger than you would expect by chance.
What you’re technically doing is seeing: what is the probability of this data given the null hypothesis, the null hypothesis being that there’s no difference between the drug and placebo. By convention, if the chance is less than 5% (that is, the p-value is less than .05), we say it’s “statistically significant”. This is the typical bar for publishing a scientific result.
But notice what we’ve done here—we’ve set the bar such that if there was no effect, we would get a false-positive 5% of the time. This is just like the diagnostic test example above. Without a prior, we can’t transform this 5% false-positive rate into the probability that the drug is effective, just like knowing the false-positive rate of a diagnostic test can’t tell you the probability you have a disease. You need a prior.
So looking at Bem’s results, we can ask: why would someone accept other scientific findings and not his? Because the other findings had a higher prior.
This gets to the heart of the promise and controversy around Bayesian methods. Technically, they’re the way to answer the question we care about: What is the chance an effect is real? But to do so, we have to specify a prior, and many scientists worry about how subjective the choice of a prior is.
Chivers quotes defenders of the current approach who say that we shouldn’t bake priors into the scientific process—we should err on the side of publishing and let individuals make their decisions on how likely a hypothesis is to be true based on the outcome.
The issue is that people generally take statistically significant results to mean an effect is real. Hence, Bem’s paper causing so much controversy.
While this whole prior business is important, and conceptually necessary to interpret scientific papers properly, I’m not sure much hinges on this debate. Of all the potential changes one could make to the statistical practices in science, this one would be pretty big and not solve any major problems.
Bem’s study was just one of many using similar methodologies that got significant results. If you asked other researchers prior to the replication crisis, they would have thought the more mainstream priming results were solid. There were a bunch of studies showing similar things. The problem with the studies was they were the result of questionable research practices—things like doing a test multiple different related ways to see which way gives you the strongest effect.
It doesn’t matter whether you’re using frequentist methods or Bayesian, if you are systematically distorting the probabilities by searching for the relationship that gives a stronger result, the conclusions are going to be distorted.
To me, pre-registration and a focus on effect size are the more important issues. I do think Bayesian methods fit better with these ideals, but I would settle for stronger pre-registration norms, which would get the “null results” out in the open, and refocus journal editors on strong methodology and theory-driven analyses. Ad hoc exploratory analyses are still valuable, but we should know when that’s what’s being done (so we can adjust our priors).
Bayes brain
One interesting idea that the book covers is the Bayesian brain hypothesis: the idea that the best way to understand the brain is as a Bayesian inference engine. This is really a family of different theories rather than one claim, and I think they’ve been very fruitful in many ways but overhyped in others.
The remainder of this post is a bonus for paid subscribers. If you want to support Cognitive Wonderland and/or join the Discord community and get access to paywalled content, consider becoming a paid subscriber.
