Every year I go through a section in Moore and Parker’s classic Critical Thinking book on medical testing. Until now, it has been a fairly abstract thing for my students, but now medical testing is a critical part of responding rationally to the pandemic. One type of test is to determine whether a person is infected. Another is to determine whether a person had the infection. While these tests are a critical source of information it is important to be aware of the limitations of testing. Since I am not a medical expert, I will not comment on the accuracy of specific methods of testing. Instead, I will look at applying critical thinking to testing.
An ideal medical test would always be accurate and never yield a false positive. Real medical tests have, for various reasons, less than 100% accuracy in the field and a good test will fall into the 90-99% range. This means that a test can falsely show that a person is or was infected or falsely show they are or were not infected. So how do you judge whether a person is infected or was infected based on a test result?
Intuitively, the chance a person was infected (or not) would seem to be the same as the accuracy of the test. For example, if a COVID-19 test has an accuracy of 90%, then the seemingly rational inference would be that if you test negative, then there is a 90% chance you did not have COVID. Or, if you test positive. There is a 90% chance you had COVID. While this seems sensible, it is not accurate and involves a confusion about conditional probabilities. I will keep the math to a minimum because math, as Barbie said, is hard.
So, suppose that I test positive for having COVID and the test is 90% accurate. If I think there is a 90% chance, I had COVID, I am probably wrong and here is why. The mistake I would be making is failing to recognize that the probability that X given Y is distinct from the probability of Y given X. In the case of the test for COVID, testing positive is the effect of COVID and obviously not the cause. As such, a 90% accurate test for COVID does not mean that 90% of those who test positive (effect) will have had COVID (cause). It means that 90% of those who had COVID (cause) will test positive (effect). So, if I have COVID, then there is a 90% chance the test will detect it. The wrong way of looking at it would be to think that if I test positive, then there is a 90% chance I had COVID. So, what is the true chance I had COVID if I test positive on a test that is 90% accurate? The answer, right now, is that I do not know. But I do know how to do the math to sort it out.
To know my chance of having COVID I would also need to know the percentage of false positives that occur with the test and, very importantly, the base rate of the infection. The base rate of the infection is the frequency of the cause. Using my made-up test and some made-up numbers, here is how the math would go.
Suppose that the 90% accurate test has a 10% false positive rate and 1% of the population in question is infected. For every 1,000 people in the population:
- 10 people will have COVID
- 9 of the people with COVID will test positive.
- 990 people will not have COVID.
- 99 of the people without COVID will test positive.
While there will be 108 positive test results, only 9 of them will have had COVID. So, a person who tests positive has an 8% of having had COVID, not 90%. In conditional terms and using these made-up numbers, if I had COVID, then there is a 90% chance I will test positive. But If I test positive, then there is an 8% chance I had COVID. The same math holds true for antibody testing to see if I had COVID.
At this point is tempting to think that testing is useless, but that would be a mistake. Testing is obviously useful in gathering data about infection rates. Testing is more likely to be right in populations with higher rates of COVID infections—this is a function of statistics rather than testing. To illustrate this, let us run the example again with one change—increasing the rate of infection to 10%. For every 1,000 people in the population:
- 100 people will have COVID
- 90 of the people with COVID will test positive.
- 900 people will not have COVID.
- 90 of the people without COVID will test positive.
There will be 180 positive test results and 50% of them will have COVID. So, if I test positive for COVID, then there is a 50% chance I had COVID. Again, this is a matter of statistics—the test accuracy, by hypothesis, has not changed. Because of this, testing groups that we know to have higher infection rates will give better statistical results that can be useful—but much of the use will be in terms of additional statistical analysis. NPR has provided an excellent discussion of antibody testing for COVID and they even included a calculator that will do the math for you.
In terms of putting your trust in a test, such as an antibody test to determine whether you had COVID or not, it is wise to keep the math in mind. Even if surviving COVID confers some immunity, a positive test might mean an 8% chance that you had COVID. And until we know the rate of infection, we are essentially guessing when doing the math. At this point the rational approach might seem to be a bit odd: assume you have COVID while also assuming that you have not had COVID.