Years ago, my coverage of medical testing in my critical thinking class was purely theoretical for most of my students. But COVID-19 changed that. One common type of medical test determines whether a person is infected with a disease, such as COVID-19. Another is to determine whether a person has had the infection. While tests are a critical source of information, we need to be aware of the limitations of testing. Since I am not a medical expert, I will not comment on the accuracy of specific methods of testing. Instead, I will look at applying critical thinking to testing.
An ideal medical test would always be accurate and never yield false results. Real medical tests have, for various reasons, less than 100% accuracy and a good test will usually fall into the 90-99% range. This means that a test can always be wrong. So how do you rationally assesses test results?
Intuitively, the chance a person was infected (or not) would seem to be the same as the accuracy of the test. For example, if a COVID-19 test has an accuracy of 90%, then if you test negative, then it seems that there is a 90% chance you do not have COVID. Or, if you test positive, then you might think there is a 90% chance you have COVID. While this seems sensible, it is not correct and arises from a common mistake about conditional probabilities. I will keep math to a minimum because math, as Barbie said, is hard.
So, suppose that I test positive for COVID and the test is 90% accurate. If I think there is a 90% chance, I have COVID, I am probably wrong and here is why. The mistake is failing to recognize that the probability that X given Y is distinct from the probability of Y given X. In the case of the test, testing positive is the effect of COVID and obviously not the cause. As such, a 90% accurate test for COVID does not mean that 90% of those who test positive (effect) will have COVID (cause). It means that 90% of those who have COVID (cause) will test positive (effect). So, if I have COVID, then there is a 90% chance the test will detect it. The wrong way of looking at it would be to think that if I test positive, then there is a 90% chance I had COVID. So, what is the true chance I have COVID if I test positive on a test that is 90% accurate? The answer is that I do not know. But I do know how to do the math to sort it out.
To know my chance of having COVID I would also need to know the percentage of false positives that occur with the test and, very importantly, the base rate of the infection. The base rate of the infection is the frequency of the cause. Using my made-up test and some made-up numbers, here is how the math would go.
Suppose that the 90% accurate test has a 10% false positive rate and 1% of the population in question is infected. For every 1,000 people in the population:
- 10 people will have COVID
- 9 of the people with COVID will test positive.
- 990 people will not have COVID.
- 99 of the people without COVID will test positive.
While there will be 108 positive test results, only 9 of them will have had COVID. So, a person who tests positive has an 8% of having had COVID, not 90%. In conditional terms and using these made-up numbers, if I have COVID, then there is a 90% chance I will test positive. But If I test positive, then there is an 8% chance I have COVID.
At this point it might be tempting to think that testing is useless, but that would be a mistake. Testing is useful in gathering data about infection rates. Testing is more likely to be accurate in populations with higher rates of COVID infections, but this is a function of statistics rather than a function of testing. To illustrate this, let us run the example again with one change, which is increasing the rate of infection to 10%. For every 1,000 people in the population:
- 100 people will have COVID
- 90 of the people with COVID will test positive.
- 900 people will not have COVID.
- 90 of the people without COVID will test positive.
There will be 180 positive test results and 50% of them will have COVID. So, if I test positive for COVID, then there is a 50% chance I have COVID. Again, this is a matter of statistics as the test accuracy, by hypothesis, has not changed. Because of this, testing groups that we know have higher infection rates will give better statistical results that can be useful—but much of the use will be in terms of additional statistical analysis. NPR provided an excellent discussion of antibody testing for COVID and they even included a calculator that will do the math for you.
In terms of putting your trust in a test, such as an antibody test to determine whether you had COVID or not, it is wise to keep the math in mind. Even if surviving COVID confers some immunity, a positive test might mean an 8% chance that you had COVID. And until we know the rate of an infection, we would essentially be guessing when doing the math. During the pandemic the rational approach might seem odd: you should have assumed you have COVID while also assuming that you have not had COVID. The same will apply to the next pandemic.