In the last essay I looked at the inductive generalization and its usefulness in reasoning about pandemics and ended by mentioning that there are various fallacies that can occur when generalizing. The most common are hasty generalization, appeal to anecdotal evidence, and biased generalization. I will look at each of them in terms of pandemics.
A hasty generalization occurs when a person draws a conclusion about a population based on a sample that is not large enough to adequately support the concussion. It has the following form:
Premise 1: Sample S (which is too small) is taken from population P.
Premise 2: In Sample S X% of the observed A’s are B’s.
Conclusion: X% of all A’s are B’s in Population P.
In the previous essay I presented a rough guide to sample size, margin of error and confidence level. In that context, this fallacy occurs when the sample is not large enough to warrant confidence in the conclusion. In the case of a pandemic, one important generalization involves sorting out the lethality of the pathogen. Math for this is easy but the challenge is getting the right information.
During the COVID-19 pandemic, there were large samples of infected people. As such, inferences from these large samples to the lethality of the virus would not be a hasty generalization. But avoiding this fallacy does not mean that the generalization is a good one as there are other things that can go wrong.
There were also inferences drawn from relatively small samples, such as generalizations from treatments undergoing testing. For example, the initial samples of people treated with hydroxychloroquine for COVID-19 were small, so generalizing from those samples risked committing a hasty generalization. This isn’t to say that small samples are always useless but due care must be taken when generalizing from them.
As a practical guide, when you hear claims about a pandemic (or anything) based on generalizations you need to consider whether the conclusion is supported by an adequately sized sample. While having a sample that is too small does not entail that the conclusion is false (that inference would be fallacious) but the conclusion would not be adequately supported. While a small sample provides weak logical support, an anecdote can have considerable psychological force which leads to a fallacy similar to the hasty generalization.
An appeal to anecdotal evidence is committed when a person draws a conclusion about a population based on an anecdote (a story) about one or very few cases. The fallacy is also committed when someone rejects reasonable statistical data supporting a claim in favor of a single example or small number of examples that go against the claim. There are two forms for this fallacy:
Form One
Premise 1: Anecdote A is told about a member M (or small number of members) of Population P.
Premise 2: Anecdote A says that M is (or is not) C.
Conclusion: Therefore, C is (or is not) true of Population P.
Form Two
Premise 1: Good statistical evidence exists for general claim C.
Premise 2: Anecdote A is is an exception to or goes against general claim C.
Conclusion: C is false.
This fallacy is like hasty generalization in that an inference is drawn from a sample too small to adequately support the conclusion. One difference between hasty generalization and anecdotal evidence is that the fallacy of anecdotal evidence involves using a story (anecdote) as the sample. Out in the wild it can be difficult to distinguish as hasty generalization from anecdotal evidence. Fortunately, what is most important is recognizing that a fallacy is occurring. A much clearer difference is that the paradigm form of anecdotal evidence involves rejecting statistical data in favor of the anecdote.
People often fall victim to this fallacy because anecdotes usually have much more psychological force than statistical data. Wanting an anecdote to be true also fuels this fallacy. During the COVID-19 pandemic, there were many anecdotes about alleged means of curing or preventing or curing the disease and the same will happen during the next pandemic. Even if the anecdotes are not lies, they do not provide an adequate basis for drawing conclusions about the general population. This is because the sample is not large enough to warrant the conclusion. As a concrete example, while there were some early positive anecdotes about hydroxychloroquine. Then wishful thinking and Trump’s claims caused some people to accept the anecdotes as adequate evidence, but this was bad reasoning.
Appeals to anecdotal evidence often occur in the context of causal reasoning, such as the case of hydroxychloroquine, and this adds additional complexities.
As with any fallacy, it does not follow that the conclusion of an appeal to anecdotal evidence is false. The error is accepting the conclusion based on inadequate evidence, not in making a false claim. It is also worth noting that anecdotal evidence can be useful for possible additional investigation but is not enough to prove a general claim.
As noted earlier, there were large samples of infected people that allowed generalizations to be drawn without committing the fallacy of hasty generalization. But even large samples can be problematic. This is because samples need to be both large enough and representative enough. This takes us to the fallacy of biased generalization.
This fallacy is committed when a person draws a conclusion about a population based on a sample that is biased to a degree or in a way that prevents it from adequately supporting the conclusion.
Premise 1: Sample S (which is too biased) is taken from population P.
Premise 2: In Sample S X% of the observed A’s are B’s.
Conclusion: X% of all A’s are B’s in Population P.
The problem with a biased sample is that it does not represent the population adequately and so does not adequately support the conclusion. This is because a biased sample can differ in relevant ways from the population that affects the percentages of A’s that are B’s.
In the case of COVID-19 there was a serious problem with biased samples, although the situation improved over time. I will focus on an inductive generalization about the lethality of the virus.
The math for calculating lethality is easy but the main challenge is sorting out how many people are infected. Since the start of the pandemic, the United States had a self-inflicted shortage of test kits. Because of this, many of the available tests were being used on people showing symptoms or who were exposed to those known to be infected. This sample was sample was large but biased: it contained a disproportionate number of people who were already showing symptoms and missed many people who were infected but asymptomatic.
If we face a similar situation in the next pandemic, the sample will probably have a lethality rate higher than the real lethality rate. To use a simple fictional example: imagine a population of 1000 people and 200 of them are infected with a virus. Of the 200 people infected, 20 show symptoms and only they are tested. Of the 20 people tested, 2 die. This sample would show a mortality rate of 10%. But the actual mortality rate would be 2 in 200 which is 1 %. This would still be bad, but not as bad as the biased sample would indicate. This shows one of the many reasons why broad testing is important: it is critical to establish an accurate lethality rate. An accurate lethality rate is essential to making rational decisions about our response to any pandemic.
As a final point, it is also important to remember that the lethality varies between groups in the overall population—we know this based on the death data. But to determine the lethality for each group, the samples used for the calculation must be representative of the population. While overall lethality is important, making rational decision making also requires knowing the lethality for various groups. For example, pathogens tend to be more lethal for seniors and they would need more protection in the next pandemic.
As always, stay safe and I will see you in the future.