Also Known as: Biased Statistics, Loaded Sample, Prejudiced Statistics, Prejudiced Sample, Loaded Statistics, Biased Induction
Description:
This fallacy is committed when a conclusion is made about a population based on a sample that is biased. It has the following form:
Premise: Sample S, which is biased, is taken from population P.
Conclusion: Claim C made about Population P based on S.
This fallacy is a flawed Inductive Generalization:
Premise: X% of all observed A’s are B’s.
Conclusion: Therefore X% of all A’s are B’s.
An Inductive Generalization can be a good (strong) argument, provided that the sample is large enough (see the Hasty Generalization) and not biased. The fallacious version can be presented in this form:
Premise: X% of all observed A’s are B’s in biased sample S.
Conclusion: Therefore X% of all A’s are B’s.
Those committing the fallacy do not, of course, identify their sample as biased. They would present it as if it were a good (strong) Inductive Generalization and the sample would need to be evaluated to determine its bias.
The fallacy is committed when the sample is likely to be biased. A sample is biased or loaded when the method used to take the sample is likely to result in a sample that does not adequately represent the population from which it is drawn.
Biased samples are unreliable. As a blatant case, imagine a person is taking a sample from a bucket of colored balls. Some of the balls are metal and some are plastic. If they used a magnet to select a sample, then the sample would include a disproportionate number of metal balls. In this case, any conclusions drawn about all the balls would be unreliable since there would probably be no plastic balls in the sample.
Biased samples are less likely to contain numbers proportional to the whole population. Bias is a relative concept and the same sample that is representative for one purpose could be representative for another.
For example, if a person wants to find out what most Americans thought about gun control, a poll taken at a large NRA (National Rifle Association) meeting would be a biased sample. But if they wanted to know what NRA members think, it would not be biased.
As another example, if a sample was taken at rally for gun control, that sample would also be biased if the goal was to determine what most Americans think about gun control. But if the goal was to determine the opinions of people who rally for gun control, the sample would not be biased.
Since the Biased Sample fallacy is committed when the sample (the observed instances) is biased or loaded, a good generalization requires an unbiased sample. The best way to do this is to take samples in ways that avoid bias. There are three general sample types aimed at avoiding bias. These methods (when used properly) will result in a sample that matches the whole population reasonably closely. Three types of samples are as follows:
Random Sample: This is a sample that is taken in such a way that only chance determines which members of the population are selected for the sample. Ideally, any individual member of the population has the same chance as being selected as any other. This type of sample avoids being biased because a biased sample is one that is taken in such a way that some members of the population have a higher chance of being selected than others. Unfortunately, creating an ideal random sample can be very difficult.
Stratified Sample: This is a sample that is taken by using the following steps: 1) The relevant strata (population subgroups) are identified, 2) The number of members in each stratum is determined and 3) A random sample is taken from each stratum in exact proportion to its size. This method is obviously most useful when dealing with stratified populations. For example, a person’s income often influences how she votes, so when conducting a presidential poll it would be a good idea to take a stratified sample using economic classes as the basis for determining the strata. This method avoids loaded samples by (ideally) ensuring that each stratum of the population is adequately represented.
Time Lapse Sample: This type of sample is taken by taking a stratified or random sample and then taking at least one more sample with a significant lapse of time between them. After the two samples are taken, they can be compared for changes. This method of sample taking is important when making predictions. A prediction based on only one sample is likely to be a Hasty Generalization (because the sample is likely to be too small to cover past, present, and future populations) or a Biased Sample (because the sample will only include instances from one time)
People often commit Biased Sample because of bias or prejudice. A person might intentionally or unintentionally seek out people or events that support their bias. As an example, a person who is pushing a particular scientific theory might tend to gather samples that are biased in favor of that theory. A person who is pushing a political narrative might gather samples that are most likely to seem to support their narrative.
People sometimes commit this fallacy from laziness or sloppiness rather than from bad intentions. It is easy to just take a sample from what is readily available rather than taking the time and effort to generate an adequate sample and draw a justified conclusion.
Defense: The defense against committing this fallacy is ensuring that your samples are not significantly biased. When considering a generalization made by someone else, the defense is to check to see if the sample is biased. If you have no way of determining this, you should suspend judgment.
Example #1:
Bill is assigned by his editor to determine what most Americans think about a new law that will place a federal tax on all modems and computers purchased. The revenues from the tax will be used to enforce new online decency laws. Bill, being technically inclined, decides to use an email poll. In his poll, 95% of those surveyed opposed the tax. Bill was quite surprised when 65% of all Americans voted for the taxes.
Example #2:
The United Pacifists of America decide to run a poll to determine what Americans think about guns and gun control. Jane is assigned the task of setting up the study. To save mailing costs, she includes the survey form in the group’s newsletter mailing. She is very pleased to find out that 95% of those surveyed favor gun control laws and she tells her friends that most Americans favor gun control laws.
Example #3:
Large scale polls were taken in Florida, California, and Maine and it was found that an average of 55% of those polled spent at least fourteen days a year near the ocean. So, it can be safely concluded that 55% of all Americans spend at least fourteen days near the ocean each year.