While my husky, Isis, and I have both slowed down since we teamed up in 2004, she is doing remarkably well these days. As I often say, pulling so many years will slow down man and dog. While Isis faced a crisis, most likely due to the wear of time on her spine, the steroids seemed to have addressed the pain and inflammation so that we have resumed our usual adventures. Tail up and bright eyed is the way she is now and the way she should be.
In my previous essay I looked at using causal reasoning on a small sale by applying the methods of difference and agreement. In this essay I will look at thinking critically about experiments and studies.
The gold standard in science is the controlled cause to effect experiment. The objective of this experiment is to determine the effect of a cause. As such, the question is “I wonder what this does?” While the actual conducting of such an experiment can be complicated and difficult, the basic idea is rather simple. The first step is to have a question about a causal agent. For example, it might be wondered what effect steroids have on arthritis in elderly dogs. The second step is to determine the target population, which might already be taken care of in the first step—for example, elderly dogs would be the target population. The third step is to pull a random sample from the target population. This sample needs to be representative (that is, it needs to be like the target population and should ideally be a perfect match in miniature). For example, a sample from the population of elderly dogs would ideally include all breeds of dogs, male dogs, female dogs, and so on for all relevant qualities of dogs. The problem with a biased sample is that the inference drawn from the experiment will be weak because the sample might not be adequately like the general population. The sample also needs to be large enough—a sample that is too small will also fail to adequately support the inference drawn from the experiment.
The fourth step involves splitting the sample into the control group and the experimental group. These groups need to be as similar as possible (and can actually be made of the same individuals). The reason they need to be alike is because in the fifth step the experimenters introduce the cause (such as steroids) to the experimental group and the experiment is run to see what difference this makes between the two groups. The final step is getting the results and determining if the difference is statistically significant. This occurs when the difference between the two groups can be confidently attributed to the presence of the cause (as opposed to chance or other factors). While calculating this properly can be complicated, when assessing an experiment (such as a clinical trial) it is easy enough to compare the number of individuals in the sample to the difference between the experimental and control groups. This handy table from Critical Thinking makes this quite easy and also shows the importance of having a large enough sample.
|Number in Experimental Group
(with similarly sized control group)
|Approximate Figure That the difference Must Exceed
To Be Statistically Significant
(in percentage points)
Many “clinical trials” mentioned in articles and blog posts have very small samples sizes and this often makes their results meaningless. This table also shows why anecdotal evidence is fallacious: a sample size of one is all but completely useless when it comes to an experiment.
The above table also assumes that the experiment is run correctly: the sample was representative, the control group was adequately matched to the experimental group, the experimenters were not biased, and so on for all the relevant factors. As such, when considering the results of an experiment it is important to consider those factors as well. If, for example, you are reading an article about an herbal supplement for arthritic dogs and it mentions a clinical trial, you would want to check on the sample size, the difference between the two groups and determine whether the experiment was also properly conducted. Without this information, you would need to rely entirely on the credibility of the source. If the source is credible and claims that the experiment was conducted properly, then it would be reasonable to trust the results. If the source’s credibility is in question, then trust should be withheld. Assessing credibility is a matter of determining expertise and the goal is to avoid being a victim of a fallacious appeal to authority. Here is a short checklist for determining whether a person (or source) is an expert or not:
- The person has sufficient expertise in the subject matter in question.
- The claim being made by the person is within her area(s) of expertise.
- There is an adequate degree of agreement among the other experts in the subject in question.
- The person in question is not significantly biased.
- The area of expertise is a legitimate area or discipline.
- The authority in question must be identified.
While the experiment is the gold standard, there are times when it cannot be used. In some cases, this is a matter of ethics: exposing people or animals to something potentially dangerous might be deemed morally unacceptable. In other cases, it is a matter of practicality or necessity. In such cases, studies are used.
One type of study is the non-experimental cause to effect study. This is identical to the cause to effect experiment with one rather critical difference: the experimental group is not exposed to the cause by those running the study. For example, a study might be conducted of dogs who recovered from Lyme disease to see what long term effects it has on them.
The study, as would be expected, runs in the same basic way as the experiment and if there is a statistically significant difference between the two groups (and it has been adequately conducted) then it is reasonable to make the relevant inference about the effect of the cause in question.
While useful, this sort of study is weaker than the experiment. This is because those conducting the study have to take what they get—the experimental group is already exposed to the cause and this can create problems in properly sorting out the effect of the cause in question. As such, while a properly run experiment can still get erroneous results, a properly run study is even more likely to have issues.
A second type of study is the effect to cause study. It differs from the cause to effect experiment and study in that the effect is known but the cause is not. Hence, the goal is to infer an unknown cause from the known effect. It also differs from the experiment in that those conducting the study obviously do not introduce the cause.
This study is conducted by comparing the experimental group and the control group (which are, ideally, as similar as possible) to sort out a likely cause by considering the differences between them. As would be expected, this method is far less reliable than the others since those doing the study are trying to backtrack from an effect to a cause. If considerable time has passed since the suspected cause, this can make the matter even more difficult to sort out. The conducting the study also have to work with the experimental group they happen to get and this can introduce many complications into the study, making a strong inference problematic.
An example of this would be a study of elderly dogs who suffer from paw knuckling (the paw flips over so the dog is walking on the top of the paw) to determine the cause of this effect. As one might suspect, finding the cause would be challenging—there would be a multitude of potential causes in the history of the dogs ranging from injury to disease. It is also quite likely that there are many causes in play here, and this would require sorting out the different causes for this same effect. Because of such factors, the effect to cause study is the weakest of the three and supports the lowest level of confidence in its results even when conducted properly. This explains why it can be so difficult for researchers to determine the causes of many problems that, for example, elderly dogs suffer from.
In the case of Isis, the steroids that she is taking have been well-studied, so it is quite reasonable for me to believe that they are a causal factor in her remarkable recovery. I do not, however, know for sure what caused her knuckling—there are so many potential causes for that effect. However, the important thing is that she is now walking normally about 90% of the time and her tail is back in the air, showing that she is a happy husky.