My adventures in assessment began in 2004 when I was assigned to the General Education Assessment Committee. I have served on that committee since then and have become the co-chair. I have also always done the assessment for the Philosophy & Religion unit (it is part of a larger department formed during the consolidations of the 1970s). As would be expected, assessment for the unit is just an extra smooshed into the infinitely expandable 20% of my Assignment of Responsibility (which also includes advising, research, web mastering, facilitating, publishing, and 4-9 committees).
When I first started in the assessment business, most of it seemed quite arbitrary. For example, we were told by the assessment guru that we needed five outcomes to assess. I still assess five outcomes. As another example, we were told that our goal should be that 80% of Xs do Y; for example, that 80% of students achieve a rating of competent or better in an outcome. When one asked why 80% was picked, the answer was simply that this was what was to be done. To be fair, almost everyone uses 70% as the cutoff for a C grade because that has what has always been done and what everyone does. Which is to say that we use an appeal to tradition and an appeal to common practice.
As is always the case, the years have seen more and more added into the assessment process—I now attend multiple committee meetings, collect data from all the faculty in the unit, and create stacks of spreadsheets with sweet pie charts. I then do an analysis of each outcome, craft improvement narratives and write an extensive reflection. All of this gets loaded into a LiveText system for review by multiple levels of administrators. Its final fate is to be part of various collective reports. While much could be written about any part of this system, the focus of this essay is on the improvement narrative.
As noted above, the performance of the unit is assessed in terms of five outcomes. For example, the written communication skills of students are assessed as part of the written communication outcome. Once the data for the academic year is gathered and properly spreadsheeted, I do an analysis and then an improvement narrative for each assessment method within the outcome. For example, I do an analysis of draft performance relative to final paper performance as part of the written communication outcome and then write an improvement narrative that maps out a timetable and budget for the improvement. The budget part is easy: we do not get a budget for any improvement, so that is a simple copy-paste operation.
The idea of writing an improvement narrative can be reasonable. After all, if the unit is falling short on an outcome or otherwise encountering problems, then it makes sense to develop a plan for addressing these shortcomings or problems. For example, the students tend to do poorly in the area of written communication. One obvious reason for this is that students are generally less prepared in K-12 in writing argumentative papers than they were in the past. As such, the faculty must address this shortcoming in student preparation in order to bring our results up to the 80% target. That said improvement narratives can also be problematic.
One practical concern is that there is a practical limit to improvement. To use an obvious analogy to sports, an athlete can only improve so far before the cost of improvement becomes prohibitive and there is a point at which further improvement is no longer possible. The same is also true in education: even if improvement is possible, the cost of doing so will be prohibitive and hence there will be a point at which improvement is not rational. One example of this is when trying to improve in one area must come at the expense at making another area worse. To illustrate, the more a philosophy professor tries to make up for the K-12 educational shortfall in writing skills the less time they will have to address everything else—such as philosophy. Continuing the athlete analogy, it makes sense that professors also have skill caps—they will reach a point at which they cannot get any better and improvement will no longer be possible. It might be objected that a professor can always improve something and hence there must always be improvement narratives. This leads to the more abstract concern of endless improvement.
As might be imagined, one cannot simply write “we are meeting our goals and need no improvement” as part of the improvement narrative. There must always be an improvement narrative that includes improvement narratives. This assumes the possibility of endless improvement. On the one hand, it could be claimed that there is always room for improvement—unless the professor is a perfect being teaching a perfect class to perfect students. This is obviously true. On the other hand, endless improvement is a practical impossibility. One could even say that there is a seeming paradox here: imperfect beings can always improve, but imperfect beings will always hit their limits. As such, imperfect beings cannot always improve. So, what should be done about improvement narratives?
One approach is to be practical and realistic: focus on actual problems and areas that can be improved while having the freedom to claim that areas where the acceptable levels are being maintained are not going to be improved. That is, to say that the performance is good enough given the available resources and pay. This is what I strive to do and is what good assessment should be like. I take the same approach to my running: I know where I can improve and, more importantly, that time and injuries have put a hard cap on my performance. That is to say that without magic or technology, I’ll never run another 2:45 marathon or a 33 minute 10K.
Another approach is to engage in the narrative for the sake of the narrative: if people must write an improvement narrative for everything, even when they know damn well that there is no way that something is going to be improved, people will craft suitable fictions to appease the administration. On the plus side, this will check all the requisite boxes. On the minus side, it is demoralizing to do this and, to be honest, dishonest. I try to avoid doing this.
One especially interesting thing about improvement narratives is that there is the expectation, as noted above, of endless improvement. However, for most faculty their salary is largely detached from their performance and improvement: they receive no improved compensation or benefits for improving the results of outcomes. The only practical incentive is to stay above the level at which one can be fired for cause and to perform well enough for tenure and promotion; other than that, there is no practical incentive to improve. Most faculty, of course, are motivated by pride and professionalism to do a good job and improve—so it is good that people are not solely motivated by gain. But higher education is being pushed ever deeper into the business model and a key concept in business is that you get what you pay for. So, if constant improvement is expected, then there should also be a corresponding increase in compensation based on this. If such improvement is valued and required, then there should also be adequate support provided to achieve that improvement. Expecting someone to always improve without any improvement in compensation and without any increase in resources is to expect far too much. Now, back to my data analysis and narrative crafting.