While assessment has become embedded deeply into the body of education, when it first appeared I suspected it would be yet another fading fad of the academy. I recall, through the haze of the years, thinking of a modification of the old saying and coming up with “those who can do; those who can’t teach; those who can’t teach assess.” Back in those days, many professors regarded assessment as something of a scam: clever assessment “experts” getting well-paying positions or consulting gigs and then foisting the grunt work onto professors. The wilier (or lazy) professors responded by simply making up assessment data and noted that the effectiveness of their fictional data was identical to that of the real data. That is, not effective at all. I, like many professors, found myself in brave new world of assessment.
I eventually got dragged deep into the system of assessment. At the start, it fell on me to complete the assessment work for Philosophy & Religion. In 2004 I was given an eternal assignment to the General Education Assessment Committee (GEAC) and then made a co-chair. This resulted in me being on all the assessment committees. As such, I now have some experience with assessment.
On the one hand, I do retain much of my early skepticism of assessment. Much of it still seems to be at worst a scam and at best a waste of time. There is clearly a lot of money to be made in this area, money that is bleed away from other areas. Assessment also eats into the time of the faculty, time that could be used for teaching or research. There are also good questions about the effectiveness of assessment—even assessment that is done properly and sincerely.
On the other hand, my reading of Aristotle and experience as a teacher has led me to accept that there is merit in assessment that is properly done. The good and proper purpose of assessment is to evaluate the effectiveness of education in classes and in the institution as a whole. This is entirely reasonable—as Aristotle noted in his Nicomachean Ethics, if one aims to become a morally good person, then one needs an index of progress. In the case of virtue, Aristotle used pain and pleasure as his measure: if you feel increasing pleasure at doing good and increasing pain at doing wrong, then you are making progress. Using this indirect measure (to use an assessment term) enables one to assess their progress (or lack thereof). In the case of education, there must also be assessment. Otherwise one does not know how well they are doing at the task of educating the students.
One mantra among the assessment elite is “grades are not assessment.” While this has been challenged, I heard this being repeated as an article of faith by experts as recently as this month (March, 2018). To be fair, there is some truth to this mantra. One obvious concern is that grades often include factors irrelevant to properly assessing the quality of work. Professors often grant extra credit that is not based on merit and such things as attendance and participation can go into grades. For example, my students can get +5 points added on to a paper grade if they turn the paper in by the +5 bonus deadline. If these grades were blindly used for assessment, they would be off. This, I agree, is a factor than can make some grades less useful or even useless for assessment. However, it is easy enough to adjust for many of these situations. For example, knowing that the +5 bonus papers have a +5 bonus would allow them to be easily assessed using the grades. What I do, of course, is assess the papers using the proper assessment rubrics—to avoid getting another lecture on why grades are not assessment.
Another obvious concern is that professors tend to be inconsistent in their grading. For example, the way I grade a paper is different from that of a colleague so that a paper I grade as an 84 might be graded as a 79 or a 90 by a colleague. Part of this can be due to a professor being a harder or easier grader; part of it can also be due to using different standards. While this is a concern, the same problem can obviously apply to assessment. Different assessors will be harder or easier in their assessment. While having a standard rubric can help offset this, the subjectivity remains whether it is a grade or assessment. Another approach is to have multiple faculty assess the same artifacts. While a good idea, schools are rarely willing to compensate faculty for this extra work—and assessing entire classes of work takes long hours.
There are also the concerns that some faculty are bad at properly grading work and hence their grades are not legitimate assessments. While it is true that some faculty are bad at grading, this is not a problem with grading but a problem with the faculty. Addressing the shortcoming would fix two problems: bad grades and assessment. There is also the fact that people can be just as bad at assessment, especially when people are assigned to assess work outside of their field. For example, an English professor might be asked to assess philosophy papers for critical thinking or an engineering professor might be asked to review biology lab reports for written communication.
In closing, much of assessment is (to be honest) ineffective and a waste of resources. But, it is beloved by many state legislatures and is part of the business model that now rules higher education. As such, it is not a fading fad. The challenge, then, is to turn assessment into something effective and useful—something I have been endeavoring to do.
There is some assessment that is easy – in teaching subjects like math or physics, for example, there is often a right answer and a wrong answer – and an “84” vs a “79” is not a matter of an easy or hard grader.
In my field (3D Digital Design), there is a mix – our assessment is based as much (if not more) on design principles and even purposeful breaking of the rules as it is an understanding of methodologies and processes. The latter can refer to a software-independent conceptual understanding of workflow (and how this stuff actually works or a software-specific set of tools and procedures that are necessary to get the job done. One can, for example, understand the basic concepts of Chemistry, but without a solid understanding of math or a mastery of using the tools in a lab, that conceptual understanding will only go so far.
The goals of assessment are sometimes elusive also. I can have a student who (to use your example) might turn in all the work on time and done completely – but showing very little initiative in design conceptualization – or the mastery or creative implementation of the tools.
Often, my best students are also my worst students – those who do not follow the “rules of the class” and master the material in their own way in their own time, based on their own research/project schedules and in the direction they want to go – which may be apart from what the department or school might intend as a generic focus. I know that these students will go far – they will work at the best studios, they will produce the most innovative work, they are true artists and creators – but they present a real dilemma when it comes to giving them a “fair” grade based on the same rubric as the others. But these are the students with whom I can have this frank discussion – tell them flat out that I believe that their work is more important than their GPA and that they have some decisions they need to make. Fortunately, my industry is very heavily portfolio-based; very few of my BFA students go on to an MFA or PhD degree (and those that do are usually the “good students”.
I’m involved heavily now in the opposite kind of assessment – the methods used in assessing teacher effectiveness. For one set of data, we currently use an outside software product to conduct surveys at the end of classes. The results of these surveys are compared in many ways to peers in the department, the school, the college, the institute, and are used as part of an overall faculty assessment metric for the purpose of determining raises and/or promotions.
There are HUGE flaws inherent in this system. Starting with the data itself, all questions fall on a Likert scale (“on a scale of 1 – 5, how organized was this professor?”). If you use a mean rather than a median, you can get entirely different results – so which is correct? There are many factors that have nothing to do with teacher effectiveness that affect the outcome of these surveys, regardless of whether you use a mean or median – things like class size, class level, degree of difficulty, and even student understanding of what the numbers mean. I have spoken to many students who, for example, will say “I never give a “5”. No one is perfect. For me, the highest score is a “4””. Fair enough, but those students can potentially deny a very able professor a raise or promotion. In other cases, two students may have the exact same view of a professor, but one will give a “3”, while the other gives a “4” – based only on their interpretation of what the numbers mean.
It has been statistically shown that there is an inverse relationship between the difficulty of a class and the ratings of a professor – such that a student can learn a tremendous amount from a very able professor, but because the class was hard and the professor was demanding, he will get low scores.
Class size is also a factor – when we have large lecture-based intro classes (200-300 students), the professor can do nothing but BE the mean. That’s fine, statistically – but completely unfair to that professor when it comes to comparing evaluations with his peers for purposes of determining raises.
I could go on and on – the student ratings system is only part of the overall mix; teacher effectiveness is also based on peer observation, outside research & scholarship, and service to the institute and community – but with so many colleges and schools within our institute, the department heads and deans all have their own ways of assessing this data.
To me, a big part of assessment should be well after the fact – the effectiveness of a teacher in a 100-level class should be determined in part by how those students fare in the 200-level class, etc. Post-graduate surveys can also be helpful. For me, personally, I look back on a class I took as a sophomore; I hated the class, I hated the professor, I thought the work was rote, boring, and dull – and yet I look back at that class as having been one of the most valuable to my career.
I do agree with you in large part when it comes to students. On the one hand (mine), I am teaching adults. They know how much work they are doing, they see their own results. They know what deadlines they have made or missed, they know what their attendance has been. What is the correct way to grade a superb student? What if a student is way ahead of the curve, and produces excellent work without having to put in much effort, and doesn’t push himself? Or skips a lot of classes? What about a student who puts in ten times the effort, comes to every class, sees me during my office hours, and struggles to produce mediocre work? The bottom line is, “they know”.
On the other hand (administration and students alike), you are correct – they love this stuff. They need comparative information, they thirst for data. It’s like Ed Koch used to say in NY – “How’m I doin’?” There are some students who will grub for every point they can – who, at the end of the semester, will ask if they can re-do substandard work for a better grade, or if they can do anything for “extra credit”.
All we can do is teach and try.
Michael LaBossiere says
True; student assessment of professors seems to mostly a matter of how well they liked the class/professor and how easy it was to get a good grade. Students, like most people, tend to dislike it when they are pushed hard to be better at something.
From the Walter Williams article quoted above:
Via David Thompson
US student debt is now at $1,000,000,000,000 (one trillion dollars).