When I began my academic career, most of my effort focused on teaching, research and advising. Over the years, administrative tasks have devoured more and more of my time. One of the most ravenous beasts in the administrative pack is assessment. This data is now critical for accreditation and an essential part of how state funding is allocated under the punitive performance-based funding system that the state has imposed on public higher education.
Back when assessment became part of the bureaucracy, the main goal was to fill binders with paper containing outcome data. After several years of this, there was a realization that outcome data by itself did not show the value added by the educational process. To use an obvious analogy, if you just looked at the times of a cross country team at the end of the season with a new coach, you would be hard pressed to determine the impact of the coach. Because of this, measuring improvement eventually became a thing to do.
As part of the current assessment project, I am required (but not actually paid—it is not part of my official assigned responsibilities) to collect data from the classes in philosophy and religion that measures improvement in the courses. To this end, I have used an obvious approach: assessing draft papers and comparing them to the final papers. However, I also need other statistical data and the more objective the better. To this end, I created an Argument Basics Assessment Instrument (ABAI, because bureaucracy loves acronyms) which is being developed into a standardized “test” to be used in the classes. The objective of the ABAI is to determine student starting ability and their ending ability in the course. This approach is obviously not perfect, since the improvement could be due to other factors—such as skills and knowledge acquired in other courses. There are also some problems inherent to before and after assessment in a class.
One obvious problem is motivation. If the students know the assessment device does not count towards their grade, they are far less likely to take it seriously or even do it. This will tend to bias the results towards poor performance. If the device is used as part of their grade, then there are many concerns. Cheating becomes a factor, which can make the data less accurate. Also, students are more likely to try to prepare for such assessment, which can throw off the results when it is intended to get before data (and even after data). For example, if the first ABAI counted as a grade, then students might study argument basics before taking the ABAI and the assessment would reflect studying rather than their starting point. As such, the challenge is a dilemma: if the assessment does not count towards the grade, then students will tend to not take it seriously and this will skew the results in a negative way. If the assessment does count towards their grade, students would be more inclined to prepare for the assessment and perhaps even cheat, thus skewing the results. One, perhaps morally questionable approach to solving this problem, is to surprise students with the assessment and not reveal whether it counts or not. Another approach is to be honest with the students and note in the assessment report that the results will be biased based on the approach taken.
Another obvious problem is that if the assessment devices count towards the grade, then the “before” assessments will tend to have poor results which could hurt the students’ grades. A solution is to scale the first assessment, but this can cause the students to take it less seriously. Another solution is to count only the “after” assessment, but this runs into the same problem. In the case of paper drafts used for before/after assessment, students generally tend to put effort into the drafts when they are optional. But this tends to skew the results because the better students tend to do drafts. When drafts are mandatory but do not count towards the grade, then many students tend to not take them seriously, which skews the results.
The cynical approach is obvious: simply pick the method that generates the sort of data the bureaucracy wants (which is usually that X% of students improve Y%). A less cynical approach is to test various methods to see what seems to yield the most accurate results—but there is the obvious problem of calibrating the system. After all, assessing the assessment leads to the sort of infinite regress that plagues epistemology: how do you know the method is accurate? If you use a method to assess the method, then one must ask how you know that method is accurate. And so on, into infinity. Not surprisingly, the bureaucracy tends to avoid peering to deeply under the hood.
When dealing with a bureaucracy, it is usually best to cynically take the path of least resistance.
This is relevant:
Fourth is box-ticking, and it’s pretty much what you’d imagine. In academia we spend more and more of our time assessing and monitoring and describing and proposing than actually doing what we do. Box ticking is a ritual substitute for actual action.
https://www.thedailybeast.com/nearly-half-of-you-reading-this-have-bullshit-jobs
Ugh.
In addition to all the potential issues with students – grading, motivation, cheating, etc, you also have the issue of those doing the evaluation.
You and the other evaluators are stakeholders in the outcome – a good assessment means more state funding – meaning better classrooms, better facilities, pay raises, etc. A bad assessment may result in cuts.
Of course, if you follow the other bureaucratic/political model, the bad assessments would be the targets for funding; politicians love to throw money at problems to show that they are doing something.
I have often proposed that in our department, rather than evaluate a teacher or students at the end of a class, students should have to pass an “entrance” exam to be admitted to the next level. So many courses in so many institutions will set an entrance requirement as, “Must have gotten a “C” or better in 201″, which to me is meaningless.
In my classes, where I get a fairly wide range of skills and abilities among students, I will often make the first assignment a quiz or a survey, to help me to pace the class or to recommend remedial material for some or advanced material for others. The motivation for the students to take this quiz is that they then become contributors, as well as stakeholders, in a particular class. It is in their best interest to take the quiz/survey honestly and completely, as it will directly impact their experience during the semester.
This also serves as an informal assessment tool for the prerequisites. Occasionally I will find some consistency – that a majority of the students who completed 201 section 01 are less prepared than those who completed 201 section 02; which then reflects back on the instructor and/or individualized curriculum.
it’s a balance – because we want our teachers to have as much academic freedom as possible and to teach to their strengths, but they need to do this within a set of core outcomes determined by the department and the school.
While the problems you cite aren’t trivial, they exist in any assessment model – and there needs to be a way of weeding them out. The larger the data set the better – it’s not easy to do this in small classes.
That does pose a moral problem and some schools have run into issues involving cheating on their assessment data.
I’ve stuck to the data even when it shows that we do not meet the goals; this is due to the influence of Ben Franklin.
Florida has taken the view that problems are best dealt with by punitive measures-if a school faces serious obstacles, that does not matter much. What matters is the results (or at least the results they want). The state also seems to have a clear agenda. For example, my school would have gotten the funding this year because we hit the target for the 6 year graduation rate and edged out another school. The response was to change the measure to the 4 year graduation rate after the fact and thus we lost the funding.
You are right about the problem with pre-reqs; students could get through one class and yet not be prepared for the next. This can be addressed by ensuring faculty are doing their job, but there are the usual political problems with that sort of policing.
The response was to change the measure to the 4 year graduation rate after the fact and thus we lost the funding
In 2010, the housing market hit rock bottom in NJ. I filed a tax appeal, because my home value had dropped by a full 1/3.
The township re-assessed not only my property, but the whole town, and.the results were as expected. When I got my tax bill, however, I discovered that they had just raised the tax rate to make up for.the potential loss of revenue from reduced value.
Bureaucrats will get what they want.
Giving them what they want is not so cynical, as you suggest. It really has nothing to do with education or pedagogy or effective teaching. Nor, as you have painfully discovered, is it even about funding. It’s about whatever their agenda is, and they are just fine with changing the rules midstream in order to meet their own unspoken goals.
This can be addressed by ensuring faculty are doing their job, but there are the usual political problems with that sort of policing.
This is a big issue. We have discussed this at length, and continue to do so, in our Academic Senate and Accademic Affairs committee. Our issue is the balance between the academic freedom of a professor to teach in the manner he or she wants, versus the goals of any individual course as set forth by the department. One particular issue is that of a department requiring a common exam among several sections of the same course.
There’s a lot to this debate, with valid and salient arguments on both sides – and that’s at a private university. Add the state bureaucracy unto the mix trying to get what they want, and all you get is chaos, and the only winners are the politicians.
One more point…Giving in to the bureaucrats will actually serve the students better. Your efforts to provide meaningful and accurate assessment data as a means to an end are meaningless,as the results have shown,
If you are a good, effective teacher, then for you to spend time outside of the classroom doing this kind of bullshit only harms the students by compromising the education they (and the Florida taxpayers} are paying for.
The moral and ethical thing for you to do is to quickly figure out what the bureaucrats want and give it to them as efficiently as you can, and get back into the classroom and serve your students and the citizens of Florida.
Of course, if the professor in question is a terrible teacher whose lectures and curriculum do more harm than good, then the moral choice would be to big that person down with enough administrative work to qualify for a course release. Or two. Or make him a dean.
Cue up the dean jokes.
The Dean is hospitalized after a heart attack. As he is lying in his hospital bed reflecting on his near brush with death, an attendant arrives with a lovely bouquet of flowers. The Dean asks the attendant to hand him the card and finds that it reads, “By a vote of 26 to 3 with 2 abstentions, the faculty wish you a speedy recovery.”
Or maybe it should be “Queue up the dean jokes”?
A woman called the public relations office of a college and requested a
speaker for her club saying ” I want nothing lower than a dean.”
The staffer replied, “Madam, there is nothing lower than a dean.”
Of course, if the professor in question is a terrible teacher whose lectures and curriculum do more harm than good, then the moral choice would be to big that person down with enough administrative work to qualify for a course release. Or two. Or make him a dean.
Heh…heh, heh…heh.
Also, see “College Education, costs of”. It is a riddle, wrapped in a mystery, inside an enigma. /sarc