One of the problems becomes faculty complicity in the thing we claim to hate. Everyone knows all of the problems with teaching evals. but we are all willing to use them as at least a crude tool. I think of teaching evals a bit as being akin to the SATs. I know what the margins mean -- a kid who gets a 1600 is no dummy, a kid who got a 400 is likely to struggle in college, but tell me to differentiate in a meaningful way between 1050 and 1250 and I really don't think anyone could say with a straight face that they know what it means.
Same with teaching evals. On a traditional 1-5 scale, 5 being good, if someone is averaging 4.7 all the time, we basically say they are fine -- great, even! -- unless there is something else that stands out. If someone is consistently averaging 2.3 we have a problem. But what does a 3.3 mean -- especially if conscientious students are saying (3 says average -- this person was fine! Give 'em a 3!) Making matters worse, on some scales 3 is not a midway point, but it connotes neutrality, which lowers the average of good people even if the student is not making a value judgment, ditto raising the value of poor performers.
I generally agree with this take, and especially with the point that high-effort data collection needs to be justified by high-stakes outcomes -- a lesson that could be applied far beyond academia. (Ask your doctor how much of their time they spend on insurance paperwork.)
That said, I do have a quibble with the implication that high-quality data collection must be onerous. It usually *is* onerous, but a lot of the time it could be made far less painful than it is. Again looking at the doctor's office: How often have you been handed a packet of forms where you fill in the exact same data repeatedly in slightly different formulations? That isn't a necessary part of the process. It's just wasted effort created by our fragmented health care system.
Quality data collection is always going to require more effort at some point. But if more of that effort were invested in designing an efficient process (which includes integrating it with existing processes), a lot less would be required to carry it out.
> Ask your doctor how much of their time they spend on insurance paperwork.
I think the point you're making here is that that is not high-stakes, but from the doctor's perspective, it is. That's how the ensure they get paid. (Same stakes as are being discussed in this post.)
I agree with your point that better integration could streamline the effort required on all parties, in a huge variety of applications.
Yes! We were having just these discussions last year in our non-tenure CC. If you *really* wanted to evaluate faculty and teaching, you *could*, but it would be an enormous logistical hurdle and cost/time sink that no one would want to participate in (especially if the whole aim of the exercise is not to reward the high-achievers but to get rid of the bad apples).
The thing that drives me nuts about assessment regimes is first of all that grading IS assessment of precisely the things that most assessment designers claim to want to know about a course: did students meet the stated learning objectives? You can even use grading to measure whether students over time are doing worse or better on those objectives if you hold the objectives, the graded materials, and the rubrics for grading constant over multiple versions of the class. But at a lot of institutions, the people pushing assessment just completely ignore grading as data.
I think that shows one of the hidden premises of a lot of assessment, which is that data produced by faculty practice is basically seen as being enemy action, e.g., that faculty are viewed as ipso facto unreliable assessors of student learning because the assessors are really trying to evaluate faculty themselves without necessarily saying so.
If not, then it wouldn't just be grading that assessors would be interested in. Nobody charged with doing this kind of work ever pursues qualitative evaluations of faculty based on interviews or self-narration about the adjustments we make to syllabi, problem sets, assignments, and so on. And yet I think that would give anybody genuinely interested in assessment a rich insight into how we make judgments all the time about what's working and not working in our curriculum, our individual courses, and our pedagogy. The fact that this kind of thinking just never ever enters the picture to me shows that there's a kind of idolatry about the quantitative going on. Even if the point is to find weak teaching or bad-faith professionalism, then asking people to continuously narrate their thinking about classes and pedagogy combined with a portfolio of their teaching work is going to show you something meaningful. I guarantee that if I asked an entire college or university's faculty to do this kind of writing or speaking about their teaching, I'd be able to spot the people who were putting in minimal effort or were totally bullshitting, especially if it was in an interview format. (It's EXACTLY what we do with tenure and promotion--most of a typical dossier is qualitative assessment of a faculty member's work, including self-assessment.) But as you say, to do that right, qualitative or quantitative, doesn't scale well--it's hugely time-consuming. So we suffer the death of a thousand tiny cuts just for the sake of assessment theater.
I take that as mostly a response to the not-uncommon practice of grading on a variable curve, where the top student gets an A independent of any objective standards. Which makes grades a less reliable source of data than they might be.
I certainly know a good number of folks who have a more absolute standard in mind, but purely relative grading is common enough to be worth checking against some external standard (either a separate test or a later course).
Yeah, that to me is another reason why I think grading on a curve is a mistake. If you set learning objectives and everybody meets them, the idea that you have to reimpose a hierarchy that creates distance between the people who exceed the objectives the most and the people who just meet them seems to me to really cross the streams. Even if meritocracy is a good idea, I'd rather it sort itself out more naturally in the life and career of our graduates. I just want to distinguish between the students who didn't meet the objectives and the ones who did.
One of the problems becomes faculty complicity in the thing we claim to hate. Everyone knows all of the problems with teaching evals. but we are all willing to use them as at least a crude tool. I think of teaching evals a bit as being akin to the SATs. I know what the margins mean -- a kid who gets a 1600 is no dummy, a kid who got a 400 is likely to struggle in college, but tell me to differentiate in a meaningful way between 1050 and 1250 and I really don't think anyone could say with a straight face that they know what it means.
Same with teaching evals. On a traditional 1-5 scale, 5 being good, if someone is averaging 4.7 all the time, we basically say they are fine -- great, even! -- unless there is something else that stands out. If someone is consistently averaging 2.3 we have a problem. But what does a 3.3 mean -- especially if conscientious students are saying (3 says average -- this person was fine! Give 'em a 3!) Making matters worse, on some scales 3 is not a midway point, but it connotes neutrality, which lowers the average of good people even if the student is not making a value judgment, ditto raising the value of poor performers.
I generally agree with this take, and especially with the point that high-effort data collection needs to be justified by high-stakes outcomes -- a lesson that could be applied far beyond academia. (Ask your doctor how much of their time they spend on insurance paperwork.)
That said, I do have a quibble with the implication that high-quality data collection must be onerous. It usually *is* onerous, but a lot of the time it could be made far less painful than it is. Again looking at the doctor's office: How often have you been handed a packet of forms where you fill in the exact same data repeatedly in slightly different formulations? That isn't a necessary part of the process. It's just wasted effort created by our fragmented health care system.
Quality data collection is always going to require more effort at some point. But if more of that effort were invested in designing an efficient process (which includes integrating it with existing processes), a lot less would be required to carry it out.
> Ask your doctor how much of their time they spend on insurance paperwork.
I think the point you're making here is that that is not high-stakes, but from the doctor's perspective, it is. That's how the ensure they get paid. (Same stakes as are being discussed in this post.)
I agree with your point that better integration could streamline the effort required on all parties, in a huge variety of applications.
Yes! We were having just these discussions last year in our non-tenure CC. If you *really* wanted to evaluate faculty and teaching, you *could*, but it would be an enormous logistical hurdle and cost/time sink that no one would want to participate in (especially if the whole aim of the exercise is not to reward the high-achievers but to get rid of the bad apples).
The thing that drives me nuts about assessment regimes is first of all that grading IS assessment of precisely the things that most assessment designers claim to want to know about a course: did students meet the stated learning objectives? You can even use grading to measure whether students over time are doing worse or better on those objectives if you hold the objectives, the graded materials, and the rubrics for grading constant over multiple versions of the class. But at a lot of institutions, the people pushing assessment just completely ignore grading as data.
I think that shows one of the hidden premises of a lot of assessment, which is that data produced by faculty practice is basically seen as being enemy action, e.g., that faculty are viewed as ipso facto unreliable assessors of student learning because the assessors are really trying to evaluate faculty themselves without necessarily saying so.
If not, then it wouldn't just be grading that assessors would be interested in. Nobody charged with doing this kind of work ever pursues qualitative evaluations of faculty based on interviews or self-narration about the adjustments we make to syllabi, problem sets, assignments, and so on. And yet I think that would give anybody genuinely interested in assessment a rich insight into how we make judgments all the time about what's working and not working in our curriculum, our individual courses, and our pedagogy. The fact that this kind of thinking just never ever enters the picture to me shows that there's a kind of idolatry about the quantitative going on. Even if the point is to find weak teaching or bad-faith professionalism, then asking people to continuously narrate their thinking about classes and pedagogy combined with a portfolio of their teaching work is going to show you something meaningful. I guarantee that if I asked an entire college or university's faculty to do this kind of writing or speaking about their teaching, I'd be able to spot the people who were putting in minimal effort or were totally bullshitting, especially if it was in an interview format. (It's EXACTLY what we do with tenure and promotion--most of a typical dossier is qualitative assessment of a faculty member's work, including self-assessment.) But as you say, to do that right, qualitative or quantitative, doesn't scale well--it's hugely time-consuming. So we suffer the death of a thousand tiny cuts just for the sake of assessment theater.
I take that as mostly a response to the not-uncommon practice of grading on a variable curve, where the top student gets an A independent of any objective standards. Which makes grades a less reliable source of data than they might be.
I certainly know a good number of folks who have a more absolute standard in mind, but purely relative grading is common enough to be worth checking against some external standard (either a separate test or a later course).
Yeah, that to me is another reason why I think grading on a curve is a mistake. If you set learning objectives and everybody meets them, the idea that you have to reimpose a hierarchy that creates distance between the people who exceed the objectives the most and the people who just meet them seems to me to really cross the streams. Even if meritocracy is a good idea, I'd rather it sort itself out more naturally in the life and career of our graduates. I just want to distinguish between the students who didn't meet the objectives and the ones who did.