I think the one thing that really pushed its way through my head in writing that is that the expert judgment we have embedded in our self-assessment is less about our training or our particular scholarly expertise and more about what my colleagues Ken Sharpe and Barry Schwartz discussed in their book Practical Wisdom: we have the expertise of having done a particular work for a long time and having a professional ethos that drives most of us to want to do it better. It's absolutely right to say that this kind of expertise has some significant limitations--every profession in the world defends itself in these terms when it's criticized (cue the Teddy Roosevelt quote: "It is not the critic who counts" etc.) but there are blindspots in every profession's work routines. (Among them, how the people who receive their services feel about the quality of what they received.) But surely experience should count for something, and yet, a lot of assessment work only brings experientially-based self-assessment by faculty into view in limited ways, and discounts its value for the most part.
I think that's in large part an effect of the "low effort" goal implicit in many of these attempted systems, the idea being to not tie up the valuable time of experts doing low-stakes evaluation by finding ways for non-experts to handle it in a routine way. In many ways, what we're ending up with, particularly in things like our "merit" system is the worst of both worlds: having recognized that routine and algorithmic systems are flawed, we're shifting back toward having experts spend inordinate amounts of time doing assessment, when the real answer ought to be "The resources involved here don't justify the time spent on it, scrap the whole thing."
(And that's above and beyond the truly penny-wise-pound-foolish stuff like shifting travel booking and expense tracking from relatively cheap administrative staff to the faculty themselves. Though there's a kind of awkward class dynamic there, too. But I'm wandering off topic...)
It's why I really do like Gawande's Checklist Manifesto. The point of that is saying "there are metrics of your performance that are automatic that you're not responsible for; here's a checklist of what YOU the expert say is necessary for a procedure to go well based on evidence (which can be revised by expert inquiry and further evidence); let's see if being reminded of the checklist every time and being double-checked by another person in the procedure changes the automatic metrics that would be logged anyway; if it does, win!" The problem then is just that neoliberal middle managers get a hold of that virtuous feedback loop and want to infinitely extend it to make checklists about everything, to make checklists about checklists, to make progress on checklisting incremental into forever, and so on, without anyone asking whether there's evidence that too many checklists is a cognitive load that causes its own problems. That's the thing with assessment in academia: the only parsimony ultimately comes from the recalitrance of the faculty, which gets caricatured as such rather than appreciated as professionals doing their damnedest to make a professional point.
I think the one thing that really pushed its way through my head in writing that is that the expert judgment we have embedded in our self-assessment is less about our training or our particular scholarly expertise and more about what my colleagues Ken Sharpe and Barry Schwartz discussed in their book Practical Wisdom: we have the expertise of having done a particular work for a long time and having a professional ethos that drives most of us to want to do it better. It's absolutely right to say that this kind of expertise has some significant limitations--every profession in the world defends itself in these terms when it's criticized (cue the Teddy Roosevelt quote: "It is not the critic who counts" etc.) but there are blindspots in every profession's work routines. (Among them, how the people who receive their services feel about the quality of what they received.) But surely experience should count for something, and yet, a lot of assessment work only brings experientially-based self-assessment by faculty into view in limited ways, and discounts its value for the most part.
I think that's in large part an effect of the "low effort" goal implicit in many of these attempted systems, the idea being to not tie up the valuable time of experts doing low-stakes evaluation by finding ways for non-experts to handle it in a routine way. In many ways, what we're ending up with, particularly in things like our "merit" system is the worst of both worlds: having recognized that routine and algorithmic systems are flawed, we're shifting back toward having experts spend inordinate amounts of time doing assessment, when the real answer ought to be "The resources involved here don't justify the time spent on it, scrap the whole thing."
(And that's above and beyond the truly penny-wise-pound-foolish stuff like shifting travel booking and expense tracking from relatively cheap administrative staff to the faculty themselves. Though there's a kind of awkward class dynamic there, too. But I'm wandering off topic...)
It's why I really do like Gawande's Checklist Manifesto. The point of that is saying "there are metrics of your performance that are automatic that you're not responsible for; here's a checklist of what YOU the expert say is necessary for a procedure to go well based on evidence (which can be revised by expert inquiry and further evidence); let's see if being reminded of the checklist every time and being double-checked by another person in the procedure changes the automatic metrics that would be logged anyway; if it does, win!" The problem then is just that neoliberal middle managers get a hold of that virtuous feedback loop and want to infinitely extend it to make checklists about everything, to make checklists about checklists, to make progress on checklisting incremental into forever, and so on, without anyone asking whether there's evidence that too many checklists is a cognitive load that causes its own problems. That's the thing with assessment in academia: the only parsimony ultimately comes from the recalitrance of the faculty, which gets caricatured as such rather than appreciated as professionals doing their damnedest to make a professional point.