I was trying to write a different thing this morning, but it wasn’t going well, so I’m going to pivot to something easy, and maybe try again next week. The easy topic is prompted by the monthly flurry of excitement about the jobs report. In particular, the part of it that changed the past, exemplified by this tweet from Justin Wolfers (which is in the middle of a thread; click through for more):
This is, as I said, a monthly ritual, where the official government employment statistics are released and everybody writes breathless tweets and news stories and opinion pieces about them. And here’s the really important thing: those monthly numbers are always wrong.
Each of those reports includes not just the statistics for the most recent month, but revisions to the numbers for the last several months. Sometimes, as with the report today, those are pretty substantial; on very rare occasions, you will even see sign changes— small decreases in employment changed to small increases, or vice versa. Years back, I talked about this with a colleague in the econ department who said that academics studying unemployment don’t regard any of these reported numbers as trustworthy until something like a full year has passed, because of all the changes that keep coming.
What you don’t get with these monthly reports is a bunch of thinkpieces about how the reports are going to destroy public confidence in economic data and the credibility of economists more generally. (Admittedly, there’s probably not a lot lower for them to go…) We don’t see lots of economists denouncing the people reporting these numbers as irresponsible hacks, or wringing their hands about how to restore trust in the profession. They just move on, because while these monthly data drops are deeply flawed, they’re the best information we have, and substantially better than nothing at all.
This is one of the very few areas (possibly the only area) where I feel like scientists and science reporters could stand to learn from their colleagues in economics. Early data are necessarily going to be imperfect, and preliminary results are likely to change, and everybody in both fields knows it. But folks in the sciences are way more likely to treat this as some kind of massive crisis for the field as a whole. Thus we get the umpty-zillion pieces about “How can anybody possibly trust the CDC any more?” because their pandemic guidance has shifted over time, and all that.
The right play here is to copy the economists: just shake it off, and move on. The imperfections of the jobs reports are widely known but barely mentioned outside of the obligatory “Last month’s numbers were also revised [thusly]” sentence in the news story template. Scientists and science reporters should just do what the econ folks do and low-key own it, rather than all this highly public angst.
Everybody, in every field, is doing the best they can to make decisions based on highly imperfect preliminary data. Just say that, and move on.
I feel like I’ve done this rant before, but of course that’s why it’s an easy fallback topic. If you’d like to be at the front of the line for the next time I fall back on this, here’s a button you can click:
If you want to pass this on to somebody else:
If you just want to rag on economists, the comments will be open. And if you’re just annoyed at having to read this again, here’s a cute photo of my dog by way of apology:
My… let's call it frustration, with the CDC isn't that their guidance has changed over time; It's that they got it wrong at the start when others got it right at the same time with the same information. We knew *very* early on that this is a strain of SARS, so it at least seems reasonable to do what was done to deal with SARS, e.g. masks. The CDC advised otherwise, even for healthcare workers.
I think there's a difference here, though. At some point, being wrong all the time about a dynamic and complex system where your early predictions and estimations are part of that complex system and can measurably perturb or affect its behavior is something that requires reckoning with. Scientific knowledge accepts that it is provisional and needs to be revised in the face of data that doesn't accord with what is predicted or expected, but at certain levels of persistent mismatching of data with expectation, the scale of the revision required is non-trivial, right? When the Hubble shows you that the early universe is not expanding at anything like the rate expected, you know there's something seriously wrong with the model you have. At certain scales of revision, you know you have to rethink the whole damn thing from soup to nuts--economics, for example, really seems to me to need to rethink its understanding of human motivation and behavior as well as the way it moves from models to empirical descriptions of real-world systems.