How Useful Is the Bag of Words?

Dipping into the roiling cauldron of linear algebra

As mentioned previously, I am not implacably opposed to the idea of employing the systems being pushed as “AI” to do simple tasks, but I have thus far struggled to find a use case involving things that I actually do. Last week, I realized I had stumbled into one— a task that was mildly annoying to do, likely within the capability of a current LLM, and not so high-stakes that it flaming out would do any real damage. So I gave it a shot… and it didn’t work.

After doing it by hand, though— because it needed to get done last week— I came back to the task this morning, taking screenshots along the way to document the process. This time out it worked better, though it’s not clear to me how big an improvement this was… But if nothing else, it’s good for a blog post.

Gemini output from “Please create an image representing an AI sorting a spreadsheet and dividing it into smaller files in a cartoon format suitable for illustrating a blog post”

The situation is that my Quantum Optics/ Quantum Information students had to do in-class oral presentations on their research paper topics last week (after turning in rough drafts, before the final draft). Part of my process for doing these is to collect feedback from the other students on each of the presentations, and provide those comments to each speaker.

In the more distant past, I would hand out paper comment sheets then collect and redistribute those, but more recently I’ve been doing this with a five-question Google Forms survey (choose the speaker from a radio-button list, then answer four questions about the talk), to cut down on the amount of dead-tree clutter in my office. The end result was a Google Sheet with roughly 150 rows of content, not quite in the order of the talks, which I wanted to divide up into individual comment files for each student that I could then share with them.

This feels similar enough to the kinds of things I see people talking about doing with “AI” that it seemed worth a shot. It’s not a complicated task by any means— after the initial failure, I did it by hand with “Sort” and a bunch of copy-pasting in about an hour¹— and I don’t do this all that often (maybe 1-2 times per academic year), but it seemed like a decent place to test out the general process.

For this morning’s re-test, I cut the file down to about 50 rows of comments and only eight speakers, to speed things up. The raw data looked like this:

(I deliberately scrambled the order of the rows so sorting by speaker would be less trivial…)

This exists as a Google Sheet, and Gemini (which I have access to through my work account) has an option to import files from Drive, so I initially attempted that, which didn’t work, generating a message saying that it didn’t get the file. This was the same thing that happened the first time around, so I tried it again:

And that, folks is a complete hallucination. Wrong names, wrong number of columns, made-up comments. Last week, I tried it a second time and got more of the same hallucinated nonsense, which is when I gave up.

This morning, pointing out the error got a different response:

Okay, that’s progress, and consistent with my suspicion last week that the “import from Drive” option doesn’t actually work properly².

Given that, I decided to try a direct upload instead, and got this:

That’s more like it, but the preview has the wrong number of columns. The actual .csv file, though, is a bit better. Here’s a side-by-side comparison with the original (sorted by Excel):

Original file on the left, “AI” sorted version on the right

They’re not in the same order, which suggests that Gemini did some additional sorting (by what, I’m not sure), but otherwise it seems to have done the right thing— there are the right number of rows per speaker, and corresponding answers have stayed together. It’s good enough for my general purpose, so let’s see how it does with the next step:

Again, we seem to have gone somewhat astray, here— the counts of entries are way off. The actual files, though, aren’t bad:

Individual file on the left, full sorted data on the right

This is, in fact, just the five entries for this one student split off into their own .csv file. And, to be extra fair, when I pointed out that the counts were off, it corrected the issue:

And that is approximately the thing I was looking for. Not really optimal from the perspective of what I need to do with these— it would be nicer if the Drive integration actually worked so these were files I could directly share with the students. Even nicer would be something that just sorted and split the data and then directly emailed them out, but again, I need to review these myself, so integrating that into the process is not necessarily a Bad Thing even if it slows things down.

So, to answer the question in the post title: How useful is the Bag of Words³? Meh.

It’s not un-useful, in that it did converge moderately quickly to the thing I wanted, but it took a little while to get there, and a non-trivial amount of checking the results at each step. For this class size, it’s probably at least competitive with doing it myself by hand; for a significantly larger class, even the multiple rounds of prompts might be a net win. But then, I’m not sure I would do this with a significantly larger class in the first place…

So, I wouldn’t call this a revolutionary development, by any means. If I were more comfortable with Python for manipulating data files I’d probably sneer at it as slow and awkward. I’m not wild about the errors along the way, especially the ones that were just errors in the reporting not the generated files, but fixing them wasn’t too difficult.

On the other hand, if I were significantly less comfortable dicking around with spreadsheets, I’d probably find it much more impressive. I suspect I’m squatting in a sort of uncanny valley, here, where I know enough to be unimpressed with its performance at simple tasks, but don’t have a need to do things that are complicated enough for it to be impressive.

So, that’s my initial experiment with the roiling cauldron of linear algebra being sold as AI these days. Probably not going to become a regular part of my workflow, but if I start to end up with more tasks in this vein, I would consider giving it another shot.

---

And that’s the kind of squishy lukewarm reaction that is my signature as a blogger. If you want more of this for some reason, here’s a button:

And if you want to suggest ways to do this better or berate me for going over to the Dark Side, the comments will be open:

1. Yes, this could be done much more efficiently with a bit of code, but it would’ve taken me longer to put that together than just cut-and-pasting did. I also needed to skim-read the individual comments to get a sense of how they aligned with my own impression of each talk, which adds some time.

2. Which, in turn, is 100% consistent with my past experience of using Google products for work…

3. OBBetteridge: No.