Ideas

Q&A

A critic’s second thoughts on robo-writing

shutterstock/h.hopp-bruce/globe staff

For more than a decade, Les Perelman has battled the rise of the robo-readers — the automated essay-scoring engines used for many standardized tests. Perelman, the recently retired director of MIT’s Writing Across the Curriculum program, has denounced scoring engine reliance on proxy measures for essay quality, such as word count, sentence length, and the frequency of 50-cent words. He has publicly gamed their algorithms by submitting rambling essays of gibberish peppered with fancy words that scoring engines reward with high marks and boilerplate praise.

And when scoring engine developers, such as Pearson and ETS (Educational Testing Service), adapted their algorithms to make instructional software for student writers, Perelman went after those programs, too. So, it was surprising to learn that Perelman, scourge of automated writing assessors, has joined their ranks.

He is now the chief research scientist for WriteLab, a startup company based in Berkeley, Calif., that launched its Web-based writing tutorial last month, partnering with nearly three dozen college writing centers in a nationwide rollout.

Advertisement

In a recent interview from the cluttered basement office of his home in Lexington, Perelman discussed his change of heart. Joining the conversation via Skype were WriteLab’s chief executive officer, Matthew Ramirez, a doctoral student of English at the University of California Berkeley who has a background in computer programming, and Donald McQuade, a Berkeley professor of English who is WriteLab’s chief learning officer.

Get Truth and Consequences in your inbox:
Michael A. Cohen tekes on the absurdities and hypocrisies of the current political moment.
Thank you for signing up! Sign up for more newsletters here

Below is an edited excerpt.

IDEAS: Les, you’ve spent years arguing that computers can’t tell good writing from wordy nonsense. Why join a startup offering automated writing instruction?

PERELMAN: Because I’m not a luddite. In November 2014, I attended a presentation that Don and Matthew gave in Washington, D.C., at the National Council of Teachers of English conference. I went there thinking, “Oh, boy, another one to shoot down.” But, by the end of it, I was convinced that they were doing it the right way.

I’m making full disclosure. I have stock options in WriteLab. I am an interested party. But the reason I got involved in this venture, risking my reputation, is because if we don’t do it well, other people are going to do it badly.

Advertisement

IDEAS: What’s the difference?

PERELMAN: The difference is that we’re building off what a computer can actually do well. Most automated writing tutors are basically retrofitted essay-scoring engines, based on software designed for standardized tests. But WriteLab started with basic principles of good writing.

IDEAS: Which writing principles do you mean?

RAMIREZ: Our starting point is what’s known as the practical style — going for strong punchy verbs, getting subjects and verbs close to the start of the sentences, transitioning cleanly from old to new information, and writing concisely.

IDEAS: Don’t you still run up against the limits of a computer to understand a piece of writing?

Advertisement

PERELMAN: The important thing is that it’s all suggestions and questions. So, if you, as the writer, use a lot of adverbs, the program might pop up and ask how effective a particular sentence might be without the adverb. Or if you use the passive voice, it will ask, if you know the actor in the sentence, could you try the active voice and see if that’s better. We’re trying to help students revise by making them more aware of their prose.

RAMIREZ: For most students I’ve met, writing is something they feel they’re good at or not good at. They’ve made decisions about themselves. I’m a math person, or I’m a writing person. They don’t see it as the kind of thing they could even practice. And they don’t practice in the typical college class where it takes weeks to get writing back with feedback from the teacher. By that point, the student has forgotten most of what he did the night before the due date to turn the thing in on time. He has already received a grade and isn’t really interested in continuing to work on that draft, because the high-stakes period of fear and trembling has passed.

McQUADE: I want to emphasize this idea of practice. My metaphor for this is sports, because I spent a lot of my life either swimming or playing water polo at a very high level. Once you’ve practiced something sufficiently, you begin to internalize the kind of moves, whether it’s in the pool or the moves that you make as a writer. You become more comfortable and confident, to the point that when you are trying to articulate something, you make these moves almost automatically, because you’ve practiced it often enough to relax into your own eloquence.

IDEAS: A writer’s goals and stylistic choices will differ substantially between an expository essay, a short story, a newspaper article, a wedding toast, or a scientific paper. How do you take that into account?

McQUADE: This is a tool that encourages writers to make decisions. The machine learns from those decisions. The principle is that you start where the writer is able. Rather than saying here’s the standard, and you have to meet it, we took what they were giving us and we built off that.

I’ve spent many years teaching writing and researching feedback to student writing and struggling to find ways to effectively assist a writer to articulate herself in a way that’s important to her. It’s about decisions. It’s not about correcting. Or correctness.

IDEAS: Do you expect writers to reject a lot of WriteLab’s suggestions?

RAMIREZ: Absolutely. We want writing suggestions to be personalized, and useful. Instead of trying to get at that with rules, we take advantage of having a lot of users giving us a lot of feedback on the software’s suggestions. That allows us to refine the software features at a high level using recent developments in cloud computing that enable this sort of real-time, machine learning.

The best analogy would be Google search. The recommendation system that appears when you start typing is based most heavily on your recent searches. But it’s also affected by what everybody else is searching for. It’s the same thing here. Maybe you like to use the passive voice to manage long clauses and you reject suggestions to switch to the active voice. Does the larger pool of people also reject these suggestions in this particular context? If not, the software will lower the probability of making this suggestion for you in the future, but less than it would if the masses of people were also rejecting the suggestion.

IDEAS: But what about “the period of fear and trembling” — there are limits to self-editing.

RAMIREZ: The software serves as a supplement to a teacher, rather than a replacement. It enables students to turn in prose that’s much more refined, but not by any means finished. So when a student goes through the system and says this draft is now ready to be read by an instructor, what they’re saying is, I’ve done as much as I can, I’m ready for your feedback. There’s a deep psychology to feedback on student writing. For example, in beta testing, the students really gave feedback on our suggestions in defense of their choices. This was terrific. They took the opportunity to articulate their position in a way they probably wouldn’t have the courage to do to a human instructor.

If you tell most college students they should make a change to their writing, then they’ll make that change. They’re not thinking about what it is they want to do with their writing if it seems it’s at all to the teacher’s displeasure. But this software is something that they’re totally comfortable fighting, because people are very much of the mindset that technology is there to serve them and not the other way around.

Chris Berdik is a freelance writer in Milton.

Related:

Why we need to learn to trust robots

2014 | Les Perelman: Flunk the robo-graders

2014 | Joanna Weiss: The man who killed the SAT essay

2013: Will a robot take your kid’s job?