After the EU AI Act’s transparency guidelines went into effect, and following the announcement that most major AI labs will soon be watermarking their text output to comply, there has been some debate about how much of an impact this is going to have, if any, on the quality of the text produced by AI models.
Since the focus of AI labs shifted to code production, I think the written output of the models has gotten much worse.1 But it’s still interesting to try to understand if watermarking could make it even worse.
I found the discussion around technology writer John Gruber’s recent post, Anthropic’s ‘Watermark’ Text Adulteration in Claude Is a Perversion of Writing, very interesting.
On watermarking, Gruber wrote:
It’s unacceptable for a tool to sacrifice an iota of clarity, coherence, meaning, quality, etc. for the purpose of embedding hidden clues within the text to suggest its provenance.
He got a lot of pushback, with many debating how you would even measure quality, or define it.
But some argued that watermarking does not affect quality at all: it’s essentially just using different random numbers to generate the text. Daniel Jalkut, whom Gruber linked to in his follow-up, published a piece titled AI Can’t Adulterate its Own Writing, in which he wrote:
…there are also ways of tweaking the algorithm that would not diminish its randomness. For example if a random number generator routinely reversed the order of the digits in a generated number, the randomness would remain the same.
Similarly, in another discussion, someone described it as essentially swapping two faces of a die.
In a follow-up piece, Gruber wrote:
I hope that’s true. I believe it’s possible that it is true. I think it’s highly unlikely that it is true.
I think Gruber is right.
Quality loss across responses
Anthropic make a bold claim: “Watermarking does not impact the quality of Claude’s output.” The paper they link to and based their design on (Google’s SynthID-Text, published in Nature in 2024) assures everyone that the method “preserves response quality”.
So it’s easy to assume there’s been a lot of fuss over nothing, that they really are just swapping some random numbers around.
But their no-quality-impact claim, which they tested with humans, is based on single responses. Not the way millions of people actually use these systems: as back-and-forth conversations, over time, over multiple prompts and responses.
The 56-page supplementary PDF they link to at the bottom provides some more info on the method used, and it suggests the no-quality-impact claim isn’t telling the full story. Notably, it says that there is a “diversity/detectability trade-off”:
While single-sequence non-distortion guarantees the quality of each individual response, it does not necessarily preserve diversity across multiple responses
Okay, so what does that mean? They go on to explain:
when sampling several responses to the same prompt, the similarity between the responses is greater for the watermarked responses than the unwatermarked responses.
Now, if they had truly only swapped one set of random numbers for another, we would not be seeing any such difference compared to the unwatermarked responses.
They go on to write:
This could be problematic in scenarios where inter-response diversity is important, or could lower the overall quality of a system which generates many responses then selects the best one
Okay, so in their own words, it can “lower overall quality” after all. But nothing to worry about, because this would only concern you if you cared about “inter-response diversity”, for which they only provide one example: a system which generates and selects responses.
But “inter-response diversity” affects how most people use these systems too. Especially for writing tasks where you want creativity.
In response to Gruber’s point about sacrificed quality, some argued that the model has no notion of the best next word to use for what each individual is trying to convey in their specific work, and so we shouldn’t get too hung up on the next word choice. But then surely they’d agree that you, the individual, would know if it’s suitable or not once you see it.
Let’s say you prompt: “I don’t like the ending to this sentence. Please suggest a new one: [sentence].” The model suggests an alternative, but you don’t like it. No problem, hit the retry button. Well, now you’re affected by watermarking, because you have hit the “inter-response diversity” problem: like throwing a loaded die, you’ll see similar responses more than you would with an unwatermarked model.
Natzir Turrado, a technical consultant, actually tested this (article in Spanish) by reproducing SynthID-Text on an open model with the paper's settings and a watermark key. He asked the watermarked model five times to describe a memorable trip to a coastal town. Four of the five answers he got back opened in a near-identical way: “My heart still sings/dances when I think back to my trip to the quaint coastal town of…”. For the unwatermarked model, three opened in different ways.2
And though they don’t appear to have tested for it, the same thing that makes answers to the same prompt look similar could happen across different prompts too. The watermark only looks at a small sliding context window when choosing the next token, so any time two pieces of writing pass through the same short token sequence, they could be recreating the same context window, leading to a more predictable and less diverse output. And writing is full of identical short phrases that could trigger this behaviour.
Here’s another example of what this affects: one way people try to determine the confidence level of a response, to lower the chances of hallucinations, is to repeat the same prompt a number of times, to explore the different pathways the model takes. They then look at the results to see where the consensus lies. Well, this is also affected. With watermarking, you’re less able to trust the consensus you see forming, because the answers are now less independent: they’re steered by the watermark key.
What’s interesting is that while the SynthID paper points to the loss of quality across responses, I couldn’t find anywhere in the paper where they tested this particular aspect of the watermarking method with humans. If they didn’t, the wonderful human test results they point to did not involve testing the most affected part of the watermarking method.3
So it seems to me that they are sacrificing quality, by limiting the creative range of their models through watermarking, and they don’t think you’ll notice or care much.4
A pattern Jasmine Sun found looking at why AI models don't write well. Here’s what Katy Gero, a poet and computer scientist, had to say:
The same whimsicality that made GPT-2’s voice fresh also made it prone to other unpredictable behavior. “If you’re a big corporation like Google or OpenAI, you want a chatbot that’s going to make money. The chatbot that’s not going to make you money is the one that’s a weirdo,” Gero said.
Natzir Turrado pointed me to a reply he received on X from one of the SynthID paper’s authors, Sumanth Dathathri:
There's a lot of easy fixes to the diversity issue. I think if you tried the same prompt with Gemini/Claude many times, you'd get very diverse responses :)
It’s a bit ambiguous, but I wonder if his “easy fix” is shifting the problem to the user. It reads to me like he’s acknowledging that they’ve replaced the previous die with a loaded one, but the solution is just to roll it “many times” (my emphasis), and it’ll eventually land on each of its faces. If that is what he’s saying, why not admit that such a change does make the system worse than it was before for the user?
Turrado responded: “Curious: which easy fixes don’t trade off detection strength?”
No reply from Dathathri so far.
A recent paper recognised that this is an “understudied fundamental trade-off in LLM watermarks: quality-diversity-power, i.e., the trade-off between the quality of the watermarked text, the output diversity under the watermarking scheme, and the overall watermarking signal strength.”
There are ways to mitigate some of the effects to make them less noticeable to an individual using a chat app, e.g. by rotating watermark keys between each of their requests. But these are not fixes. They only dilute the effect, and have their own costs, e.g. making watermarking less effective on shorter texts, and more costly to detect. As another study pointed out: “achieving a perfect distortion-free watermark is impossible”.