Rendered at 05:37:55 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
viraptor 1 days ago [-]
It's an interesting review, but it seems to mix some things together. Gruber's claim is that we shouldn't sacrifice quality, but none of those sources show that's what's happening. Having less diversity in answers could mean you're more consistently getting a better/worse/same quality answer. There are lots of people complaining about not being able to get a real temperature=0 answer - they'd actually welcome the consistency too.
The practical example of not liking the sentence is not great in my opinion. Instead of just re-rolling the answer, why not continue in that thread and say what you don't like? Or request 20 alternatives, say which go in the right direction and request 20 more? Retrying from scratch with one sample is a waste of time and doesn't indicate anything about quality.
Basically, it's all theoretical - show us the actual benchmarks/samples where the quality is consistently degraded.
k1m 15 hours ago [-]
> Having less diversity in answers could mean you're more consistently getting a better/worse/same quality answer.
I wrote the article. I'm trying to show how diversity loss is itself a measure of quality loss in many applications. And as I pointed out in another comment, the paper's authors themselves acknowledge as much in their paper.
> Instead of just re-rolling the answer, why not continue in that thread and say what you don't like?
The re-roll is actually a feature of many chat apps with a dedicated button. So it's something users are familiar with.
But the results are also quite different. The prior answers in your example of asking for x alternatives influence the following answers too much in a way a re-roll doesn't.
viraptor 9 hours ago [-]
The paper is pretty clear that (non-distortion) watermarking doesn't impact the quality of responses, but may change the quality of the system. Those are very different things and the paper treats them as such.
On the other hand when you quote that, you drop the "system": «it can “lower overall quality” after all». And make unsupported claims from that: «But “inter-response diversity” affects how most people use these systems too.» - which could both use a citation and doesn't show is a practical problem.
> The prior answers in your example of asking for x alternatives influence the following answers too much in a way a re-roll doesn't.
They're influenced to not do what the last answer did which is the whole point. Can you define "too much" or show why it's an issue?
ethanj8011 1 days ago [-]
> Natzir Turrado, a technical consultant, actually tested this (article in Spanish) by reproducing SynthID-Text on an open model with the paper's settings and a watermark key. He asked the watermarked model five times to describe a memorable trip to a coastal town. Four of the five answers he got back opened in a near-identical way: “My heart still sings/dances when I think back to my trip to the quaint coastal town of…”. For the unwatermarked model, three opened in different ways.
A test of 2/5 vs 4/5 seems to be an obviously invalid experiment.
dannyw 1 days ago [-]
The linked article in Spanish contains substantially more detail. Natzir Turrado tested hundreds of prompts.
For this example; he tested 30 prompts at 5 pass-pairs each; so 300 times.
Thanks. Looks like it does line up with the original findings, however I do feel pretty confident that this could be ameliorated using slightly more complex watermarking strategies. (I do think that overall watermarking is not an effective governance solution for text)
cowlby 1 days ago [-]
This already happens as Claude watermarks all my content with "You're half right, and the half that's wrong is the useful part".
oidar 1 days ago [-]
"X is the useful part" I hate this claudism. But a good thing about recognizing these is that you can pretty much skip any youtuber that uses any claudish phrasing. But it's damn near everyone now a days using claude to write their scripts.
zdc1 23 hours ago [-]
In the post-Claudism future, we'd need a transcription -> watermark detection workflow to tell who writes their own stuff
cleandreams 1 days ago [-]
It's good news. All text output should be watermarked and it should be serviceable but not exceptional or "creative."
Benjaminsen 11 hours ago [-]
AI models are incredibly bad at generating high-quality, diverse outputs as a function of how they work.
To the point that I built modelsalt.com in an attempt to create a structured solution to get more diverse outputs without quality loss.
k1m 10 hours ago [-]
That's really interesting! Thanks for sharing. Will be trying it soon.
Majromax 17 hours ago [-]
This article is wrong. SynthID-style watermarking does not affect one-shot generation quality. When news about the watermarking went around a couple of months ago I too thought there must be a quality/detectability tradeoff (as there is for images), but I dug into the math more and realized that my intuition was incorrect.
They key idea is this: changing the random seed for LLM generation definitionally does not change the distribution of tokens in expectation over all seeds.
This can already be used to embed information in LLM streams, leading to LLM steganography work (https://arxiv.org/abs/2606.09135, but the idea dates back much further). Take your target message, whiten it via a one-time pad (stream cypher), then use it as the 'compressed stream' under an arithmetic decoding process (https://en.wikipedia.org/wiki/Arithmetic_coding) with the LLM-supplied logits as the assumed probabilitiy distribution.
SynthID pulls off the same kind of trick, save that its 'tournament structure' allows application after the sampling step rather than before. The random seed under SynthID becomes context sensitive (keying on the prior N tokens as well as the watermark seed), but in expectation the next-token distribution is still the same as the unwatermarked case.
If there's no change in token distribution, then there can't be a change in quality.
However, the article and its sources are right about one factor: random generation diversity will suffer. Applying SynthID to an LLM stream acts like fixing the PRNG seed. Repeated generations with the same prompt and same seed will deliver the same result, up to small amounts of nondeterminism in the architecture (mostly GPU scheduling and roundoff). This matters under the same circumstances that it would matter if the provider said "we sample using the XYZ PRNG with fixed seed '42'".
In my opinion, 'sample diversity' and 'sample quality' are not the same thing, and conflating the two concepts leads to poor intuition. SynthID and the like probably matter most if one is trying to repeatedly sample an LLM in order to infer the probability distribution (e.g. using a commercial LLM provider as a Jev-style classifier, looking for calibrated answers).
k1m 16 hours ago [-]
I wrote the article. How you define quality is exactly the issue.
> In my opinion, 'sample diversity' and 'sample quality' are not the same thing, and conflating the two concepts leads to poor intuition.
But you're assuming everyone has the same notion of quality as you. As I point out in the piece, even the paper's writers use "quality" to show how watermarking can "lower the overall quality of a system"
> SynthID and the like probably matter most if one is trying to repeatedly sample an LLM in order to infer the probability distribution
Which is another impact I write about in the piece. If you evaluate the output quality of an LLM system in this way, you're impacted more by watermarking.
Majromax 8 hours ago [-]
> I wrote the article. How you define quality is exactly the issue.
I agree, but I think you're using an unintuitive definition of quality. Since SynthID essentially acts as a seeded RNG, you will not be able to tell the difference between natural output and SynthID outputs with one sample per prompt.
We wouldn't be talking about "quality loss" if a model provider just ran `rng.seed(user_id)`, but it would have the same user-visible outcome.
24 hours ago [-]
octoberfranklin 1 days ago [-]
The gaslighting by watermarking proponents is so off the charts.
The practical example of not liking the sentence is not great in my opinion. Instead of just re-rolling the answer, why not continue in that thread and say what you don't like? Or request 20 alternatives, say which go in the right direction and request 20 more? Retrying from scratch with one sample is a waste of time and doesn't indicate anything about quality.
Basically, it's all theoretical - show us the actual benchmarks/samples where the quality is consistently degraded.
I wrote the article. I'm trying to show how diversity loss is itself a measure of quality loss in many applications. And as I pointed out in another comment, the paper's authors themselves acknowledge as much in their paper.
> Instead of just re-rolling the answer, why not continue in that thread and say what you don't like?
The re-roll is actually a feature of many chat apps with a dedicated button. So it's something users are familiar with.
But the results are also quite different. The prior answers in your example of asking for x alternatives influence the following answers too much in a way a re-roll doesn't.
On the other hand when you quote that, you drop the "system": «it can “lower overall quality” after all». And make unsupported claims from that: «But “inter-response diversity” affects how most people use these systems too.» - which could both use a citation and doesn't show is a practical problem.
> The prior answers in your example of asking for x alternatives influence the following answers too much in a way a re-roll doesn't.
They're influenced to not do what the last answer did which is the whole point. Can you define "too much" or show why it's an issue?
A test of 2/5 vs 4/5 seems to be an obviously invalid experiment.
For this example; he tested 30 prompts at 5 pass-pairs each; so 300 times.
https://natzir.com/posicionamiento-buscadores/que-es-como-fu...
To the point that I built modelsalt.com in an attempt to create a structured solution to get more diverse outputs without quality loss.
They key idea is this: changing the random seed for LLM generation definitionally does not change the distribution of tokens in expectation over all seeds.
This can already be used to embed information in LLM streams, leading to LLM steganography work (https://arxiv.org/abs/2606.09135, but the idea dates back much further). Take your target message, whiten it via a one-time pad (stream cypher), then use it as the 'compressed stream' under an arithmetic decoding process (https://en.wikipedia.org/wiki/Arithmetic_coding) with the LLM-supplied logits as the assumed probabilitiy distribution.
SynthID pulls off the same kind of trick, save that its 'tournament structure' allows application after the sampling step rather than before. The random seed under SynthID becomes context sensitive (keying on the prior N tokens as well as the watermark seed), but in expectation the next-token distribution is still the same as the unwatermarked case.
If there's no change in token distribution, then there can't be a change in quality.
However, the article and its sources are right about one factor: random generation diversity will suffer. Applying SynthID to an LLM stream acts like fixing the PRNG seed. Repeated generations with the same prompt and same seed will deliver the same result, up to small amounts of nondeterminism in the architecture (mostly GPU scheduling and roundoff). This matters under the same circumstances that it would matter if the provider said "we sample using the XYZ PRNG with fixed seed '42'".
In my opinion, 'sample diversity' and 'sample quality' are not the same thing, and conflating the two concepts leads to poor intuition. SynthID and the like probably matter most if one is trying to repeatedly sample an LLM in order to infer the probability distribution (e.g. using a commercial LLM provider as a Jev-style classifier, looking for calibrated answers).
> In my opinion, 'sample diversity' and 'sample quality' are not the same thing, and conflating the two concepts leads to poor intuition.
But you're assuming everyone has the same notion of quality as you. As I point out in the piece, even the paper's writers use "quality" to show how watermarking can "lower the overall quality of a system"
> SynthID and the like probably matter most if one is trying to repeatedly sample an LLM in order to infer the probability distribution
Which is another impact I write about in the piece. If you evaluate the output quality of an LLM system in this way, you're impacted more by watermarking.
I agree, but I think you're using an unintuitive definition of quality. Since SynthID essentially acts as a seeded RNG, you will not be able to tell the difference between natural output and SynthID outputs with one sample per prompt.
We wouldn't be talking about "quality loss" if a model provider just ran `rng.seed(user_id)`, but it would have the same user-visible outcome.