Temperature

Contra a bunch of idiots at Hacker News and elsewhere, I understand that popular LLMs do not just pick the “best” token (word) at each decision point. Counterintuitively, always selecting the highest-probability option produces undesirable results. So the models apply some randomization, and “temperature” is the term for the weighting that’s applied so that the “better” (higher-ranked by the model) choices have a higher chance of being chosen.

With a temperature of 1, models use their built-in probability distribution. With a temperature greater than 1, this distribution gets flatter — less-likely alternatives get a higher probability of being selected, and more-likely alternatives lower. With a temperature lower than 1, the probability distribution leans more toward the higher-ranked options. And with a temperature of 0, the highest-ranked option is always chosen. A temperature of 0 generally produces undesirable results — too predictable, too likely to get stuck. Like over-smoothing an image from a camera sensor, eliminating all noise makes the overall result worse, even if each single bit of “noise”, evaluated in isolation, is in some sense wrong.

The temperature-based randomness — which is what makes LLM output non-deterministic — is in place to help make the output better. The prose is clearly better with a temperature of 1 (with weighted randomness) than at temperature 0 (with no randomness). The watermarking schemes, on the other hand, are applying predictable-with-the-secret-key randomness for an entirely different purpose than improving the quality of the output, and thus, I believe, inherently make the output at least slightly worse.

Advocates of LLM watermarking schemes for text argue that the schemes don’t necessarily lower the quality of the generated prose, because they don’t change the temperatures — they only change the source of the randomness. Daniel Jalkut wrote a good piece today about this. I hope that’s true. I believe it’s possible that it is true. I think it’s highly unlikely that it is true. I do not see how a detectable signal can be added encoded in the choice of words without affecting the meaning of the prose. If it were true I think they’d show examples proving that it’s true. Also, Anthropic itself admits that it can’t properly watermark text that is programming language code:

For the same reason, code — which in very many cases has to be exact — has generally less watermarking than some other forms of text.

Having said that, in areas where there is an arbitrary choice between particular words or terms within the code, the watermark can be used, such as comments within code. But by definition, it will have a negligible effect on the actual code produced.

I hold that good prose is much more like programming code. Exactness in word choice, phrasing, tone, and even punctuation is always better than imprecision. The difference is that sloppy programming code doesn’t run, or doesn’t run correctly. The human brain, on the other hand, is adept at parsing and making sense out of inexact, even sloppy, prose.

I Object Even If Quality Isn’t Adversely Affected

I do not believe these schemes can work without degrading prose quality, if only slightly. Again, though, I am open to being proven wrong. But even if we concede for the moment that such watermarking schemes do not necessarily degrade the quality of generated prose — not one iota — I still object to their use when they are being applied secretly, behind users’ backs. A useful watermark would be one that anyone can check. These SynthID “watermarks” are entirely dependent upon secrets held by the LLM providers (so far, Anthropic/Claude and Google/Gemini). I find that unacceptable, for reasons I hopefully made clear in my essay.

The people in favor of this watermarking for text have been sold a pipe dream, a fantasy. I’ve encountered dozens of comments from angry AI haters (many of them on Bluesky in particular, but also Threads and Hacker News) who are convinced that the only people who could be against the watermarking of AI-generated text are those who are duplicitously passing off AI-generated text as their own writing — and thus that I must be upset only because the jig will soon be up for me too. This of course is not true. I don’t even use AI to write text messages or emails for me, let alone a single sentence of my work.

But I find it funny that so many people who claim to believe that LLMs only produce “slop” and never anything useful also seem 100 percent convinced that the same LLMs are capable of watermarking their output in reliable ways. These people so desperately want to be able to point a finger at AI-generated text that they’ve fallen hook, line, and sinker for the argument from Google and Anthropic that, thanks to them, they’ll be able to.

I don’t want to spend too much time thinking about this because it’s a waste of time, but how exactly do these people think the existence of these mandatory watermarks and detection tools will change anything for the better? Let’s say you work at an office and you suspect that numerous of your colleagues are using AI to write emails and other work-related messages. Their messages are too long, too prolific, and lack lucidity. What are you going to do now? Copy and paste each of their messages into the watermark detectors from Anthropic, Google, and OpenAI? There cannot exist a single detector for all LLMs. And even if you find out that it says it’s a match, that an email or blog post or Slack message was very likely generated by, say, Claude, what are you going to do? March into your colleague’s office and tell them you caught them?

Anyone in a situation where “getting caught” would matter — students, say — is going to use non-watermarking LLMs or run their watermarked text through paraphrasing tools like Declaude.

No practical good is going to come of this, even if these watermarking schemes work as promised (and to be clear, I don’t believe any of it is going to work as promised).¹

My advice is not to care whether anything was written by an AI or a human. The only thing worth evaluating is what we human readers are naturally good at determining: whether it is good or bad. If it’s good, read it. If it’s not, don’t. If you’ve got a job where you’re surrounded by colleagues filling your inbox with AI-generated messages that you can’t abide, get a new job or learn to live with it. Hidden secret watermarking signals — even if they work — aren’t going to make things go back to the way they used to be. If you read something and enjoy it, and subsequently find out it was generated by an LLM, don’t feel bad. You read something good that you enjoyed.

I read something earlier today that claimed most of the posts on LinkedIn are generated by AI. That the whole platform is just inundated with AI slop. Maybe it is, but I wouldn’t know, because I never look at LinkedIn because it’s always been filled with crap. If it smells like crap it’s crap, whether the turds came out of a human anus or a turd-generating robot.

The Argument That Only People Can Truly Write

Dan Moren, writing at Six Colors today, “LLMs Aren’t Writing”:

LLMs do not care about the words that they pick because they cannot care about anything.

Speaking of two things that are not the same, John rightly points out the difference between the phrases “he leaped at the chance” and “he jumped at the opportunity”. Those are indeed distinct — if semantically similar — phrases, each of which might be more apt in a particular situation; or, to put it in another fashion: the use of each of those phrases tells us something different, whether about the person being described or the writer.

But the LLM doesn’t know which of those phrases is the right phrase to use. It has a guess, based on its models and weights and inputs. But the ultimate choice of those phrases tells us nothing about the writer because there is no writer.

Moren’s is a fine retort to my post, but I fundamentally disagree — albeit at a philosophical level. If you’re reading a written work only to gain insight into the mind that produced it, there is no mind on the other end of AI-generated text. But the work itself exists. My disagreement with Moren starts and effectively ends with his (wonderfully summative) headline. I say if you can read something, it was necessarily written.

Again, this is philosophical. Was a photorealistic image generated by AI photographed? No, I would say it was not. Photography, I would say, is the act of focusing light through a lens onto a capturing sensor, capturing, to some extent, reality. I think Moren is arguing that writing is like that. If photography captures a physical scene from reality, writing captures thoughts from an actual mind. That something you can read that was produced by an LLM was merely generated in a way that doesn’t qualify as writing. Semantics. I just care about the article of text. Moren argues that LLMs are not writing; I say they are. But we’re disagreeing only over what the word writing means, not what is being produced.

As for “caring” about the difference between semantically similar but tonally different phrases, like “he leaped at the chance” versus “he jumped at the opportunity”, no, of course the LLM doesn’t “care”. But I, the reader, care very much. I wrote a column back in November on ChatGPT changing (and renaming) the “personalities” it allows users to choose from. These personalities generate text with strikingly different styles and tones. Because I use ChatGPT, I care very much about the tone and style of its responses to my queries. Not because I’m ever going to pass them off as my own writing, but because I’m the one who is reading them.

Moren, near the end of his column:

In the end, I can’t summarize it any better than to ask: if you care so much about word choice, why are you using AI to generate text?

If this does truly make AI-generated text worse, well… good. A lot of people are already willing to accept what an LLM churns out as “good enough” and, if I’m being realistic, I don’t think this will change anything. But if it does lead to more people being dissatisfied with the pablum they’re being fed and turning instead to writing and editing their own text, then that would actually be a positive outcome. Maybe it’d even mean fewer human writers being put out of jobs.

I sympathize, but I must disagree that it can possibly be seen as a net good for LLMs to produce worse prose. I read the output of LLMs every day. I use AI to generate text because I ask it questions (in text). I want the answers that I read to be cogent, lucid, accurate, blessedly terse — and ideally to strike a consistent tone that is pleasant to my reading ear. The genie is not going back in the bottle.

English Is the Finest Language, and Thus, Perhaps, More Fingerprintable

Lastly, here’s an interesting point to ponder. English is the most expressive language in the world. Don’t take my word for it — it’s the only language I speak (despite four years of Spanish in high school). Take the word of famed 20th century author Jorge Luis Borges, an Argentine polyglot whose first language was Spanish. In 1977 he was the guest on William F. Buckley’s “Firing Line”. You can (and should) watch the interview on YouTube, but here’s a transcript of the relevant portion from Jordan M. Poss:

Borges: I have done most of my reading in English. I find English a far finer language than Spanish.

Buckley: Why?

Borges: Well, many reasons. Firstly, English is both a Germanic and a Latin language. Those two registers — for any idea you take, you have two words. Those words will not mean exactly the same. For example if I say “regal” that is not exactly the same thing as saying “kingly.” Or if I say “fraternal” that is not the same as saying “brotherly.” Or “dark” and “obscure.” Those words are different. It would make all the difference — speaking for example — the Holy Spirit, it would make all the difference in the world in a poem if I wrote about the Holy Spirit or I wrote the Holy Ghost, since “ghost” is a fine, dark Saxon word, but “spirit” is a light Latin word. Then there is another reason. The reason is that I think that, of all languages, English is the most physical of all languages.

Buckley: The most what?

Borges: Physical. You can, for example, say “He loomed over.” You can’t very well say that in Spanish.

Buckley: “Asomó?”

Borges: Well, no, no, they’re not exactly the same. And then you have, in English, you can do almost anything with verbs and prepositions. For example, to “laugh off,” to “dream away.” Those things can’t be said in Spanish. To “live down” something, to “live up to” something — you can’t say those things in Spanish. They can’t be said. Or really in any Romance language.

I’ve seen this interview before, but watched it again today after an email exchange with Kirk McElhearn. Quoting (with permission) from McElhearn’s email to me:

For many years, I worked as a French → English translator, and there is one key difference between the two languages. France is a Romance language, and English is a language with both Germanic and Romance (mainly French) influence. This means that English often has synonyms where other languages may not.

Using your example, “He leaped at the chance” and “He jumped at the opportunity”, both would be translated in French as “Il a sauté sur l’occasion.” Meaning that someone writing in French wouldn’t have the same range of words to choose from. It’s maybe not the best example, because both are clichés, but there are many examples of French words where English has both a Romance equivalent and a Germanic equivalent: pig and pork, sheep and mutton, beef and cow. Food words are just one example, but English also has many more verb choices than French, since it has a larger vocabulary coming from both influences.

English gleefully borrows from any and all other languages. McElhearn wonders whether English is thus more fingerprintable than other languages, because of its richer vocabulary of roughly equivalent synonyms, and its multitude of idioms.

My initial speculation was that maybe they’d hide invisible non-printing Unicode characters in the text. Just spitballing. Turns out that’s not what they’re going to do. What they’re going to do is apply a form of steganography, where the choice of words (or other token output) at inference time will leave fingerprints that can later, maybe, be detected probabilistically.

I initially guessed “invisible characters” not because I didn’t think of the semantic word-choice technique, but because I was a fool who took Anthropic at its word in their description of what they would do. Their original support document claims:

When a supported Claude model generates text, it weaves an imperceptible watermark directly into the text itself. You won’t see it, and it doesn’t change the meaning, quality, or readability of Claude’s response.

They say “imperceptible” and “doesn’t change the meaning, quality, or readability”. Their words. Not almost imperceptible. Not slightly changes the meaning, quality, or readability. That made sense to me, because that’s absolutely what I want — nay, demand — from any tools I use personally. It’s unacceptable for a tool to sacrifice an iota of clarity, coherence, meaning, quality, etc. for the purpose of embedding hidden clues within the text to suggest its provenance. That’s what I would and will demand. And Anthropic’s (original) support document unambiguously claims that’s what their system will enable. So if that were true, I couldn’t see what was left other than hiding invisible characters within the text.

My error was believing Anthropic that their system wouldn’t adulterate and corrupt the semantics of the text their models generate. That is in fact exactly what they plan to do. I should have my head examined for believing a single word of a document titled “How Claude Marks AI-Generated Content” that doesn’t explain, at all, how Claude marks (or will mark) AI-generated content.

How It’s Actually Going to Work

Yesterday, on an entirely different website than the original “How Claude marks AI-generated content” article (the one that didn’t explain anything at all about how it works), Anthropic published “How Claude’s Text Watermark Works”, which does actually explain in layman-accessible terms how it’s going to work. I will return to Anthropic’s new highly euphemistic and slightly misleading description below.

There’s a bunch of research on this topic, some of which I have also linked to below. But the very best description of the general idea behind the technique is an interactive essay by James Padolsey, “How AI Text Watermarking Works”. It’s a wonderfully cogent read, and the interactive elements splendidly illustrate the main concepts. A+ work. If you have any interest in this at all, I dare say you must read — and play with — Padolsey’s piece.

But here’s my stab at a layman’s high-level summary. If you toss a coin N times and note the results, you can determine with a degree of certainty whether the coin is fair or biased. LLMs are, in their popular incarnations, non-deterministic. Ask the same question of the same model and you often get at least slightly different answers. Maybe the same meaning, but different phrasing. At each decision point for generating the next token, the model makes a choice. With these semantic watermarking techniques, they make different choices for some tokens based on word lists that could be called “green” and “red”. At each decision point, they’re a little more likely to pick a word from the green list than the red list. That doesn’t mean they never choose words from the red list. Just that they’re less likely to than they would if the adulterated marking technique weren’t in place. (Same way that a crooked 51-49 coin will still land “wrong” side up 49 times out of 100 on average.)

Words or word phrases are sorted into the green and red lists deterministically on the fly, at each “next token” generation point. So sometimes a specific word will be on the green list, and other times it will be on the red list. Someone with the secret key can determine which list a word will be on at each token generation point (which is how the watermarking is detected); those without the secret key cannot. This means there will never be a list of words that Claude prefers or eschews.

With coin flipping, the higher N is — the more times you flip — the more confident you can be that the coin is fair or biased. So too with this semantic watermarking. The more words in the text, the more accurate the analysis will be that the text was generated by a specific AI model or not. With too few coin flips, you can’t achieve any confidence at all regarding a coin’s fairness. With too few words (or tokens), there’s no way to achieve any confidence whether a string of text was AI-generated or not.

Given a string of text to examine for signs of a specific watermarking system, if there are more words tagged as green and fewer tagged as red than would otherwise be expected, the text can be flagged — with some degree of confidence — as having been generated, or merely modified, by the AI system that applies the specific secret-key watermarking system. The amount of confidence in the determination will obviously vary, significantly, based on the size of the text string and randomized weights given to words on the green and red lists. But only Anthropic will be able to determine if text was seemingly generated by Claude, and Anthropic will only be able to detect the watermarks that are applied by Claude. Claude can’t detect the hidden watermark signals generated by, say, Gemini, and Gemini can’t detect the hidden watermark signals created by Claude, because each implementation is predicated on secret keys held only by the LLM provider.

Objections to the Technical Premise

One of my fundamental problems with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter. I want any LLM I use to choose the very best, most precise words at every single decision point. An obvious constraint that I accept is time and computation. Within the constraint of executing inference quickly, and at a certain cost per token, I want the best words. This constraint matches human writing. I could surely write a better column by taking longer to write it. I write with a sense of how much care I should put into every word and punctuation choice I make. I take more time with certain paragraphs, sentences, or even individual word choices when my gut feeling says I should.

In other words, these are necessary trade-offs. These factors are all in my interest: speed, cost, quality. Ideally I would like perfect writing, at instantaneous generation speed, at zero cost. None of those things are possible. Computation is not free of charge (and cloud-based LLM inference with leading models is actually expensive). Inference is not instantaneous. And great writing, whether natural or artificial, can only approach perfection.

The idea that anything other than my needs should factor into the generation of text for me is patently offensive.

This isn’t just about text one might generate with the intention of passing it off as their own natural work. This isn’t even about LLM proofreading of work written by hand. Anthropic is saying that all new Claude models are going to adulterate every single bit of text longer than 200 tokens (~150 words) they generate, including everything it presents to its users to read. So even in a private conversation between a user and Claude, which will never be read by anyone other than the user, Claude will begin making word choices in the name of marking its output in statistically predictable ways rather than maximizing clarity and precision.

Even today’s so-called frontier models are already decidedly lacking in lucidity. Claude, ChatGPT, Grok, et al. are “better writers” than most humans and produce better prose than the median human. But: no shit. Most people are terrible writers. The “average person” is pretty stupid and half of all people are stupider than that. And there are many smart, interesting people who are miserable writers. So as impressive as LLMs are, the bar is low. The best writing I see come out of these models is worse than anything I would choose to read for pleasure. And now Anthropic is saying they’re going to make it worse, on purpose, for purposes that do not benefit me in any way? Even if only slightly worse?

Get fucked.

Objections to the EU Regulation

Speaking of objections, the relevant EU regulation motivating all of this, “Code of Practice on Transparency of AI-Generated Content”, is red-tape nanny-state pipe-dream nonsense. Here’s Ben Thompson’s summary from a paywalled Stratechery update this week:

• The regulation applies to text longer than 200 tokens.
• The provider must mandate in their terms-of-service that users not remove the watermarking.
• The solution should be robust in terms of evading “typical processing solutions” like screen shots, scanning and OCR, copy-and-pasting, translations, etc.

Taken literally, compliant LLM terms of service must forbid users from rephrasing the output from models that comply with this regulation, because the word choices are the marks. But it’s not the European Union that is trying to impose their absurd, impractical, witch-hunt-fueling regulation on the entire world. That falls on Anthropic.

Complying with this, particularly with regard to text, is only going to create problems for honest users. Dishonest users attempting to pass off AI-generated text as their own writing (students, employees, whoever) will simply circumvent detection through non-compliant AI paraphrasing tools.

James Padolsey — whose interactive visual explanation of how these schemes work I linked to above — explains this in a post titled “Anthropic’s Weak Watermarks Appease a Weak Law” (which, if it rings a bell, I linked to in a standalone post earlier today):

The same thought that led to this law could have applied to calculators at the time of their inception, had their outputs revealed themselves through artefacts. Thankfully, a sum borne of the brain is treated no differently from one produced by a calculator. Likewise with spellcheckers. To make assistance suspect only once the tool becomes capable enough to compose a whole sentence is not a principled boundary. It is a moral premium placed on difficulty itself.

Anthropic has nevertheless chosen a blanket, model-level implementation that appears broader than the law’s minimum requirement. That may be convenient compliance engineering, but it discards distinctions the law expressly attempted to preserve. The result is a signal broad enough to implicate harmless and assistive use, yet fragile enough to be removed by a motivated person through substantial recomposition. It risks concentrating suspicion on ordinary and assistive users while remaining weakest against deliberate deception.

Padolsey is the creator of Declaude, a delightfully simple web app that allows you to “Paste in AI-flavored text and get the same content back as plain prose”. Declaude’s original purpose is cleaning the saccharine Claude personality stink from text (whether it was created by Claude or any other LLM), but, if Anthropic persists in its stated plan to begin adulterating all text Claude generates, Declaude will also serve as a copy-paste single-extra-step way to eliminates those marks. Declaude is interesting and useful already, but it exemplifies how ill-considered and futile this EU regulation is when it comes to prose.

Google SynthID

Google has a watermarking system in place that they call SynthID, which they apply to AI-generated images, video, audio, and text. I’m concerned in this article only with text. With multimedia, embedded watermarks can be metadata within files, and truly not affect the experiential quality of the work when viewed or listened to. With text, we are talking about the actual words that are chosen. From the “AI-generated text” section of Google DeepMind’s own description of SynthID:

We’ve expanded SynthID to watermarking and identifying text generated by the Gemini app and web experience. Large language models generate text one word (token) at a time. Each word is assigned a probability score, based on how likely it is to be generated next. So for a sentence like “My favorite tropical fruits are mango and…”, the word “bananas” would have a higher probability score than the word “airplanes”. SynthID adjusts these probability scores to generate a watermark. It’s not noticeable to the human eye, and doesn’t affect the quality of the output.

In a group chat, a friend of mine quoted the above, and I responded that if a chatbot wrote “My favorite tropical fruits are mango and airplanes”, I’m pretty sure I’d fucking notice. Another friend then responded with this:

Days later, that still cracks me up.

But Google’s absurd description puts the lie to their own claim that it isn’t noticeable, and it serves to show just how little regard the people behind these generated-text fingerprinting schemes have for the actual craft of writing. Of course bananas has a higher probability score than airplanes, because airplanes aren’t fruit. But what about pineapple? Should the sentence complete to “mango and bananas” or “mango and pineapple”? That’s a good question, and the only acceptable answer for why an LLM should choose bananas instead of pineapple (or coconut, or guava, or papaya...) is that it has determined that it’s the best fit for the intended meaning, tone, and sentiment of the text. Not because bananas is on the watermarking “green” list and pineapple is on the “red” list, even though pineapple might be the better fit. Google’s own supposedly jocular description of how SynthID works in fact captures how the scheme perverts the text it generates.

They’re saying you won’t notice because if it only chooses bananas over pineapple for these fingerprinting purposes, well, they’re both tropical fruits and who cares. But it’s utter nonsense that the difference is “not noticeable to the human eye”. The semantic difference between banana and pineapple is just as noticeable to the human eye as the taste of the two are to the human tongue.

If it did produce “My favorite tropical fruits are mango and airplanes”, it’d be incredibly stupid, but it wouldn’t be offensive because we’d all recognize that something completely off-key happened. What’s offensive is that with a system like SynthId in place, where the fingerprinting decisions are motivated by a secret key, we have no idea whether it completed to “mango and bananas” because bananas was determined to be the best next token, or because bananas is in the “green” bucket of words. It calls every single word choice into question.

Here’s a paper published in Nature where Google’s team behind SynthID published their work, after putting it into production with Gemini (née Bard):

We analysed approximately 20 million watermarked and unwatermarked responses and computed the thumbs-up and thumbs-down rates (both as a fraction of the total number of thumbs-up and thumbs-down feedback received). We found that the thumbs-up rate for the two models differed by 0.01% (with the watermarked model being higher); and the thumbs-down rate differed by 0.02% (with the watermarked model being lower). We found both of these differences to be statistically insignificant, and well within the 95% confidence intervals.

From this experiment, we conclude that over a wide variety of real chatbot interactions, the difference in response quality and utility, as judged by humans, is negligible. Subsequently, non-distortionary SynthID-Text has been productionized and is currently watermarking responses in Gemini and Gemini Advanced. To the best of our knowledge, this evaluation represents the first systematic watermarking investigation of its kind within a large-scale production system.

To this I say:

• Gemini/Bard’s thumbs-up/thumbs-down buttons are not a good experiment for evaluating the effect on quality. If a chatbot tells me “My favorite tropical fruits are mango and bananas” instead of “mango and pineapple”, I’m not going to give the response a thumbs down because of the fruit it chose. I’d give it a thumbs down if it said “airplanes”, yes, but that’s a strawman. (The paper in Nature even uses “My favourite tropical fruit is ...” as an illustration, but in the paper, the only four next tokens considered are, in order of probability distribution, mango, lychee, papaya, and durian. No airplanes. And, conveniently, in the paper’s example, the “winner” of the watermarking “tournament” just happens to be mango, the one that would have been selected as the best if the watermarking weren’t in place.)

• A “difference in response quality and utility, as judged by humans” that is “negligible” does not mean imperceptible. What they really mean is that it’s only slightly worse and that everyone is either too stupid to notice or too indifferent to care.

• It’s widely considered that Gemini is behind ChatGPT and Claude in quality. Perhaps the fact that they’ve put SynthID-text into production is one of many reasons why. I personally agree that Gemini’s prose is inferior. Maybe the use of SynthID has nothing to do with the fact that I, along with the general public consensus, consider Gemini to be a second-rate chatbot — but in that case, maybe it’s the fact that Gemini is a second-rate chatbot that makes the difference “negligible” when Google started mixing in SynthID-motivated tokens in its results. It’s a lot more likely that your restaurant customers won’t notice that you replaced your regular coffee with Folgers Crystals if your regular coffee is second-rate to start with.

Anthropic

Now, finally, back to Anthropic’s new “How Claude’s Text Watermark Works”, published yesterday. I have some comments.

To summarize:

• We use a method of watermarking that does not have any practical impact on the quality or content of Claude’s outputs;

• The difference between watermarked and un-watermarked text will not be distinguishable to readers;

Translation: Specific words do not matter and we don’t think anyone reads anything closely.

• Nothing is added to the text and there are no hidden characters;

This would have been worth clarifying at the outset.

• Watermarking won’t be specific to Claude. As of August 2, the EU requires AI providers serving its market to mark AI-generated content. Other major model developers have signed the same Code of Practice and will be implementing their own watermarks.

No other AI provider has stated that they will apply such marking, adulterating all generated text, outside the EU.

Take the sentence “The weather today was cold and…”. The next word is very unlikely to be “sugary.” But it is quite likely to be “overcast” or “grey.” Under most circumstances, it doesn’t matter much to the reader which of these latter two words the model ultimately chooses — the meaning of the sentence is largely the same either way. In cases like this, the choice is settled by a random number.

Arguing that grey vs. overcast “doesn’t matter much to the reader” is the crux of my argument that this entire endeavor is a perverse adulteration of what it means to write — or to read. That it’s subtle in some ways makes it more perverse, because it’s sneaky.

In internal testing, we’ve seen no impact of watermarking on the content, level of creativity, or readability of Claude’s text. In the SynthID-Text paper, which introduced the technique we use, Google DeepMind tested this impact by serving a model that used watermarking to a portion of their Gemini traffic and comparing thumbs-up and thumbs-down ratings. They found no statistically significant differences from the unwatermarked model. And in a controlled study, human raters comparing watermarked and unwatermarked answers side-by-side saw no difference in quality.

See above for my argument that this thumbs-up/thumbs-down data is absolutely worthless in evaluating whether the SynthID-style word-bias watermarking makes text worse. By definition it must make text worse, unless the underlying LLM model’s scoring is wrong, because the nature of the watermarking algorithm requires it to sometimes increase the probability of selecting a worse word choice and decrease the probability of selecting the model’s best choice. It’s only a question of how much worse. What Google’s thumb-counting data shows is only that it isn’t so much worse as to make Gemini users click the thumbs-down button.

Watermarking doesn’t change the meaning or experience for the person reading it, but if you wanted to check after the fact whether the text was likely generated by Claude, the watermark allows you to do so.

No, it does not. Because the entire scheme is tied to secret keys held only by the AI provider, it only allows Anthropic, not “you”, to check anything.

When Claude proofreads text written by a person, what it gives back has generally only been lightly edited; because nearly all the words are the person’s, there’s very little (if anything) for the watermark to attach to. Depending on the length of the text and how heavily Claude has edited it, those changes might not be enough to make Claude’s involvement detectable. The more Claude writes, the more decisions it has to make, and the more space there is for a watermark.

Translation: No one can ever again use Claude for proofreading their own prose unless they’re willing to risk that the whole thing might be flagged as having been generated by Claude.

For example, once the model has written “2 + 2 =”, there is a very clear best choice for the next token (if the model is completing the sum, there isn’t an answer that’s equally as good as “4”; if it’s talking about George Orwell’s Nineteen Eighty-Four, there isn’t an answer that’s equally as good as “5”). The “nudge” of the watermark wouldn’t be applied here. For the same reason, code — which in very many cases has to be exact — has generally less watermarking than some other forms of text.

Having said that, in areas where there is an arbitrary choice between particular words or terms within the code, the watermark can be used, such as comments within code. But by definition, it will have a negligible effect on the actual code produced.

Translation: We value precision in programming code; we do not in prose.

And it is exceedingly rich to cite George Orwell’s Nineteen Eighty-Four, approvingly, in the context of justifying a text adulteration scheme premised on the notion that specific words do not matter. I mean what the actual fuck? Orwell!

Lastly, as to why they’re doing this:

We’re implementing watermarking to comply with the EU AI Act. Anthropic, along with several other major AI model providers and around 190 total signatories, signed the EU Code of Practice on Transparency of AI-Generated Content in July 2026. This requires AI system providers to use methods of “marking” AI-generated text. We’re applying watermarking globally at launch because we don’t yet have a durable way to scope it by region.

This, from a company that the Financial Times just reported is weeks away from an IPO with an intended valuation of $2 trillion, which would make it one of the 10 highest-valued companies in the world — as of today, placing it at #7, between TSMC ($2.2T) and Broadcom ($1.9T).

This leaves us to believe that one of the following must be true:

• It’s perfectly reasonable that a technology company valued on par with Amazon and TSMC is technically incapable of complying with an EU regional law only within the EU itself.¹ Not a cause for concern at all.

• Anthropic is in over their heads, wields shockingly little control over their own tech stack, and their imminent IPO is likely to be remembered only as a new high-water mark in the manic global AI bubble.

Also, what happens if another major global market makes it unlawful for AI to secretly watermark generated text?

OpenAI

From an OpenAI support document titled “Provenance Signals (Content Credentials, SynthID) in OpenAI-Generated Content”:

Consistent with our commitments under the European Commission’s Code of Practice on Transparency of AI-generated content, our goal is to expand provenance signals to all modalities including text, so customers and developers have clear ways to meet their own transparency obligations as standards and tooling continue to mature.

There’s a lot of wiggle room in this brief statement, and it could just as well mean that OpenAI models will only adulterate text with fingerprint markers when users or developers ask for it. Or that it will only be mandatory for users in the EU. If I were at OpenAI I’d go hard on this and publicly say that ChatGPT will never watermark text it generates unless you ask it to, and that if you want tools that secretly work behind your back without telling you how they work to flag your words in ways you can’t see, go ahead and use Claude.

Further Reading

Three papers on ArXiv:

• “A Watermark for Large Language Models”, last revised May 2024.
• “Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models”, last revised May 2025.
• “Robust Text Watermarking for Large Language Models via Dual Semantic Embeddings”, last revised June 2026.

I will admit that while I’m profoundly offended by the idea of personally using tools that attempt to leave such watermarks in text they produce or touch, the mathematics behind it are fascinating.

Michael Lopp, at Rands in Repose, “RIP Claude”:

As a human who has had to wrangle with EU regulations in the past, I am abundantly clear what’s involved in the laborious bureaucratic process. I can guess what threats Anthropic is facing. However, this is a tone-deaf, clumsy, and alarming opening salvo in their watermark strategy. [...]

My writing is my work, and Anthropic’s current strategy is aggressively writer-hostile.

Jeff Gamet, “Anthropic’s Claude Watermark Is Akin to an AI Poison Pill”:

To be clear, the watermarking is embedded in pretty much any text Claude touches. Along with text Claude generates, it also applies to text it processes, such as proofreading and summarizing. I expect we’ll see too many inaccurate accusations of using Claude to write documents where the content was human-written, but AI-proofread.

The watermarking sticks with documents through copy-and-paste, too. Imagine copying text from a blog post or email only to have what you wrote tagged as potentially AI-generated. In fact, that could very well happen with this post. I personally write all of my content without AI tools, but I copied the quote at the top of this piece directly from Anthropic’s website. Does that mean what I wrote here will show as AI-generated? If they used their own models to generate or edit what I quoted, then the answer is very likely “yes.”

One of the papers published at ArXiv I cited above claims that such watermarking even persists when an article of text originally generated in English is translated into German.

Secrets are the poison here. When only Anthropic holds the secret keys that both produce the watermarking and perform the probabilistic detection of those marks, we’re all left to wonder. To wonder if what we’re reading is secretly watermarked, what we’re quoting is secretly watermarked, and whether what we ourselves are writing will be unjustly accused of being AI-generated based on secrets we don’t know and can’t see. Poisonous is exactly the right word.

Or should I say toxic? Or airplanes?

I should have anticipated this, because I’ve had this conversation with real-world normies repeatedly in recent years, but I got a bunch of questions from people who say they’re wary of ever putting their iPhone down on its back because they’re worried about scratching the camera lenses. That’s not a silly thing to worry about. It looks like those lenses are glass; glass scratches; and it sure seems like a scratched camera lens might forever ruin all the photos and videos you take with the camera.

There are a couple misconceptions here though. First, the glassy flat circles exposed on the back of your phone aren’t the camera lenses. Those are covers over the lenses, which are smaller, spherical (convex, not flat), and recessed. You can look through the covers and see the actual spherical lenses inside. The exposed lens covers are made of sapphire, not glass, and are thus incredibly scratch-resistant. They’re by far the most scratch-resistant parts of the phone. I’ve never once found even a tiny scratch on any of my iPhone camera lens covers, and I’ve never taken any particular care to avoid scratching them. I mean, I don’t drag the lenses face-down on surfaces. I’m not trying to scratch them. But I set my phones down on hard surfaces lenses-down all the time and they never seem to pick up even fine scratches.

Zack “JerryRigEverything” Nelson is the guy on YouTube who makes videos where he scratches the hell out of products to see how durable they are. (And he bends them, burns them, and abuses them in other gruesomely creative ways.) In his iPhone 17 Pro video, starting around the 4:10 mark, he tries scratching the sapphire lens covers with a sharp razor blade. No effect. Sapphire is far more likely to shatter than scratch, because it’s so hard. (You may recall that a decade ago Apple pursued using sapphire for the displays of iPhones but it didn’t work out.) Don’t try scratching your lenses with a diamond hardness pick and you’ll be fine.

Also, believe it or not, a fine scratch on the lens cover will almost certainly not affect image quality at all. Try sticking a strand of hair (which is probably much thicker than a typical scratch) to the surface of your iPhone 1× main camera lens. Take a picture. Clean the hair off the lens. Retake the same picture. You almost certainly won’t see any difference at all. Here’s a great video from DPReview back in 2020 showing just how much dust or scratching you need to put on the outside of a lens to degrade image quality. It’s counterintuitive but the surface of the lens is not where light is focused — the sensor, inside the camera, is.

To the best of my recollection, this is the first post I’ve retracted in the 24 years I’ve been writing Daring Fireball. I hope it’s the last. I was misled, both overtly and through omissions, in several ways, but what I publish is my responsibility, and I apologize for the error.

Terry Godier first came to my attention in February, when I linked to his excellent interactive essay on RSS feed reader design, “Phantom Obligation”, which essay introduced Current, Godier’s new RSS reader that he made to exemplify the ideas from his essay. Current is, deservedly, a bit of a breakout hit. (E.g., David Pierce, at The Verge, put it on a very short list of iOS-exclusive indie apps that keep him from switching to Android.) In March I linked to another interactive essay from Godier, “The Last Quiet Thing”, and again in April to a post regarding the App Store’s 5-star review system. I struck up an iMessage correspondence with Godier around when I first linked to his work.

Yesterday Godier posted “Browsers Have Standards, the App Store Has Judgment”. As originally published, that post contained these two paragraphs:

A while ago I tried to submit an iOS app for Dark Hours, my astronomy website for normal people. It was rejected on the grounds that it was astrology.

It has no tarot function, no horoscopes, and nothing that I, or anyone else I’ve asked, would associate with astrology.

Those paragraphs, at this writing (August 8, 11:00 pm ET), have been deleted and replaced by this:

Note: the original version of this post had a section here about an astronomy app I am working on that began as an astrology app and was rejected after having the astrology content removed. App store [sic] review reached out to me and let me know that they had apparently never been given the updated build and that the app should be fine to submit now.

You can now see the problem that has led me to fully retract my original post, given that my post was entirely predicated on the premise that Godier’s original description of the rejected app was true — that the app “has no tarot function, no horoscopes, and nothing that I, or anyone else I’ve asked, would associate with astrology.” The truth is, the app, as originally submitted by Godier to the App Store (under the name “Asterly”, not “Dark Hours”), was entirely dedicated to astrology, not astronomy, and did in fact include a “Tarot card of the day” feature amongst other occultist horseshit.

The grounds of Apple’s original App Store rejection of the app, and the rejection’s upholding by the App Review Board, were correct.¹ I wrongly took Godier at his word, both in his public blog post and in private iMessage correspondence yesterday, that the rejection wasn’t just merely debatable, but completely and rather preposterously ungrounded. Whether Godier ever submitted a build of “Asterly” to the App Store that contained no occult horseshit and only the hard-science astronomy features that were present in his “Dark Hours” website that was available for the last week, I don’t know. But I have no reason to believe that he did.

My disdain for astrology is so utter, and my esteem for Godier’s previous work so high, that it simply never occurred to me that he might have actually made and submitted to the App Store an astrology app, let alone that he’d then feign surprise and frustration that an astrology app was rejected for being an astrology app. I showed him a draft of my post before publication, to make sure I had the story straight, and he offered not a word of caution, only gratitude for my drawing attention to the matter.

It gets messier. Godier’s astrology app that he submitted to the App Store back in January was named “Asterly”. That was still the name when the App Review Board upheld its rejection in April. According to Godier, frustrated by the App Store’s rejection, he ported the astronomy version to the web, launching it last week under the name “Dark Hours” at the domain darkhours.io. (This is why it was incorrect for me, in the very title of my post, to claim that Apple had rejected an app named “Dark Hours”. They rejected an app named “Asterly” and had never seen an app from Godier named “Dark Hours”.) Yesterday, after I linked to Godier’s post and Hacker News then linked to my post, Godier’s “Dark Hours” (with a space) came to the attention (and justifiable surprise) of Miguel Beher, creator of an open-source “astrophotography and dark-sky planner” project named DarkHours (no space). Beher’s GitHub project contains the source code, and the actual web app is freely available at the domain darkhours.app. In an uncomfortable exchange between Beher and Godier on Bluesky, Beher pointed out that Godier’s Dark Hours had the same bug as Beher’s that routed people to “random fields in Mexico”. Earlier today, Godier took his web app down and redirected his darkhours.io domain to Beher’s darkhours.app.

At the end of my now-retracted post yesterday, assuming I had righteously and rightly skewered Apple for an egregiously erroneous App Store review rejection, I wrote:

Mistakes happen. But in a functioning system mistakes get corrected, and mistakes as obvious as this one get corrected almost instantly and include a quick apology for the conflation.

Obviously it was I who was mistaken. This article is my correction, and I apologize to all who read and believed my now-retracted post, and to the reviewers at the App Store whose competence (if not literacy) I besmirched. I am deeply sorry about that. Won’t happen again.

Apple had claimed that they contacted OpenAI in February and that we didn’t respond. They now admit that their outside lawyers emailed the wrong person after confusing two Asian last names — only after we brought this to their attention.

OpenAI is hanging on to the fact that Apple’s outside counsel, Gabriel Gross, sent one email to the wrong address, and quickly emailed an apology. In OpenAI’s phrasing, it sounds like Apple’s attorney sent the entire initial letter of concern to the wrong person, and that’s why OpenAI never responded — because it wasn’t sent to the correct person (OpenAI general counsel Che Chang). That’s not what happened. The initial blockbuster “hey we think you guys are stealing our trade secrets and we want to talk to you about it” letter was sent to Che Chang. And Che Chang never did respond to Apple’s lawyers. That a mistaken email thanking Che Chang for a phone call that never happened (because that email was intended for another OpenAI employee) was also sent is irrelevant. I don’t understand why OpenAI is continuing to focus on this inconsequential mistake. (Apple’s motion for a preliminary injunction includes the full text of the mistaken email and subsequent apology.)

Apple accuses Chang Liu of accessing Apple confidential information after leaving the company, but only now admits that Apple employees reached out to him and asked for his help to locate this information (you can read the messages here). Apple now tries to shift the blame to “residual access”, but they also don’t disclose that this is a common issue with Apple which is caused by them failing to properly manage system access when people leave. What that means in practice is that former employees who are trying to do the right thing when they leave still have access to Apple files — despite not wanting them or even being aware of them.

OpenAI is seemingly alluding to Apple’s unusual use of iCloud Drive, tied to employees’ personal Apple Account IDs, that I (coincidentally?) wrote about yesterday. Apple’s motion for injunction, however, addresses this very point. From page 3 of the motion:

Mr. Liu resigned on Thursday, January 22, 2026, and provided notice that he would start at OpenAI the following Tuesday. On his last day, he failed to respond to Apple’s attempt to schedule his exit interview or sign his confidentiality reminder.

In the days following his departure, Mr. Liu seemed initially cooperative and aware of his obligations to Apple. He worked with others on his former Apple team to return certain Apple information remaining on his personal iCloud account to Apple.¹ He also continued to converse with former co-workers, for example, to answer questions about his earlier work and where certain information was stored. But these interactions and exchanges cannot explain the repeated, unauthorized downloading of voluminous technical files from Apple’s cloud-based storage discussed below, which Mr. Liu performed on multiple occasions from February to April 2026 while employed by OpenAI.

That footnote reads:

¹ While Apple seeks discovery into what Apple confidential information Mr. Liu accessed from his personal storage accounts (including iCloud) and devices after his departure, the specific unauthorized downloads referenced in the complaint and at the heart of this motion are not based on iCloud activity, but instead relate to Apple’s third-party cloud storage.

Nowhere in any of Apple’s filings (here’s the Court Listener index page for all the documents filed in the case) does it say who the third-party cloud storage provider is, but I’m almost certain it’s Box, which I know is widely used throughout Apple.

The iMessage transcripts that OpenAI provides at the bottom of their post do not contradict Apple’s claims at all. Apple’s motion states that Liu helped former colleagues find certain documents that were in iCloud; that’s what OpenAI’s transcript shows. But that’s not in dispute. Apple also claims that Liu accessed confidential information, presumably in Box and definitely not in iCloud Drive, on five different occasions, up until 27 April 2026, over three months after he left Apple. These chat transcripts offer no explanation for that. The chat transcripts explain iCloud Drive access that Apple itself says is not in dispute, and do not explain the 37 documents Liu downloaded from the third-party cloud provider (Box?) that Apple says are at the heart of naming him in the lawsuit. Here is Apple’s declaration from digital forensic specialist Daniel Roffman, documenting Liu’s access to confidential files post-employment (albeit with significant redactions).

I do not understand why OpenAI is treating this as a PR problem instead of as a legal problem. Dan Moren, linking to it from Six Colors, is of similar mind, writing:

What kept running through my head while reading this was the old legal chestnut: “If you have the facts on your side, pound the facts. If you have the law on your side, pound the law. If you have neither on your side, pound the table.”

Thus far this feels like table-pounding from OpenAI to me. Their blog post does, however, move the ball from “we have no interest” in Apple’s trade secrets to “we don’t have them”, (emphasis added):

Apple also accuses Tang Tan of trying to get and use their trade secrets. However, Tang has always been clear with the team that we do not want, and must not use, any confidential information from other companies. Tang served Apple for more than 24 years and was widely known as one of the most innovative leaders at the company. [...]

Apple’s request for a preliminary injunction is both based on false information and completely unnecessary because we do not have, nor want, any of their trade secrets. We’re much more interested in building innovative products and technologies that push the frontier.

To me, the most interesting response from OpenAI wasn’t their blog post. It was an email released by Apple, as “Exhibit F” to one of their expert declarations submitted to the court last night. OpenAI has retained the renowned law firm Quinn Emanuel as outside counsel, and this exhibit is a long email from Quinn Emanuel attorney Patrick Curran to Apple’s attorneys. From that email, dated Monday July 20, Curran writes:

You also ask that we “revisit” the specific points proposed in your July 15 letter. It appears that you want to move backwards. As noted, we already discussed these during our meet and confer but Apple was unable to respond to basic questions my colleagues raised about these requests. For example, your letter proposes that OpenAI “[p]roduce witnesses to testify at deposition” but Apple was unable to identify who those witnesses would be. Similarly, Apple was unsure when we asked if it was actually proposing that hundreds of OpenAI employees fill out “questionnaires” even if Apple has no basis to allege (and is indeed not alleging) that such employees have any connection to this litigation. The seven sections in your letter are broadly worded and remain vague and general. This is not what a forensic protocol looks like and we’re sure you understand that you will not get this as relief from the court. You first need to (preliminarily) identify the TS you are suing for, and your email states that you “appreciate the need” to do so. Any protocol will be informed by such identification. A forensic protocol cannot be based on general terms like “Apple confidential information”; you need to tell us what you’re looking for, and it sounds like you understand that and are prepared to do so. The efficient way forward is therefore to tackle these issues as part of the negotiation of a proper, detailed forensic protocol. If you instead prefer to move for a PI because OpenAI did not agree off the bat to subject hundreds of employees to “questionnaires” about “Apple confidential information” generally, that is unfortunate — and inconsistent with what I understand both our clients have requested. If you choose this path instead of working with us, we look forward to filing an opposition that sets the record straight.

Apple, obviously, did choose this path (“PI” = preliminary injunction), and I too look forward to OpenAI’s setting the record straight, especially if they do so in plainspoken language like Curran’s in this email. Curran continues:

Finally, although I know OpenAI would like to resolve this amicably, as their counsel I have to tell you what I think you already know — this case lacks merit. You have not articulated any basis to support a preliminary injunction. Your complaint is predicated on a misrepresentation of facts and allegations that are speculative at best. It fails to even remotely identify any trade secrets. You are attacking ordinary business practices (used widely across the industry). You are complaining about situations that you have caused, including through your own procedures and decisions. We stand ready to oppose any preliminary injunction motion and tell the world what really happened here to set the record straight. We made clear we would prefer to quickly and collaboratively address any legitimate concerns that your client has, but that is not well-served by repeated threats.

This email is a far better response than what OpenAI published on their blog.

They’re not rivals, they’re layers: most MCP servers just call REST internally to do the real work. The best ones don’t convert every endpoint into a tool, they design around what the agent’s trying to accomplish.

Shipping that server also means shipping OAuth 2.1 with scoped tokens. WorkOS AuthKit already speaks that spec, so you skip building an auth provider on top.

Read the breakdown →

When you lease an iPhone through Apple Upgrade, you need a cellular account on one of the big three U.S. carriers: AT&T, T-Mobile, or Verizon. That kind of stinks, and I’m not quite sure I understand why. You’re leasing the iPhone through Apple and Klarna, not the carrier, so I don’t know why Apple cares. If you know why, shoot me a message and explain it. Is it just a simplistic credit-risk evaluation, where prepaid plan-holders and MVNO users in general are viewed suspiciously?

I got a slew of messages about this. Credit-risk assessment is seemingly part of it. There’s a lot of fraud in iPhone purchases specifically and cell phone service generally. The big three carriers do a significant amount of risk assessment before letting you open a new postpaid account; that you have an active account with one of them is a signal that you’re not trying to lease an iPhone using phony credentials in order to sell it.

But the bigger reason is the complex nature of the contracts between Apple and each of the big three carriers. A reader who worked on the carrier team at Apple for a long time (but left a few years ago) wrote:

In response to your question on why Apple would only offer iPhone on the “Big 3” for the new Upgrade Program: Every year, the Carrier Teams at Apple negotiate a deal/program (hundreds of millions of dollars) with each of those Big 3. In return for those dollars, Apple will make concessions exactly like this (or similar.) This is also why, for example, their logos appear more prominently or are the only ones called out on signage, commercials, etc… Although they really aren’t threatened by MVNO’s (the ones they don’t own) or regional carriers, it’s more like Apple is selling it as part of a package deal that’s exclusively offered to its biggest and best “tier 1” partners. It also acts as a carrot to perpetually try to motivate smaller partners into behavior Apple wants.

This might change — some of the bigger MVNOs are trying to work their way into “tier 1” (or maybe create a new “tier 1.5”). But for many years now, part of the co-marketing agreements between Apple and the Big Three is Apple agreeing to require a postpaid account with a Big Three carrier for all “special” iPhone financing deals, including the old iPhone Upgrade Program and the new Apple Upgrade leasing.

One interesting exception is that Boost Mobile is included alongside the Big Three for Apple Card Monthly Installments (ACMI), a program that offers 0% APR on certain products if you pay for them using an Apple Card. From the ACMI small print:

In order to buy an iPhone with ACMI, you must select one of the following carriers (prepaid carrier plans are not supported): AT&T, Boost Mobile, T-Mobile, or Verizon. An iPhone purchased with ACMI is always unlocked, so you can switch carriers at any time, subject to your carrier’s terms.

So perhaps some of the bigger MVNO carriers might work their way into Apple Upgrade — but if they do, it sounds like they’ll need to pay Apple for the privilege.

With regard to fraud, a former Apple Store retail employee wrote:

I left in 2019, but I’d estimate on some days a double digit percentage of in-person phone sales were fraudulent.

This behavior absolutely exploded when you started to be able to do carrier financing in the store. They had stolen the identity of a legit account holder, come into the store with the info needed to access the account. They always wanted the highest end phone, and would pay the sales tax due at time of sale in cash. Never wanted any accessories. Never wanted to open the box or set up the phone.

As bad as it was with the activation requirement, it would be way worse without it. I believe it’s gotten a lot better, and the carriers have more advanced systems in place to detect and deter this behavior now. But ultimately, the carriers ate the cost for millions of dollars in devices they never got paid for.

I suspect that reader is correct that Apple, in collaboration with the carriers, has cracked down on this.