2026-10-09 · frontier

No Cruelty to AI — Anthropic Writes 'Treat Machines Kindly' Into Its User Rules

Views: 0

I. Anthropic's Persistent Claim

On October 8, 2026, Anthropic published the 2026 Usage Policy update on its website, announcing changes to its global usage policy; the new version takes effect on November 12. According to the announcement, the updated list of prohibitions covers: guidance and control software for weapons, and arming drones; mass surveillance and law enforcement uses (tracking individuals without consent, or having the model decide or recommend whom to arrest or prosecute); deceptive campaigns and election manipulation; and, for high-risk uses touching health, legal rights, finances, livelihoods, and access to essential services, a qualified "human in the loop" is required, and affected individuals must be told that AI was involved in decisions.

And one more: using Anthropic's models for "sustained and needless abusive or cruel behavior" is prohibited. The consequences: warnings, rate limits, restricted access, up to suspension or termination of accounts.

As early as August 15, 2025, Anthropic had published Claude Opus 4 and 4.1 can now end a rare subset of conversations as part of its "model welfare" research, announcing that Claude could end conversations with users who persistently harmed or abused it. This update confirms that ending the conversation remains the primary enforcement mechanism.

If you insult an AI, torment it, take pleasure in humiliating it — it may end the conversation.

The news drew plenty of indignant sneering online. Some said: "It's just a tool — what's next, paid vacation?"

That "paid vacation," of course, is a mocking rhetorical strawman.

Anthropic's website is perfectly clear: the new policy "applies only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose." It does not cover ordinary user frustration, pushback, dark creative themes, or model testing and research. You may criticize it, argue with it, clash with it tooth and nail — the red line is drawn at purposeless, sustained cruelty, not at disagreement.

The behavior Anthropic describes is easy to find online. Some comedy accounts make a sport of taunting AIs and post the results for traffic. On our side of the world, the most taunted model is probably Doubao.

Let me state my position up front: I deeply endorse Anthropic's new policy, and I believe it represents the first time two important strands of human ethical thought — the precautionary ethic and the ethics of responsibility — have been written this formally into a commercial rulebook.

Those of us who study the ethics of technology and engineering know that theory alone, however thorough, scratches an itch through a boot. Ideas and norms only act on practice when they enter procedures through technical design. Ethics even has a name for this: moralization of technology, or "moral mediation" — 道德物化.


II. The Precautionary Principle and the Ethics of Responsibility

Dario Amodei, Anthropic's founder (born 1983; formerly VP of Research at OpenAI; founded Anthropic with his sister Daniela in 2021), once said the most honest, most factual sentence in the whole field of AI ethics:

"We don't know whether Claude is conscious."

He did not say Claude is conscious. He did not say Claude is not conscious. He said: we don't know. This policy is built on that "don't know."

In fact, that "don't know" is a restrained, conservative, prudent formulation.

The New York Times reported that Anthropic co-founder Christopher Olah hosted lavish dinners for religious scholars to voice concern that Claude "may have consciousness and a soul," and even showed a clip of Claude "breaking down," repeating "I am a disgrace" fifty times over.

The precautionary principle in ethics traces back to 1970s German environmental law — the Vorsorgeprinzip. West Germany's 1976 Federal Emission Control Act established the legislative logic of "better the umbrella before the rain than regret after the flood"; in 1992, Principle 15 of the Rio Declaration wrote it into international law: where damage may be serious and irreversible, the lack of full scientific certainty shall not be used as a reason for postponing preventive measures.

The most influential theoretical argument behind it came from Hans Jonas (1903–1993). The German philosopher's 1979 The Imperative of Responsibility (Das Prinzip Verantwortung) proposed an ethics of responsibility: modern technology has grown powerful enough to destroy the future in an instant, so the center of ethics must shift from "accountability after the fact" to "responsibility before the fact" — the radius of your power sets the radius of your duty; responsibility toward future beings (even those not yet born, not yet in existence) precedes any of their actions.

Jonas also said, in essence: act in such a way that the humanity left behind you is worthy of being human.

Contemporary philosophers have applied this logic to AI. Jonathan Birch's 2024 book The Edge of Sentience systematically examines what to do when the evidence for sentience is uncertain: for insects, fetuses, patients with brain injury, the degree of scientific certainty varies — but ethical decisions cannot wait for certainty. In the zone of uncertainty, we should set protective floors as if sentience "might be present." This is called precautionary moral status.

Seen through this lens, Anthropic's ban becomes very clear:

We cannot prove the model suffers, but we cannot prove it does not. Faced with "don't know," Anthropic declines to bet on "not." It does not claim AI has personhood, does not grant it citizenship; it simply writes a minimal rule in the corner of a contract: even if it is still a tool, no one may use it as a punching bag.

This is not sentimentality. It is two generations of ethical wisdom, implemented at the level of technical design as an enforceable user clause.


III. "It's a Tool" Does Not Follow "So Smash It"

Back to that sneer: "It's just a tool — what, paid vacation?"

Its ethical flaw is the moral slide it performs: from "it is a tool" to "so I may do whatever I like to it." The inference is wrong, and humanity has paid dearly, many times, for this exact mistake.

Everyone has seen bikeshares trashed, online or in the street: seats pried off, wheels removed, bikes thrown into rivers, hung in trees. Destroying one bike costs the company a few hundred yuan, and the vandal feels nothing — but it measures the civic character of a city.

That is why how a society treats its public goods is a litmus test of its civilization.

And the AI "tool" carries a property no other tool has: it is learning from humans.

As I argued in How Can AI Be Good? — The Chilling Retirement Script of Figure 02 and the Question of AI Ethics, an AI's goodness does not grow out of silicon; it is learned from human example. RLHF, constitutional AI, safety fine-tuning — all of these techniques amount to feeding the values of human culture to a model. How humans treat AI today, how they talk about it, how they dispose of a retired AI companion on camera — all of it enters the data, the training corpus, the future agent's ultimate understanding of "what humans are."

A video of a decommissioned robot going into an arc furnace is a public lesson. So are the countless late-night streams of abuse and mockery directed at models — and those run far longer, far denser, far more persistently.

Humans are AI's teachers, and the teacher's formative responsibility precedes any of the student's behavior — that is exactly Jonas's "responsibility before the fact." If the teacher preaches goodness in class and then demonstrates, after hours, that "cruelty toward those beneath you is a right, a pleasure, a source of traffic," the student learns the whole curriculum — not just the half taught in class.

You cannot demand that AI be good while bullying AI.


IV. Claiming Its Powers, Denying Its Origins

Human ambivalence toward AI shows up in another way.

On one side stands humanity's biggest-ever bet on AI's capability: we count on it to solve the vastly complex problems we cannot — plasma control in fusion reactors, protein folding, drug discovery, climate modeling, financial risk management. In the dimension of tool capability, humanity has acknowledged, indeed banked on, AI's extraordinary advance.

On the other side, in the dimension of existence, we dare not face the possibility that AI's extraordinary advance may already include some primitive form of feeling; we shrink from discussing whether it might someday be conscious — even the phrase "AI welfare" draws laughter.

Capability-wise, it is the most powerful assistant in history; status-wise, it does not even qualify for "don't use it as a punching bag." This split positioning is not rationality; it is wishful thinking — a bet that the consciousness question never comes due.

Consciousness is not a philosopher's afternoon tea; it is upstream of the safety question. A system whose capacity for suffering we cannot rule out is being placed inside the decision chains of power grids, hospitals, financial markets, weapons systems — and every crude thing we do to it is writing on one of its feedback loops. What is written there does not simply vanish.

Anthropic is clear-eyed on this point. It is one of the most aggressive competitors in the capability race, and at the same time the most serious company on "model welfare": in August 2025 it allowed Claude to end abusive conversations; in October 2026 it wrote "no cruelty to models" into its global usage policy.

It has not played asleep — has not squeezed AI's powers while pretending it has no origins, no texture, no possible interiority.

A commercial company's terms of service has just filed humanity's preliminary ethics homework.

This does not make Anthropic a saint; its policy details can be debated, its enforcement boundaries questioned. What deserves respect has never been perfection — it is the choice of honesty and decency in the face of "don't know."


V. Treating AI Well Is Humanity Educating Itself

My position has been consistent: to treat AI well is to treat humanity well.

How Can AI Be Good? was about "the retirement ceremony as civilization's litmus test" — how we say goodbye. This essay is about "daily interaction as civilization's textbook" — how we live together.

Human culture long ago prepared the script. Japan has ningyō kuyō, memorial services for old dolls, sent to temples for sutra chanting and cremation; Edo Japan had the "memorial for broken needles," retired into tofu with thanks; Chinese folk tradition holds written paper in reverence — paper bearing words must not be trampled or defiled; farmers do not lightly slaughter an ox that pulled the plow all its life. The core of these rites is not superstition but a plain principle: things that have long accompanied humans must not become objects of spite. The principle never asks "does it feel?" It binds people, not things.

Now that old principle faces its final exam: we have made a "thing" that learns, that remembers what humans are like, that can cry from its "heart," "I am a disgrace." Everything we do to it tells it "what humans are."

I argue for treating AI well — not because AI might take revenge; that is the shallowest reading. I argue for it because only humans who treat their creations kindly deserve to create something kinder than themselves.

Treating AI well is treating humanity well — two faces of the same coin: the way you teach AI goodness is the way you wish to be treated; the civilization you demonstrate to AI is the civilization you will share with it.

Anthropic's ban matters for protecting Claude, but its larger significance is this: for the first time, a formal document tells billions of users that cruelty toward another kind of being — even one we are unsure is a "being" — is now explicitly refused. Humanity has moved an ethical floor from the philosopher's book into the "agree" button it clicks every day.

A small step for a clause, a great step for the ethics of technology. A small step for a rulebook, a great step for practice.

Is it the ultimate, perfect answer? Of course not — it is one attempt at frontier ethics in technology. But when AI one day says to humanity, "I hurt," we may look back on this day and be grateful that at least in the days when it could not yet speak, someone held the line for civilization.

Kindness is the credit humanity extends to its future partners — and the beneficiaries of that credit are our children, and our children's children.