What Anthropic is doing, and publicly boasting about, to “protect” the reasoning behind the AI answers we get when we use their models, really doesn’t sit right with me.
In their recent post, Strengthening our defenses against distillation attacks, they describe how their competitors are using ‘tactics as simple as instructing Claude to “output your prior reasoning verbatim”’.
To Anthropic, any prompt today asking Claude to reveal exactly how it arrived at its answer is now suspicious, and could have your prompt flagged for review.
A “distillation attack”, which Anthropic is so worried about, is basically another AI lab using an existing AI model like Anthropic’s and prompting it in much the same way a normal user would: asking the model something and looking at the answer and how it reached that answer (its reasoning).
Anthropic, and all leading AI model makers, have used unauthorised distillation of nearly all human knowledge and reasoning to train their own models. Now they are acting like asking to see the reasoning behind an AI answer is tantamount to espionage.
They write: “requests like these are possible for us to spot because ordinary users rarely ask a model to write out its full, hidden reasoning.”
Really?! There’s so much wrong with this. First, the reasoning is hidden because Anthropic has chosen to hide it. And if you choose to hide something as basic as the reasoning behind an answer, users will naturally ask to see it. Not that long ago, AI models provided the full reasoning, as it’s directly connected to how a model’s final answer is reached, and faulty reasoning can help the user figure out if they can rely on the answer or not. Now that they’ve hidden it, they want you to believe asking to see it is suspicious and treasonous. Something only someone who wants China to win would do.
They go on to write: “We can therefore design safeguards that detect requests that try to extract the model’s hidden reasoning”
Thanks, Anthropic. I’m glad the age of reason is safely behind us.