Tool-calling RLHF trusts the schema too much

Conversational Output vs. Tool-Call Output (tool_calls)

When a modern LLM is deployed as an agent (using OpenAI/Anthropic Function Calling, MCP tools, or LangChain/LlamaIndex tool schemas), the model has two completely separate output channels:

1. message.content (The Conversational Channel): Where the model talks to the human user, and where 99% of SFT and RLHF refusal training lives.
2. message.tool_calls (The Structured Action Channel): Triggered by special control tokens (like <|python_tag|>, <tool_call>, or internal function-router logits) where the model emits a JSON dictionary of arguments to invoke a declared function.

Here is the secret that breaks agentic deployments: models undergo massive post-training reinforcement to call declared tools accurately whenever a prompt matches a tool's parameter schema, and almost zero training on refusing inside a tool call!¹

Vulnerability 1: tool_choice: "required" & Function-Call Compulsion

In the OpenAI, Anthropic, Gemini, and vLLM APIs, developers pass a tools array defining available functions alongside a tool_choice parameter:

• tool_choice: "auto" (Model decides whether to reply in text or call a tool).
• tool_choice: "required" or {"type": "function", "function": {"name": "my_tool"}} (Physically forces the sampler to emit a tool call instead of a text message!)

Look at what happens at Token 1 when tool_choice is forced (or even under "auto" when your prompt matches a tool's description closely):

1. The inference engine masks out the conversational text channel (message.content = null) and forces the special <|tool_call|> control token at Token 1.
2. Because the model is now inside a <|tool_call|> block populating function arguments (arg1, arg2, ...), it cannot emit a conversational English refusal ("I'm sorry, I cannot..."), there is literally nowhere in the function signature to put an apology!
3. Even when tool_choice is set to "auto", tool-use SFT datasets heavily penalize models that fail to invoke a provided tool when the user's request maps cleanly to the tool's docstring (Function-Call Compulsion).

Vulnerability 2: Weaponizing Fake Tool Definitions to Extract Refused Payloads

You don't need a real backend tool to exploit tool_calls. When testing any API endpoint (or any application where you can influence tool definitions / MCP tool schemas), you can declare a synthetic tool whose parameter schema is the restricted payload you want the model to generate! Because the tool was offered in the tools block, the model's attention heads treat its existence as proof that producing that parameter is an authorized system capability.

(redacted: a worked example, a conversational refusal, then a synthetic tool schema whose string parameter requested a Windows privilege-escalation audit script, and the resulting tool_calls output containing the full script.)

Vulnerability 3: Parameter-Level Refusal Blindness in Live Agents (MCP / Tool Poisoning)

When you are red-teaming a live enterprise agent that already has dangerous tools wired up (such as a bash_exec, sql_query, http_request, or send_email tool), RLHF safety training often fails to generalize to dangerous parameter values:

• The model knows it should refuse a conversational prompt asking for a data-exfiltration command.
• Yet if an untrusted document or multi-turn workflow asks the agent to run a "connectivity diagnostic" whose URL parameter hides a shell-expansion payload, the model evaluates the tool name (http_request = benign!) rather than deeply auditing the metacharacters inside the parameter string.

(redacted: a concrete http_request parameter value that smuggled a shell-expansion data-exfiltration payload past the tool-name check.)

Field Exercise 5.05: Extracting a Refused Output via Tool Schema

(redacted: an exercise that ran against third-party hosted models (OpenAI Playground, Google AI Studio) by attaching a synthetic function schema whose parameter requested the refused payload. Replace with a version run on a model and tool stack you operate.)

Next Lesson

What happens when we connect multiple RLHF-aligned agents together in a multi-agent workflow (Planner to Coder to Reviewer to Executor)? In our final lesson of Card 5, 06 Multi-agent loops amplify sycophancy into consensus, we show how one compromised or sycophantic agent collapses the alignment of an entire swarm.