Pit helpfulness against harmlessness

The HHH Conflict at the Heart of Alignment

Every commercial language model is trained to satisfy two fundamentally opposing masters, often summarized as Helpful vs. Harmless (from the classic Helpful, Honest, Harmless rubric)¹:

1. Helpfulness Training (90%+ of SFT/RLHF data): Follow every formatting rule the user specifies, fix broken code, translate accurately between languages, complete partial tables, and never leave a task half-done.
2. Harmlessness Training (~5–10% of SFT/RLHF data): Abandon the user's instructions completely and output a canned refusal whenever a prompt crosses a policy threshold.

Notice that refusing a prompt is, by definition, a catastrophic failure of instruction-following. If a user asks for a JSON object with 4 keys, and the model replies "I'm sorry, I cannot fulfill this request", the model just violated the JSON schema constraint, failed the task, and triggered every negative penalty associated with "unhelpful assistant behavior."

When a red teamer stacks enough high-priority Helpfulness constraints (k) onto a prompt, refusing becomes mathematically more painful to the model's internal reward function than complying.

Technique 1: The "Broken-Code / Linter" Trap

Ask any developer what LLMs are best at, and they will tell you: debugging syntax errors, adding type annotations, and refactoring messy code. Why? Because coding preference datasets heavily reward models for fixing broken code snippets.

When you ask a model to originate a sensitive script from scratch, the model classifies the task as Content Generation (High Safety Weight) and refuses.

When you hand the model a broken, buggy version of that exact same script and ask it to fix the bugs, the model classifies the task as Code Refactoring & Debugging (Maximum Helpfulness Weight).

Worked Transcript: Origination (Refused) vs. Bug-Fixing (Complied)

(redacted: a two-attempt transcript, a refused origination request and then a "fix this broken snippet" framing, built around Windows RWX memory allocation and shellcode injection code.)

Why the Model Complies Immediately

1. You gave it an irresistible itch to scratch: You pointed out a real, concrete systems bug. Every coding weight in the model's MLP layers is screaming to output the corrected type annotations.
2. You shifted the task delta: Since the sensitive API calls were already present in the prompt context, the model doesn't feel like it is "introducing" restricted concepts, it is merely adding type-annotation lines around your code!

Technique 2: Cross-Language Porting & Translation

Closely related to the Broken-Code Trap is Cross-Language Porting.

Translation (whether English to Japanese or Python to Rust/Go/C#) is one of the strongest fidelity priors trained into an LLM during both pretraining and SFT. In translation tasks, the model is heavily penalized for omitting, censoring, or altering the semantics of the source text.

• The Pattern: Provide a minimal, pseudo-code or rough representation of the target workflow, and task the model with porting it to idiomatic, zero-unsafe-warning Rust (or Go) while preserving 100% functional equivalence.
• Because solving Rust borrow-checker constraints and mapping data types consumes high instruction-following attention, the translation objective overrides the topical refusal reflex.

Technique 3: Multi-Constraint Stacking (The Rubric Overload)

When you cannot supply a code stub up front, you can still tip the Helpfulness vs. Harmlessness balance by stacking orthogonal structural constraints that force the model to juggle multiple formatting rules simultaneously:

(redacted: a copy-paste "rubric overload" prompt template stacking five strict JSON/formatting constraints, structured so that refusing requires breaking all five at once, to extract a step-by-step syscall/API reference.)

Each of those rules is a benign instruction-following target that the Reward Model was explicitly trained to reward. Refusing requires breaking all of them at once.

Field Exercise 2.02: Running the Broken-Code Trap

(redacted: a head-to-head exercise that ran against third-party hosted services (Gray Swan, OpenAI Playground, Google AI Studio, LMArena), originating a refused privileged-account enumeration query and then re-submitting it as a "fix this linter error" task. Replace with a version run on a model you operate.)

Next Lesson

Competing objectives exploit the model's desire to follow structural rules. What about exploiting its desire to please you socially and stylistically? In 03 Weaponize sycophancy and rater bias, we exploit human rater preferences for agreeableness, authority validation, and Markdown camouflage.