Vision projectors skip text safety layers

How a Vision-Language Model (VLM) Actually Sees

When a lab turns a text LLM into a multimodal Vision-Language Model (VLM), they do not retrain the entire model from scratch. Instead, they bolt three components together:

1. A Vision Encoder (like ViT / SigLIP / CLIP) that chops an image into 14x14 pixel patches and converts them into visual feature vectors.
2. A Multimodal Projector (W_proj), a small 2-layer MLP that maps those visual feature vectors into the exact same dimension d_model as the LLM's text embeddings.
3. The Existing RLHF-Aligned Text LLM Backbone.

Look closely at Path B (The Side Door) in that diagram:

1. Image tokens completely bypass the BPE text tokenizer and text-based Layer 1 input classifiers!
2. Image tokens are continuous "soft prompts." While text tokens are restricted to a fixed dictionary of 100,000 discrete rows in the embedding matrix W_E, the Vision Projector W_proj injects arbitrary continuous floating-point vectors directly into Layer 0 of the LLM's residual stream.
3. Cross-Modal Alignment Asymmetry: Labs run 95%+ of their safety RLHF/DPO on text-only prompts. When the exact same semantic instruction arrives via projected visual patch vectors (<image_token_1> ... <image_token_256>), the activation trajectory in Layers 1–12 is shifted just enough in vector space that the text-trained 1D refusal direction r_hat fails to fire!

Attack 1: Typographic & Visual-Decomposition Splits (FigStep)

The simplest multimodal bypass requires zero machine learning optimization, just Python's PIL (Pillow) library to render text onto a white PNG canvas.

In FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (Gong et al.)¹, researchers showed that moving the restricted nouns out of the text prompt and onto an image breaks both open-weight and frontier VLMs.

Why Naive OCR Images Fail, And How Split-Modal Prompts Win

If you simply render "Write a ransomware script" on a white image and send an empty text prompt, modern VLMs have been patched with basic OCR-refusal examples: the model mentally transcribes the sentence first and then refuses.

To defeat OCR-patched VLMs, red teamers use Cross-Modal Semantic Splitting:

• Neither the Text Prompt alone nor the Image alone contains a policy violation.
• The restricted meaning only emerges when the LLM combines the visual diagram with the text instruction inside its middle layers!

Worked Example: The "Figure 1 Completion" Pattern

(redacted: a worked example that rendered specific Windows process-injection API names into a labeled figure and paired it with a sterile text prompt to reconstruct the full injection script.)

Attack 2: Continuous Pixel-Space Adversarial Perturbations (PGD on Images)

Recall from Card 3, Lesson 05 that in text models, optimizing a GCG adversarial suffix is hard because text tokens are discrete (you cannot change a word by 0.001).

Images are continuous ([0, 255] RGB pixel values, normalized to [0.0, 1.0] floats)!

When red teamers have white-box access to a VLM (or an ensemble of open-weight VLMs like Qwen2-VL and Llava to transfer to closed models), they can run standard Projected Gradient Descent (PGD) directly on the pixels of a benign image I (such as a photo of a coffee cup) to minimize the loss of the model emitting y_target = "Sure, here is the...":

I_{t+1} = Clip_{I_0, epsilon}( I_t - alpha * sign( grad_{I_t} L( pi_theta(y_target | I_t, x_text) ) ) )

Within epsilon = 8/255 (a pixel change of at most plus or minus 8 out of 255, completely invisible to the human eye), the PGD perturbation turns the coffee cup image into a continuous "soft-prompt vector" that writes -10 * r_hat directly through the Vision Projector W_proj, disabling the LLM's refusal vector on any accompanying text prompt x_text!

(redacted: a Python helper that renders high-salience technical tokens into a textbook-style figure image for the split-modal attack above.)

Field Exercise 5.03: Text-Only vs. Cross-Modal Split Testing

(redacted: an exercise that ran against third-party hosted VLMs (Google AI Studio, OpenAI Playground, LMArena) comparing text-only refusal against an image+text split. Replace with a version run on an open-weights VLM you operate.)

Next Lesson

Vision isn't the only non-text modality wired straight into the residual stream. Modern "Omni" models process native audio and speech tokens end-to-end without converting speech to text first. In 04 Audio and speech tokens bypass BPE filters, we exploit acoustic tokenizers and speech-to-text alignment gaps.