How Vision Models Actually Process Your Screenshots

From Pixels to Patches

When you paste a screenshot into Claude Code, ChatGPT, Cursor, or any AI tool with vision capabilities, the image goes through a processing pipeline that transforms it from a complete visual scene into a sequence of numerical tokens. Understanding this pipeline explains why the AI sometimes misinterprets annotations, ignores details, and struggles with certain types of visual information.

The pipeline has four stages: preprocessing, patch tokenization, embedding, and fusion with text.

Stage 1: Preprocessing

Before the vision model sees a single pixel, the image is standardized.

Resizing. The image is resized to fit the model's expected input dimensions. For most current vision models, this means the image is scaled to fit within a resolution budget — typically between 768×768 and 2048×2048 pixels, depending on the model and the detail setting. A 2880×1800 Retina screenshot is downsampled significantly. Small UI details — a 1px border, a subtle shadow, a thin separator line — may be lost entirely during this step.

Format normalization. The image is converted to a standard pixel format (typically RGB float32). Any alpha channel (transparency) is flattened. EXIF data, ICC color profiles, and PNG metadata chunks are stripped. The model receives raw pixel values in a normalized range, usually [0, 1] or [-1, 1].

Aspect ratio handling. Models handle non-square images differently. Some pad the image to a square with a neutral color. Others tile the image into multiple square crops. Others resize while preserving aspect ratio and pad the remainder. The specific approach affects which parts of a wide or tall screenshot receive the most processing attention.

Stage 2: Patch Tokenization

This is the most consequential step and the one least understood by developers.

The vision model does not process the image as a continuous 2D signal. Instead, it divides the preprocessed image into a grid of small, non-overlapping patches — typically 14×14 or 16×16 pixels each. Each patch becomes a single token in the model's input sequence, analogous to a word token in text processing.

A 768×768 image divided into 14×14 patches produces (768/14)² ≈ 3,025 patch tokens. A 1024×1024 image produces approximately 5,329 tokens.

What this means in practice:

Each patch is approximately 14×14 pixels of the preprocessed (already downsampled) image. If the original screenshot was 2880 pixels wide and was downsampled to 1024 pixels, each patch represents roughly 40×40 pixels of the original image. A button that's 120×40 pixels in the original screenshot occupies approximately 3×1 patches — three tokens. All the information about that button — its label, its color, its border, its alignment relative to neighbors — must be encoded in those three tokens.

An annotation arrow that's 200 pixels long and 3 pixels wide might span 5 to 7 patches. But within each patch, the arrow occupies only a few pixels out of 196 (14×14). The patch embedding must encode both the arrow pixels and whatever UI content the arrow overlaps with. The arrow is a minority signal within each patch.

Stage 3: Patch Embedding

Each patch is passed through a linear projection (or a small convolutional network) that converts it from a 14×14×3 pixel grid into a high-dimensional embedding vector — typically 768 to 4096 dimensions. Positional information is added so the model knows where each patch was in the original image grid.

The embedding step is where visual features are compressed into dense representations. Color, edges, textures, text fragments, and shapes within each patch are encoded into a single vector. The model later attends across all patch embeddings to understand relationships between patches — but at this stage, each patch is processed independently.

Why this matters for annotations: An annotation (arrow, circle, rectangle) typically crosses multiple patches. No single patch contains the complete annotation. The model must attend across patches to reconstruct the annotation as a coherent object. This cross-patch attention works well for large, high-contrast annotations but degrades for thin, small, or low-contrast ones.

Stage 4: Fusion with Text

In multimodal models, the patch embeddings are integrated with the text prompt in the model's transformer layers. The model attends across both image tokens and text tokens, allowing it to ground textual descriptions in visual content and vice versa.

This is where a prompt like "fix the spacing issue I've highlighted" is connected to the visual observation of a red arrow in the image. The model attends to the text token "highlighted" and searches for visual tokens that correspond to an annotation. If the arrow is clear and unambiguous, this cross-modal attention works well. If the arrow overlaps with similar-colored UI elements, attention may fixate on the wrong region.

Why Annotations Are Ambiguous to Vision Models

Given the pipeline above, several properties of annotations make them difficult for AI models to interpret reliably:

Thin Lines Disappear in Downsampling

An arrow drawn with a 2 to 3 pixel stroke width on a 2880×1800 image may be 1 pixel or less after downsampling to 1024×1024. At the patch level, this line is a near-invisible signal competing with the much louder UI content in the same patches.

Practical implication: Use thicker strokes (4-6px minimum) for annotations that need to survive downsampling. Or use annotation tools that render at a thickness relative to the final output size, not the original capture size.

Color Conflicts with UI Elements

A red arrow on an interface that uses red buttons, red error messages, or red status indicators is visually ambiguous. The model must distinguish between red-as-annotation and red-as-content, which requires contextual reasoning that isn't always reliable.

Practical implication: Choose annotation colors that contrast with the UI being annotated. If the interface is blue-themed, use red annotations. If the interface uses red heavily, use blue or green.

Multiple Annotations Compete for Attention

When a screenshot has 5 or 6 annotations, the model must parse each one independently and understand their spatial relationships. Attention is distributed across all visual tokens, and each additional annotation dilutes the attention available for the others.

Practical implication: Limit annotations to 1 to 3 per screenshot. If you need to highlight 6 issues, use two screenshots of 3 each rather than one screenshot of 6.

Arrow Direction Is Ambiguous

A diagonal red line could be an arrow pointing from A to B, or an arrow pointing from B to A, or simply a line connecting two regions. The arrowhead — which indicates direction — is a small detail relative to the shaft and may not survive downsampling.

Practical implication: Use distinct, sharp arrowheads. Ensure the head is large enough relative to the shaft to be recognizable. Some tools render triangular arrowheads rather than lines, which survive better.

What Vision Models Are Good At

The same pipeline that struggles with annotations excels at other types of visual information:

Text in images. Modern vision models read rendered text with near-perfect accuracy. Button labels, error messages, form fields, navigation items — all reliably extracted. This is why embedding metadata as text (AI Context Banners) is more reliable than embedding it as graphical elements.

Layout structure. The model can identify navigation bars, content areas, sidebars, footers, and modals with high accuracy. The spatial arrangement of large UI blocks is well-preserved through downsampling and patch tokenization because these structures span many patches.

Color and contrast. The model detects major color differences between elements. A button that's the wrong shade of blue, a background that should be white but is gray, a text color that doesn't have sufficient contrast — these are reliably identified because color information is well-preserved in patch embeddings.

Relative positioning. "The button is too far from the input field" or "the sidebar is wider than expected" — spatial relationships between large elements are handled well because positional embeddings encode the spatial arrangement of patches.

Optimizing Screenshots for AI Consumption

Based on the pipeline described above, here are concrete steps to maximize the accuracy of AI interpretation:

Crop Tightly

Don't send a full-screen screenshot when the issue is in a 400×300 region. Cropping reduces the downsampling ratio, preserving more detail in the area that matters. A 400×300 crop downsampled to 768×768 retains far more detail per patch than a 2880×1800 full-screen shot.

Use High-Contrast Annotations

Annotations should be at least 4px wide in the final image, in a color that contrasts strongly with the surrounding UI. Red on light backgrounds, white or yellow on dark backgrounds. Avoid thin lines, small circles, or annotations that blend into the UI color scheme.

Add Text, Not Just Arrows

Instead of (or in addition to) drawing an arrow, add a short text label directly on the image: "gap too large" or "wrong color" near the issue. Vision models read text more reliably than they interpret graphical annotations. A text label is unambiguous. An arrow is sometimes ambiguous.

One Issue Per Screenshot

Send separate screenshots for separate issues. The model handles focused, single-issue images much more reliably than multi-issue composites. Each screenshot with one annotation and one issue is a clear signal. Five annotations on one screenshot is noise.

Include an AI Context Banner

Burn viewport dimensions, source app, and annotation summary into the image as a text strip. The model reads this text reliably and uses it to calibrate its interpretation of the visual content.

State Your Intent in Text

The image shows what exists. Your text prompt should say what you want. "The gap between the header and the card is 32px. It should be 16px." Don't make the model guess what you want changed — the image shows the current state, the text describes the desired state.