Visual Feedback for Agents

AI coding tools have gotten increasingly good at understanding visual feedback. Some take an all-in-one approach: v0, Lovable, Bolt, and Cursor's visual editor let you select elements directly and iterate on what you see. Others, like Claude Code, are more flexible but rely on screenshots and descriptions.

There's often a tradeoff: point at elements but stay locked in, or use any tool but describe everything in text.

This problem becomes especially obvious when working on animations and interactions, which is what got me thinking about how to give better visual feedback to agents in general.

The Translation Problem

Coding agents suck at frontend because translating intent is lossy. Here's what happens when you want a UI change:

1. You create a visual representation in your brain
2. You write a prompt like "make this button bigger"
3. The agent turns your prompt into a trajectory (grep/search for where this code lives)
4. It tries to guess what you're referencing
5. It edits the code

Search is a pretty random process since language models have non-deterministic outputs. Depending on the search strategy, trajectories range from instant (if lucky) to very long. This means added latency, cost, and degraded performance.

Current Solutions

There are two approaches today:

You control: Prompt better. Use @ mentions, write longer and more specific prompts, provide file paths manually.

Providers control: Make agents better at search. This is still unsolved research. Papers like Instant Grep and SWE-grep are trying to speed up code search, but the fundamental problem remains.

Ultimately, reducing the number of translation steps required makes the process faster and more accurate. This scales with codebase size—the larger your project, the more time wasted on search.

But what if there was a different way?

React Already Knows

React maintains exact source locations for every component through its fiber tree. When you see an error, you get a stack trace:

Error: Something went wrong
    at Button (components/ui/button.tsx:42:19)
    at Card (components/card.tsx:18:5)
    at HomePage (app/page.tsx:12:3)

This information exists at runtime. React knows exactly which file, which line, and which column every component came from. The question is: could we extract this information and give it directly to the AI agent?

The Solution

We built a system that intercepts React's component tree using Bippy. When you select an element, we return context like this:

<button class="px-4 py-2 bg-blue-500..." type="button">
  Click me
</button>
  in Button (at components/ui/button.tsx:42:19)
  in Card (at components/card.tsx:18:5)
  in HomePage (at app/page.tsx:12:3)

This format is intentionally modeled after React's error stack traces. AI models have seen millions of these during training—they know exactly what to do with them.

How It Works

Step 1: Fiber Tree Access

Chrome extensions run in an isolated world. They can access the DOM but not JavaScript objects like React's fiber tree. To get component info, we need to run code in the page's world.

import {
  getFiberFromHostInstance,
  getOwnerStack,
  safelyInstallRDTHook
} from 'bippy'

// Install before React loads
safelyInstallRDTHook()

// Get component info from any DOM element
const fiber = getFiberFromHostInstance(element)
const frames = await getOwnerStack(fiber)

The getOwnerStack function walks up the fiber tree and returns stack frames with:

• functionName — Component name (e.g., "Button")
• fileName — Source file path
• lineNumber / columnNumber — Exact source location

Step 2: Path Normalization

Bundlers transform file paths into internal formats. Webpack, Turbopack, and Next.js all have their own conventions:

(app-pages-browser)/./components/ui/button.tsx
webpack://my-app/./src/Header.tsx
turbopack://[project]/app/page.tsx

We normalize these back to actual source paths:

const BUNDLER_PREFIXES = [
  /^\(app-pages-browser\)\//,
  /^\(app-client\)\//,
  /^webpack:\/\/[^/]*\//,
  /^webpack-internal:\/\/\//,
  /^turbopack:\/\/\[project\]\//,
]

for (const prefix of BUNDLER_PREFIXES) {
  if (prefix.test(filePath)) {
    filePath = filePath.replace(prefix, '')
    break
  }
}

We also look for project markers like src/, app/, components/ to strip any remaining path prefixes and get clean, relative paths.

Step 3: Context Formatting

With normalized paths, we format the context string:

export const formatContextFromStack = (
  htmlPreview: string,
  frames: StackFrame[],
  maxLines: number = 3
): string => {
  const stackContext: string[] = []

  for (const frame of frames) {
    if (stackContext.length >= maxLines) break

    const normalized = enhancedNormalizeFileName(frame.fileName)
    let line = `\n  in ${frame.functionName} (at `
    line += `${normalized.path}:${frame.lineNumber}:${frame.columnNumber})`
    stackContext.push(line)
  }

  return `${htmlPreview}${stackContext.join('')}`
}

Step 4: Agent Directive

We add an explicit instruction to the agent prompt telling it to use the file paths directly:

IMPORTANT: The component stack shows exact file paths and line:column numbers.
Open the specified file directly - do NOT search for files.
For example, if you see "components/ui/button.tsx:42:5",
open that file immediately at line 42.

Why This Format Works

Familiarity: AI models have been trained on React errors, Stack Overflow posts, and GitHub issues. They've seen this format millions of times.

Parent context: The component hierarchy shows where props and styling come from. If you want to change a button's color but the color is defined in the parent Card component, the stack tells you that.

Precise targeting: Line and column numbers let the agent jump to the exact location. No searching, no guessing.

Results

Here's what changes in practice:

Before (text-only prompt):

1. Search for files matching "button" (2-3 tool calls)
2. Read candidate files (2-4 tool calls)
3. Find the right component in the right file
4. Make the change

After (with visual context):

1. Open components/ui/button.tsx at line 42 (1 tool call)
2. Make the change

The difference compounds. Every UI change that would have required search now goes straight to the file. On a medium-sized codebase, this cuts round-trips in half and saves seconds per interaction.

Edge Cases

Non-React pages: We fall back to HTML with class names and document structure. Not as precise, but still useful context.

Library components: Components from node_modules are marked as non-editable. We show them in the stack for context but indicate they can't be modified.

Server components: Detected and marked appropriately. The stack shows which components rendered on the server vs. client.

Try It

1. Install the UI Studio extension
2. Start your provider:

UISTUDIO_CWD=/your/project pnpm dev:cursor

3. Open your React app in dev mode
4. Click any element—watch the agent jump straight to the file

What's Next

Prop extraction: Include component props so the agent understands the API before making changes.

Style tracing: Show which CSS classes come from which files, especially useful for Tailwind.

Multi-file awareness: Pre-compute related files (tests, stories, types) so the agent can update them together.

Beyond React: Similar approaches should work for Vue, Svelte, and vanilla JavaScript. The fiber tree is React-specific, but the idea of extracting source locations from runtime state applies broadly.

The goal is to eliminate translation loss entirely. You point at what you want to change, and the agent knows exactly where to look.