Element Picker: Point, Click, Fix

The Problem with Screenshots

The webview_screenshot tool is great for seeing the big picture — the AI can look at your whole page and spot obvious layout issues. But when you need to fix a specific element, a full-page screenshot falls short. The AI can't reliably tell that a button's font weight is 400 instead of 600, or that there's 12px of padding where there should be 16px. And even if it could, it still wouldn't know the element's CSS selector, computed styles, or which component file to open.

The element picker closes that gap. You point at the element, and the AI gets everything: tag, classes, computed styles, bounding rect, parent chain, and a cropped screenshot of just that element. It's the difference between "look at this page" and "here's exactly what's wrong and where to fix it."

Quick Start — Try It in 30 Seconds

Prerequisites

Your Tauri app needs the MCP bridge plugin installed and withGlobalTauri enabled. If you haven't set that up yet, follow the Getting Started guide first.

Step 1. Type a /select command with what you want to do:

/select this button should be green instead of blue

Step 2. A blue overlay appears in your app. Move your cursor to highlight elements, then click the one you mean.

Step 3. The AI receives the element's metadata and screenshot, then responds using that context.

Example session:

You: /select the font size on this heading looks too small

AI: I'll activate the element picker so you can show me which heading.

    [Blue overlay appears in your app]
    [You hover over the heading — it highlights with a blue outline]
    [You click it]

    I can see the heading you selected:
    - Element: <h2> with class "section-title"
    - Current font-size: 14px
    - CSS Selector: .content-area > h2.section-title

    That does look small for a section heading. I'll update the CSS
    to bump it to 20px...

INFO

That's all most people need. The sections below go deeper into the two picking modes, what metadata the AI receives, and real-world workflows.

Two Ways to Point

There are two conversation styles for selecting elements, depending on who initiates the pick.

"Hey, show me" — Agent-Initiated

The AI activates a picker overlay and waits for you to click. This is what happens when you use /select or when the AI decides it needs you to point something out.

A translucent blue overlay covers your app. As you move your cursor, elements highlight with a blue outline and a tooltip shows their tag, ID, and dimensions. Click to select.

"Look at this" — User-Initiated

You Alt+Shift+Click an element at any time while using your app — no need to ask the AI first. A green flash confirms the selection was captured. Later, come back to the AI and it can retrieve what you pointed at.

This is great for when you notice something while testing and want to flag it without breaking your flow.

	Agent-Initiated	User-Initiated
Who starts it	AI activates the overlay	You Alt+Shift+Click
Visual cue	Blue overlay + highlight	Green flash (fades after 1.5s)
Timing	Synchronous — AI waits for your click	Asynchronous — pick now, discuss later
Cancel	Escape key or X button	N/A (it's instant)
Best for	AI asks you to point something out	You notice something while working

TIP

You don't need to remember which tool does what. Just describe what you want — "look at the element I pointed at" or "let me show you which button" — and the AI picks the right approach.

What the AI Sees

When you select an element, the AI receives a detailed metadata report plus a cropped screenshot. Here's what a typical report looks like:

Element: <button>
ID: submit-btn
Classes: btn btn-primary lg
CSS Selector: .form-actions > button#submit-btn.btn.btn-primary.lg
XPath: /html/body/div/main/form/div[3]/button
Bounding Rect: x=340, y=512, width=120, height=44

Attributes:
  type = "submit"
  disabled = "false"
  data-testid = "submit-button"

Text Content: "Save Changes"

Computed Styles:
  background-color: rgb(59, 130, 246)
  color: rgb(255, 255, 255)
  font-size: 14px
  padding: 8px 16px
  border-radius: 6px
  cursor: pointer

Parent Chain:
  div.form-actions (340x60)
  └─ form#settings-form (680x540)
     └─ main.content (680x800)
        └─ div#app (1024x768)

The AI also receives a cropped PNG screenshot of just the selected element, so it can see exactly what it looks like on screen.

Full field reference

Field	Description
Element	HTML tag name
ID	Element's id attribute (if set)
Classes	Space-separated CSS class list
CSS Selector	A unique CSS selector path to the element
XPath	XPath expression to the element
Bounding Rect	Position (x, y) and size (width, height) in pixels
Attributes	All HTML attributes on the element
Text Content	Visible text content (truncated to 200 characters)
Computed Styles	Key CSS properties as computed by the browser
Parent Chain	Ancestor elements with tag, id, classes, and dimensions
Screenshot	Cropped PNG image of the selected element

Real-World Workflows

Fix This Style

A designer flags a visual issue. Use /select to point at the element and let the AI propose a fix.

You: /select this button doesn't match our design system — wrong border radius

AI: I'll activate the element picker so you can show me the button.

    [You click the button]

    Got it. Here's what I see:
    - Element: <button> with class "action-btn"
    - Current border-radius: 2px
    - Located in: src/components/ActionBar.vue

    Your design system uses 8px border-radius for buttons.
    I'll update the .action-btn class in ActionBar.vue...

What Is This?

You've inherited a codebase and want to understand what a UI element actually is.

You: /select what component renders this widget? I can't find it in the source

AI: Let me activate the picker so you can show me.

    [You click the widget]

    That element is:
    - Element: <div> with class "metric-card"
    - CSS Selector: .dashboard-grid > div.metric-card:nth-child(3)
    - data-testid: "revenue-metric"

    Searching your codebase... This is rendered by the
    MetricCard component at src/components/MetricCard.tsx:42,
    used in the Dashboard page at src/pages/Dashboard.tsx:18.

I Noticed Something

You're testing the app and spot a glitch. Alt+Shift+Click it on the spot, then come back to the AI later.

[While testing, you notice a layout glitch and Alt+Shift+Click the element]
[Green flash confirms the selection]
[You continue testing...]

You: I pointed at an element earlier — there's a layout shift happening on that component

AI: Let me retrieve the element you pointed at.

    I can see the element:
    - Element: <div> with class "sidebar-nav"
    - Width: 248px, Height: 100vh
    - position: fixed, left: 0

    The layout shift is likely caused by the sidebar not reserving
    space in the document flow. Since it's position: fixed, the
    main content area isn't accounting for its 248px width...

The Picker UX — What Happens on Screen

Desktop

When the agent-initiated picker activates:

1. A translucent blue overlay covers the app
2. As you move your cursor, elements highlight with a blue outline
3. A tooltip appears showing tag#id.class (WxH) for the hovered element
4. A small cancel bar appears with move and close buttons
5. Click an element to select it, or press Escape to cancel

TIP

If the cancel bar covers the element you want to pick, you can drag it to a different position.

Mobile

On mobile devices, the picker uses a two-tap flow:

Two-Tap Flow on Mobile

The first tap highlights the element. The second tap confirms the selection. This prevents accidental selections on touch screens. Cancel by tapping the X button.

1. First tap — the element highlights with a blue outline
2. Second tap — confirms the selection and returns metadata
3. Cancel — tap the X button on the cancel bar

Alt+Shift+Click (User-Initiated)

When you Alt+Shift+Click an element:

• A green flash border appears around the element and fades after 1.5 seconds
• No overlay or picker UI appears — it's a quick, non-intrusive capture
• The selection is stored until the AI retrieves it

INFO

Alt+Shift+Click is available on desktop only. On mobile devices, use the agent-initiated picker instead.

Tips

• Chain selections — You can select multiple elements across a conversation. Each /select adds more context for the AI to work with.

• Combine with DOM snapshots — After selecting an element, use its returned CSS selector with webview_dom_snapshot to see the full subtree structure around it.

• Custom timeouts — If you need time to navigate to the right screen before picking, the picker supports timeouts up to 120 seconds (default is 60 seconds).

• Use selectors with other tools — The CSS selector returned by the picker works with webview_interact, webview_get_styles, webview_find_element, and other webview tools for follow-up actions.

Troubleshooting

Picker overlay doesn't appear

• Is your Tauri app running? Start it with tauri dev.
• Is withGlobalTauri enabled in your tauri.conf.json?
• Is the MCP bridge plugin installed and registered?
• Does the AI have an active session? It should start one automatically, but you can ask it to check with driver_session status.

Clicking does nothing

• Make sure you're clicking the app content, not the cancel bar.
• On mobile, remember the two-tap flow — first tap highlights, second tap confirms.
• Elements inside iframes may not be selectable.

Alt+Shift+Click — no green flash

• Confirm the MCP bridge plugin is loaded (check the browser console for bridge initialization logs).
• Alt+Shift+Click is desktop-only. On mobile, use the /select command instead.

Screenshot failed

• The screenshot uses html2canvas under the hood, which has some limitations with certain CSS features (e.g., complex filters, some SVG elements). The element metadata is still returned even if the screenshot fails.

Timeout

• The default timeout is 60 seconds. If you need more time to navigate to the right screen, ask the AI to use a longer timeout (up to 120 seconds).

See Also

• API Reference: UI Automation — Full tool specifications for webview_select_element and webview_get_pointed_element
• API Reference: Prompts — /select prompt specification
• Using Prompts — Guide to all available slash commands
• Suggested Prompts — Copy-paste prompt examples