13 / 16 · SCREENSHOT LOCATE & EDITPLAN

Screenshot Evidence & EditPlan: Turning Vague Feedback into Auditable Plans

When a client sends a screenshot saying "move that cabinet on the left and make it shorter," how do you prevent AI from editing the wrong object?

TAKE AWAYAn `EditPlan` specification pairing `FeedbackEvidence` with whitelisted operators.

Clients rarely specify exact glTF node paths like `/root/tokonoma/pillar_L`; they click the viewport and say "make this narrower." VECTOR bridges this gap in two steps: 1. Viewport clicks capture immutable **FeedbackEvidence** (screenshot, camera matrix, pixel coords, and raycast candidates); 2. A dual planner (DeepSeek LLM + rule fallback) compiles natural language into a whitelist-validated **`EditPlan`** that requires human confirmation before execution.

The method: three small steps

A trustworthy 3D Agent never executes blind black-box edits: 1. **Grounded Localization**: `locate` restricts the LLM to ranking only real raycast candidate parts and outputs `certain / needs_confirmation / uncertain`; 2. **Plan Before Execution**: `EditPlan` explicitly declares target Stable Part IDs, whitelisted operators, exact parameters, and protected parts—halting in `waiting_confirmation` until approved.

Capture immutable FeedbackEvidence (screenshot + camera + pixel + raycast hits)
Clicking the viewport freezes the source asset version ID, camera matrix, pointer pixel, and raycast candidate list.
Rank verified candidates and evaluate confidence tier
The locator ranks only genuine raycast candidates against the user prompt—discarding any hallucinated IDs—and flags low-confidence hits for confirmation.
Compile whitelisted EditPlan and await human confirmation
Compile a structured `EditPlan` via the DeepSeek planner (with automatic rule-parser fallback and warnings) and pause in `waiting_confirmation`.

In the Screenshot Localization & EditPlan Compiler sandbox below, inspect how a viewport click on Tatami Room (3acfc61e) compiles into an auditable `EditPlan`.

3D PIPELINE SANDBOX · D13

Raycast Localization & Controlled EditPlan Compiler

AT RISK · NEEDS FIX

VECTOR ASSET #3acfc61eTatami Room with TokonomaArchitecture · 69.8 MB · GLB

Inspect Live 3D Asset ↗

On Tatami Room (3acfc61e), a user clicks the right tokonoma alcove pillar and writes: "Adjust this pillar height to 2.40m and do not touch the tatami mats."

Select a pipeline strategy to inspect metric deltas and gate status:

Mode A: Unconstrained LLM Guesses Part Name from Screenshot & Auto-Executes

✕ The LLM hallucinates a non-existent `wood_pole_2` node or shrinks the entire load-bearing wall without human confirmation.

Mode B: Raycast Candidate Lock + Dual-Planner Whitelisted EditPlan + Confirmation Gate (Recommended)

✓ Locks onto verified raycast hit `part_tokonoma_pillar_R`, registers tatami mats in `protectedParts`, and pauses in `waiting_confirmation`.

Delivery Risk Detected

Switch to the recommended VECTOR controlled pipeline option to resolve topology, UV, or QA gate errors.

ONE IDEA TO KEEP

Bind screenshots to immutable version and camera evidence, constrain LLM localization to real raycast hits, and always review the compiled `EditPlan` before execution.

Run it in the VECTOR 3D workbench or Blender pipeline

Open a matching reference asset in the VECTOR 3D workbench or your Blender + AI pipeline.
Copy the task sheet below, let AI generate an inspectable checklist or EditPlan first, and confirm before running mesh or material operations.

Task sheet with the complete example materials

[Natural Language to Controlled EditPlan Task Sheet]
Asset: Tatami Room (Version V1, Asset ID `3acfc61e`)
Viewport Click Evidence: Pixel (642, 380), Raycast candidate hits: `["part_tokonoma_pillar_R", "part_wall_north", "part_tatami_mat_01"]`.
User Instruction: "Resize the tokonoma pillar I just clicked to 2400mm along the Y-axis and add a 2mm edge bevel; do not touch the tatami mats or the north wall."
Compile an inspectable `EditPlan` specification containing:
1. Selected target part ID (must come from the raycast candidate list) and rationale;
2. Serial whitelisted operator steps and executor assignment (Blender bmesh Worker);
3. `protectedParts` list and confirmation gate status.

CHECK YOUR UNDERSTANDING

In VECTOR’s 3D editing layer, why does the system compile an `EditPlan` and pause in `waiting_confirmation` instead of mutating the model immediately?

Because the server disk is full.So the operator can verify target parts, operators, parameters, and protected regions before spending compute or API credits, preventing unintended mutations.Because 3D models can only be edited at midnight.

REFERENCES & FURTHER LEARNING

VECTOR 3D Asset Agent · Live Asset Showcase ↗Khronos Group · glTF 2.0 Specification ↗Blender Manual · Mesh & Cycles Baking ↗

Explanations, cases, and exercises are authored by Reallier. Links provide further learning. The workplace activity and people are fictional.

Capture immutable FeedbackEvidence (screenshot + camera + pixel + raycast hits)

Rank verified candidates and evaluate confidence tier

Compile whitelisted EditPlan and await human confirmation

In VECTOR’s 3D editing layer, why does the system compile an `EditPlan` and pause in `waiting_confirmation` instead of mutating the model immediately?