0310d28b5e
Premier commit du cinquième front. Trois parties : - `unity/MyInfoMateVR/` : le projet Unity (6000.0.83f1, URP, Meta XR SDK 205), un APK unique pour tous les clients. Menu flottant à sélection au regard, appairage, chargement de scène GLB, POI, cache de contenu, télémétrie. - `unity-overlay/` : les mêmes scripts à recopier sur un projet Unity neuf, avec les pièges rencontrés consignés dans son README. - `viewer/` : viewer et éditeur de scène web autonome (Vite, TypeScript, three.js), partagé avec les autres fronts. - `docs/` : état des lieux, setup Unity, décisions d'architecture et plan d'exécution en 9 étapes. La scène est décrite par un `scene.json` poussé par `adb push` : l'app le préfère à celui embarqué dans l'APK. Les binaires (GLB, textures de l'échantillon Sponza, DLL Meta XR) passent par Git LFS dès ce premier commit — les y faire entrer après coup demanderait de réécrire l'historique. Les artefacts régénérés par l'éditeur et par CMake (`Library/`, `.utmp/`, Burst debug) sont ignorés. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
60 lines
3.3 KiB
Markdown
60 lines
3.3 KiB
Markdown
---
|
||
name: hz-meta-xr-operator
|
||
description: Drives Meta Quest and Horizon OS XR apps from an AI agent via the Meta XR Operator OpenXR API layer and MCP tools, including setup, head-pose control, controller input, and runtime verification.
|
||
allowed-tools:
|
||
- Bash(hzdb:*)
|
||
tags:
|
||
- agentic-xr
|
||
- openxr
|
||
- mcp
|
||
- vr
|
||
- ar
|
||
- runtime
|
||
---
|
||
|
||
# Meta XR Operator Usage
|
||
|
||
Meta XR Operator is an OpenXR API Layer that gives AI agents the ability to perceive, understand, and manipulate VR/AR applications at runtime via MCP tools.
|
||
|
||
**Key constraint:** Meta XR Operator tools are ONLY available when the VR/XR application is actively running. Tools will return errors if the app is not active.
|
||
|
||
## Setup
|
||
|
||
Meta XR Operator is available as an MCP server. To setup:
|
||
|
||
- **If HZDB v1.3.0+ is installed as an MCP server but the Meta XR Operator tools (e.g., `openxr_*`) are not available**, run the **"install meta xr operator mcp proxy"** tool exposed by HZDB.
|
||
- Otherwise, prompt the user to follow steps within the Meta XR Core SDK's AI Tools window in Unity to set up Meta XR Operator.
|
||
|
||
## When to Use
|
||
|
||
Use Meta XR Operator to **interact with a running XR application at runtime**:
|
||
- **Iteration** — Launch the app, verify features, interact, then exit to make changes.
|
||
- **Debugging** — Reproduce and investigate runtime bugs by inspecting poses, scene state, and visuals.
|
||
- **Test Automation** — Programmatically verify features. See the **hz-meta-xr-operator-unity-test-mechanics** skill for more details.
|
||
|
||
## Orientation & Navigation
|
||
|
||
- **Start by orienting yourself.** Use `openxr_get_head_pose` + `openxr_capture_composited_image` to see where you are.
|
||
- **Use data before vision.** Query scene data (`get_scene_root_objects`, `get_children`, `get_world_pose`) to understand layout via coordinates, then confirm visually with screenshots.
|
||
- **Don't guess positions** — always use `get_world_pose` for both target and rig. Check positions relative to the camera rig, not world origin.
|
||
- **If you can't find something visually**, step back to get a wider field of view. If still not found after 2-3 screenshots, use data tools to get coordinates and navigate there directly.
|
||
|
||
For detailed coordinate math and controller positioning, see **hz-meta-xr-operator-coordinates**. For aiming grabbed objects (guns, tools), see **hz-meta-xr-operator-grabbed-objects**.
|
||
|
||
## Verification
|
||
|
||
- **Verify both via data AND visually.** Confirm results using coordinate/scene-data tools AND compositor screenshots.
|
||
- **Take screenshots after key interactions** to verify visual state matches expectations.
|
||
- **Check for visual artifacts** (aliasing, noise) by capturing screenshots from multiple angles.
|
||
|
||
## Input Simulation
|
||
|
||
- **Smooth movements:** Use the `duration` parameter on `openxr_set_head_pose` and `openxr_set_controller_pose` for realistic motion.
|
||
- **Controller input values:** Buttons are 0/1, Trigger/Grip are 0.0–1.0, Thumbstick is -1.0 to 1.0 on each axis.
|
||
- **Hold inputs across frames** — setting an input to 1 and immediately to 0 may be missed by `Update()`.
|
||
|
||
## Common Pitfalls
|
||
|
||
- **Tools only work when the app is running.** Connection errors mean the app may have stopped.
|
||
- **Coordinate systems matter.** OpenXR is right-handed (-Z forward). Unity is left-handed (Z forward). See the **hz-meta-xr-operator-coordinates** skill for conversion.
|