0310d28b5e
Premier commit du cinquième front. Trois parties : - `unity/MyInfoMateVR/` : le projet Unity (6000.0.83f1, URP, Meta XR SDK 205), un APK unique pour tous les clients. Menu flottant à sélection au regard, appairage, chargement de scène GLB, POI, cache de contenu, télémétrie. - `unity-overlay/` : les mêmes scripts à recopier sur un projet Unity neuf, avec les pièges rencontrés consignés dans son README. - `viewer/` : viewer et éditeur de scène web autonome (Vite, TypeScript, three.js), partagé avec les autres fronts. - `docs/` : état des lieux, setup Unity, décisions d'architecture et plan d'exécution en 9 étapes. La scène est décrite par un `scene.json` poussé par `adb push` : l'app le préfère à celui embarqué dans l'APK. Les binaires (GLB, textures de l'échantillon Sponza, DLL Meta XR) passent par Git LFS dès ce premier commit — les y faire entrer après coup demanderait de réécrire l'historique. Les artefacts régénérés par l'éditeur et par CMake (`Library/`, `.utmp/`, Burst debug) sont ignorés. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
106 lines
4.4 KiB
Markdown
106 lines
4.4 KiB
Markdown
---
|
|
name: hz-meta-xr-operator-grabbed-objects
|
|
description: Aims and positions objects currently held by simulated controllers in Meta Quest and Horizon OS XR apps, including pose-offset calibration and quaternion math for grip and aim poses.
|
|
allowed-tools:
|
|
- Bash(hzdb:*)
|
|
tags:
|
|
- agentic-xr
|
|
- openxr
|
|
- unity
|
|
- aiming
|
|
- grip
|
|
- aim
|
|
- grabbed
|
|
- orientation
|
|
- position
|
|
- offset
|
|
---
|
|
|
|
# Meta XR Operator Grabbed Object Control
|
|
|
|
How to precisely aim grabbed objects (guns, tools, pointers) so a reference point (muzzle, tip) points at a target.
|
|
|
|
## Core Concept: Pose Offset
|
|
|
|
When grabbed, the object's reference point doesn't align with the controller pose. There are two offsets — **rotation** and **position** — that are constant after a grab. Measure them once, then apply their inverses to aim or position the object.
|
|
|
|
Offsets depend on which **pose type** (aim vs grip) you use. **Always calibrate — never skip based on assumptions about the grab code.** The OpenXR-to-object mapping always has a residual offset.
|
|
|
|
## Step 1: Grab the Object
|
|
|
|
Position the controller at the object and press grip (see coordinates skill for world-to-OpenXR conversion).
|
|
|
|
## Step 2: Calibrate Offsets (once per grab)
|
|
|
|
Set the aim pose to identity, then read the reference point's pose:
|
|
|
|
```
|
|
set_controller_pose(aim, position: [0,0,0], orientation: [0,0,0,1])
|
|
get_world_pose("Controller/Anchor") → anchor_pos
|
|
get_world_pose("Object/RefPoint") → ref_pos, rotation (rx, ry, rz)
|
|
```
|
|
|
|
**Rotation offset** — the reference point's yaw at identity reveals the offset:
|
|
```
|
|
forward_openxr = (sin(ry°), 0, -cos(ry°)) // for yaw-only offset
|
|
Q_offset = math_build_quat(forward_openxr)
|
|
Q_offset_inv = [-Q.x, -Q.y, -Q.z, Q.w] // quaternion conjugate
|
|
```
|
|
|
|
**Position offset** — vector from anchor to reference point in Unity world space:
|
|
```
|
|
P_offset = ref_pos - anchor_pos
|
|
```
|
|
|
|
**Note:** Recalibrate the position offset if the object's scale or model changes. The rotation offset is stable across those changes.
|
|
|
|
After computing the offsets, the controller and grabbed object should be **moved to a default position in view** (unless specified otherwise by the user or game).
|
|
Keep held objects far enough from the camera so they don't fill the screen. In `local` (EyeLevel) space, recommended ranges from the camera are:
|
|
- **Z** (forward): minimum -0.4, closer makes objects too large
|
|
- **Y** (below eyes): -0.2 to -0.4
|
|
- **X** (lateral): ±0.1 to ±0.2 (positive = right hand, negative = left hand)
|
|
|
|
## Step 3: Aim at a Target
|
|
|
|
1. **Turn head** to face the target (see coordinates skill).
|
|
2. **Requery positions** — head movement **shifts** the controller in world space, so it is **critical** to requery positions of objects after head movement:
|
|
```
|
|
get_world_pose("Object/RefPoint") → ref_pos
|
|
get_world_pose("TargetObject") → target_pos
|
|
```
|
|
3. **Compute aim orientation** and apply the inverse offset:
|
|
```
|
|
dir = target_pos - ref_pos
|
|
Q_desired = math_build_quat([dir.x, dir.y, -dir.z])
|
|
Q_aim = math_multiply_quat(Q_desired, Q_offset_inv)
|
|
set_controller_pose(aim, orientation: Q_aim)
|
|
```
|
|
4. **Refine** — the rotation shifts the reference point slightly. Requery ref_pos, recompute, and re-set once for better accuracy.
|
|
5. **Act and verify**:
|
|
```
|
|
set_controller_input(Trigger, 1)
|
|
openxr_capture_composited_image() // visual check
|
|
get_world_pose("TargetObject") // "not found" = hit
|
|
```
|
|
|
|
## Positioning the Reference Point
|
|
|
|
To place the reference point at an exact world position, subtract P_offset before converting to OpenXR:
|
|
|
|
```
|
|
anchor_pos = desired_pos - P_offset
|
|
openxr_pos = world_to_openxr(anchor_pos) // see coordinates skill
|
|
set_controller_pose(aim, position: openxr_pos)
|
|
```
|
|
|
|
To position AND orient in one call, combine with the aim quaternion from Step 3.
|
|
|
|
## Common Pitfalls
|
|
|
|
1. **Not calibrating after grab** — measure offsets after each grab.
|
|
2. **Using stale positions** — always requery after any head or controller movement.
|
|
3. **Not turning head first** — head rotation moves the controller. Turn head → requery → compute.
|
|
4. **Mixing pose types** — calibrate and aim with the same pose type (both aim or both grip).
|
|
5. **Forgetting the inverse** — `Q_aim = Q_desired * Q_offset_inv`, not `Q_desired * Q_offset`.
|
|
6. **Aiming from object origin instead of fire point** — use the actual muzzle/fire point child, not the object's transform origin.
|