0310d28b5e
Premier commit du cinquième front. Trois parties : - `unity/MyInfoMateVR/` : le projet Unity (6000.0.83f1, URP, Meta XR SDK 205), un APK unique pour tous les clients. Menu flottant à sélection au regard, appairage, chargement de scène GLB, POI, cache de contenu, télémétrie. - `unity-overlay/` : les mêmes scripts à recopier sur un projet Unity neuf, avec les pièges rencontrés consignés dans son README. - `viewer/` : viewer et éditeur de scène web autonome (Vite, TypeScript, three.js), partagé avec les autres fronts. - `docs/` : état des lieux, setup Unity, décisions d'architecture et plan d'exécution en 9 étapes. La scène est décrite par un `scene.json` poussé par `adb push` : l'app le préfère à celui embarqué dans l'APK. Les binaires (GLB, textures de l'échantillon Sponza, DLL Meta XR) passent par Git LFS dès ce premier commit — les y faire entrer après coup demanderait de réécrire l'historique. Les artefacts régénérés par l'éditeur et par CMake (`Library/`, `.utmp/`, Burst debug) sont ignorés. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
94 lines
7.3 KiB
Markdown
94 lines
7.3 KiB
Markdown
---
|
||
name: hz-meta-xr-operator-interaction-grab
|
||
description: "How to grab and manipulate Meta XR Interaction SDK grabbable interactables in Unity using Meta XR Operator. Covers identifying grabbable types (close-range HandGrab/GrabInteractable, distance grab, hinged interactables like lids/doors/drawers), performing the grab via aim pose, choosing controller motions that move the target as intended, and identifying which of the three movement providers (relative-to-hand, manipulate-in-place, pull-to-hand) an interactable uses."
|
||
allowed-tools:
|
||
- Bash(hzdb:*)
|
||
tags:
|
||
- agentic-xr
|
||
- openxr
|
||
- unity
|
||
- meta
|
||
- interaction-sdk
|
||
- grab
|
||
- hand-grab
|
||
- distance-grab
|
||
- movement
|
||
- interactable
|
||
---
|
||
|
||
# Meta XR Operator Interaction Grab
|
||
|
||
How to grab and manipulate Meta XR Interaction SDK grabbables — covers close-range hand/controller grabs, distance grabs, and constrained interactables (lids, doors, drawers). See **meta-xr-operator-grabbed-objects** for post-grab aiming, **meta-xr-operator-coordinates** for coordinate conversion.
|
||
|
||
## Identifying Grabbables
|
||
|
||
**Identify by components, never by GameObject name.** Use Unity MCP to enumerate components:
|
||
|
||
| Component | Meaning |
|
||
|---|---|
|
||
| `HandGrabInteractable` / `GrabInteractable` | Close-range hand or controller grab (most common). |
|
||
| `DistanceHandGrabInteractable` | Adds distance-grab capability to the same interactable. |
|
||
| `Grabbable` | Base grab target the interactable drives. |
|
||
| `IMovementProvider` impl | How the object moves while held: `MoveFromTargetProvider` → relative-to-hand, `MoveAtSourceProvider` → manipulate-in-place, `MoveTowardsTargetProvider` → pull-to-hand snap. |
|
||
|
||
The interactor lives under the camera rig — find it by `ControllerDistanceGrabInteractor` (or `DistanceHandGrabInteractor` for hand tracking), not a hard-coded path. Its selection volume is a `SelectionFrustum` cone (~10–15° half-angle), so aim only needs to be approximately on-target. If component inspection isn't available, fall back to the empirical test below — and say so.
|
||
|
||
## Grab Sequence
|
||
|
||
**Always use aim pose for both position and orientation.** Do not set grip pose — the Meta interaction system reads aim. (Setting aim also moves the derived grip pose at a ~5–8 cm offset, which is why aim-pose-only also works for close-range grabs.)
|
||
|
||
1. `set_controller_pose(hand, aim, position, orientation, base_space)` aimed at the target. For predictable downstream math, prefer **identity orientation** (`[0,0,0,1]`) from in front of the target — this leaves source_rotation ≈ identity so subsequent translations move the target 1:1.
|
||
2. `set_controller_input(hand, Grip, 1)`. Hold across frames; an immediate `0` may be missed. The grab latches on the frame grip is processed, only if a grabbable is hovered.
|
||
3. Move/rotate the **aim pose** (not grip) to manipulate.
|
||
4. `set_controller_input(hand, Grip, 0)` to release. Released objects often hover (no gravity in many sample scenes) — plan where to drop.
|
||
|
||
Note: the visible controller mesh follows grip pose, so it may look stationary or odd after aim-only updates — cosmetic, the grab system still reads aim.
|
||
|
||
### Close-range grab (HandGrab / GrabInteractable)
|
||
|
||
The most common case. Same aim-pose-at-target-with-identity-orientation pattern works. If the collider is small and the aim-to-grip offset misses, fall back to setting grip pose directly at the target.
|
||
|
||
### Distance grab (DistanceHandGrabInteractable)
|
||
|
||
Same sequence; the SelectionFrustum cone reaches the target from afar. Aim approximately at the object — exact alignment isn't required.
|
||
|
||
### Constrained interactables (lids, doors, drawers)
|
||
|
||
Grab anywhere on the movable surface (the far end from the pivot has the largest collider). Move the controller's aim pose roughly in the direction the object should travel — the constraint guides the path, so it doesn't have to be exact. Reverse direction to close. If a small move produces no change, you grabbed a non-movable part — re-grab on the movable surface.
|
||
|
||
## The Three Movement Providers
|
||
|
||
- **`MoveFromTargetProvider` (relative-to-hand)** — target tracks controller 1:1. Rotating the controller pivots the target *around the controller*, so distant targets swing through wide arcs.
|
||
- **`MoveAtSourceProvider` (manipulate-in-place)** — target stays anchored at grab-time position. Rotating the controller rotates the target *around itself*. Translating the controller still translates the target (rotated by the original grab orientation).
|
||
- **Pull-to-hand snap (`MoveTowardsTargetProvider` and similar)** — target snaps to the controller on grab, then follows it.
|
||
|
||
### Identifying the provider empirically (when components aren't readable)
|
||
|
||
Translation alone does NOT distinguish relative-to-hand from manipulate-in-place. Procedure:
|
||
|
||
1. **Grab and check pose immediately.** Position jumped to controller → **pull-to-hand**. Position unchanged but rotation matches controller's grab orientation → **manipulate-in-place**. Nothing changed → ambiguous, do step 2.
|
||
2. **Rotate controller in place** (no translation) by ~90° yaw and re-query target pose. Position barely moves → manipulate-in-place. Position swings dramatically (~controller-to-target distance × √2) → relative-to-hand.
|
||
|
||
Visual fallback for duplicate-named objects (where `unity_get_world_pose` only resolves the first): pull-to-hand makes the target jump to the controller; relative-to-hand swings it off-screen on rotate; manipulate-in-place keeps it roughly in place.
|
||
|
||
## Placing a Target at a Destination
|
||
|
||
| Provider | Strategy |
|
||
|---|---|
|
||
| Pull-to-hand | Move controller aim to destination; target follows within ~cm. |
|
||
| Relative-to-hand | `controller_target = destination − grab_offset` where `grab_offset = target_grab_pos − controller_grab_pos`. Keep orientation identity to avoid swing. |
|
||
| Manipulate-in-place | With identity-orientation grab: `controller_target = controller_grab_pos + (destination − target_grab_pos)`. Rotate aim in place to rotate without moving. |
|
||
|
||
For orientation: pull-to-hand and relative-to-hand rotate with the controller (relative-to-hand swings at distance); manipulate-in-place is safest for precise in-place rotation.
|
||
|
||
## Common Pitfalls
|
||
|
||
1. **Setting grip pose** — don't. Always `pose_type: aim`.
|
||
2. **Pressing grip without aiming first** — the SelectionFrustum must be hovering a grabbable on the grip-press frame. Set aim, take a screenshot (gives the engine a frame), then press grip.
|
||
3. **Not holding grip across frames** — `Grip=1` immediately followed by `Grip=0` is often missed.
|
||
4. **"Nothing happened, grab failed"** — relative-to-hand and close-range HandGrab produce *no visible change* until the controller moves. Don't release and retry; nudge aim by ~0.1 m and re-query target pose. If it moved with you, the grab worked.
|
||
5. **Aim line passes through another grabbable** — the closer one wins. If a previously-grabbed (floating) object is between you and the target, approach from a different angle.
|
||
6. **Manipulate-in-place rotates target on grab** — non-identity grab orientation snaps the target to that orientation. Use identity to preserve original rotation.
|
||
7. **Translation-only provider test** — ambiguous between relative-to-hand and manipulate-in-place. Always do the rotation-only test.
|
||
8. **Duplicate-named root objects** — `unity_get_world_pose("Name")` returns only the first sibling. Use screenshots or instance-ID enumeration for the others.
|