vr-app/unity/MyInfoMateVR/.claude/skills/meta-xr-operator-grabbed-objects/SKILL.md

name, description, allowed-tools, tags

name	description	allowed-tools	tags
hz-meta-xr-operator-grabbed-objects	Aims and positions objects currently held by simulated controllers in Meta Quest and Horizon OS XR apps, including pose-offset calibration and quaternion math for grip and aim poses.	Bash(hzdb:*)	agentic-xr openxr unity aiming grip aim grabbed orientation position offset

Meta XR Operator Grabbed Object Control

How to precisely aim grabbed objects (guns, tools, pointers) so a reference point (muzzle, tip) points at a target.

Core Concept: Pose Offset

When grabbed, the object's reference point doesn't align with the controller pose. There are two offsets — rotation and position — that are constant after a grab. Measure them once, then apply their inverses to aim or position the object.

Offsets depend on which pose type (aim vs grip) you use. Always calibrate — never skip based on assumptions about the grab code. The OpenXR-to-object mapping always has a residual offset.

Step 1: Grab the Object

Position the controller at the object and press grip (see coordinates skill for world-to-OpenXR conversion).

Step 2: Calibrate Offsets (once per grab)

Set the aim pose to identity, then read the reference point's pose:

set_controller_pose(aim, position: [0,0,0], orientation: [0,0,0,1])
get_world_pose("Controller/Anchor")    → anchor_pos
get_world_pose("Object/RefPoint")      → ref_pos, rotation (rx, ry, rz)

Rotation offset — the reference point's yaw at identity reveals the offset:

forward_openxr = (sin(ry°), 0, -cos(ry°))     // for yaw-only offset
Q_offset = math_build_quat(forward_openxr)
Q_offset_inv = [-Q.x, -Q.y, -Q.z, Q.w]        // quaternion conjugate

Position offset — vector from anchor to reference point in Unity world space:

P_offset = ref_pos - anchor_pos

Note: Recalibrate the position offset if the object's scale or model changes. The rotation offset is stable across those changes.

After computing the offsets, the controller and grabbed object should be moved to a default position in view (unless specified otherwise by the user or game). Keep held objects far enough from the camera so they don't fill the screen. In local (EyeLevel) space, recommended ranges from the camera are:

• Z (forward): minimum -0.4, closer makes objects too large
• Y (below eyes): -0.2 to -0.4
• X (lateral): ±0.1 to ±0.2 (positive = right hand, negative = left hand)

Step 3: Aim at a Target

1. Turn head to face the target (see coordinates skill).
2. Requery positions — head movement shifts the controller in world space, so it is critical to requery positions of objects after head movement:

   get_world_pose("Object/RefPoint")  → ref_pos
   get_world_pose("TargetObject")     → target_pos
   

3. Compute aim orientation and apply the inverse offset:

   dir = target_pos - ref_pos
   Q_desired = math_build_quat([dir.x, dir.y, -dir.z])
   Q_aim = math_multiply_quat(Q_desired, Q_offset_inv)
   set_controller_pose(aim, orientation: Q_aim)
   

4. Refine — the rotation shifts the reference point slightly. Requery ref_pos, recompute, and re-set once for better accuracy.
5. Act and verify:

   set_controller_input(Trigger, 1)
   openxr_capture_composited_image()           // visual check
   get_world_pose("TargetObject")              // "not found" = hit
   

Positioning the Reference Point

To place the reference point at an exact world position, subtract P_offset before converting to OpenXR:

anchor_pos = desired_pos - P_offset
openxr_pos = world_to_openxr(anchor_pos)       // see coordinates skill
set_controller_pose(aim, position: openxr_pos)

To position AND orient in one call, combine with the aim quaternion from Step 3.

Common Pitfalls

1. Not calibrating after grab — measure offsets after each grab.
2. Using stale positions — always requery after any head or controller movement.
3. Not turning head first — head rotation moves the controller. Turn head → requery → compute.
4. Mixing pose types — calibrate and aim with the same pose type (both aim or both grip).
5. Forgetting the inverse — Q_aim = Q_desired * Q_offset_inv, not Q_desired * Q_offset.
6. Aiming from object origin instead of fire point — use the actual muzzle/fire point child, not the object's transform origin.