Mocap and video are out of step in the React training sessions

React motherboard episodes used by the vtm_cosmos and MoT world models · middle camera vs OptiTrack gel pose · 15 fps

What was checked. Every action row k is computed from the sensor pose at frame k and frame k+1, and training pairs it with the video move from frame k to k+1. That only holds if the pose timestamps match the video. For each episode and hand, the speed of the gel centre from mocap was compared with how much the image changes in a 40 px box around the projected gel centre. The two signals were cross-correlated over offsets of ±6 frames (±400 ms), and the peak was refined to a fraction of a frame.

2026-09-11 train

-91 ms

-1.36 fps15 frames · range -118 to -68 ms · 39 hand-episodes

2026-09-12 train

-104 ms

-1.55 fps15 frames · range -107 to -94 ms · 8 hand-episodes

2026-09-09 val0909

+27 ms

+0.41 fps15 frames · range +20 to +42 ms · 6 hand-episodes

In the two training sessions the video moves about 94 ms before the mocap reports it, very consistently across all 25 training segments. Val0909 is close to aligned (+27 ms). So action row k in the training data describes a move that already happened about 1.4 frames earlier in the video, and the timing differs between training and val0909 by about 121 ms.

Correlation against time offset

Mean over all hand-episodes of each session. A peak left of zero means the image changes before the mocap step.

2026-09-11 train2026-09-12 train2026-09-09 val0909

Offset per episode

Circles are the left hand and hollow squares the right hand. Faded marks had a peak correlation below 0.4 and are left out of the medians.

2026-09-11 train2026-09-12 train2026-09-09 val0909left handright hand

The two signals over time

Five seconds of smooth motion, one point per fps15 frame. Both signals are standardized so they share one axis: gray is the image change near the gel, color is the mocap speed, and r is their correlation over the stretch. In the training stretch the orange peaks come one to two frames after the gray ones; moving the mocap +1.41 frames lines them up. The training stretch was picked as the one where the shift changes the correlation most, and the val0909 stretch as the one with the highest correlation, so they illustrate the effect rather than measure it. The per-episode numbers above are the measurement.

image change near the gel (video) mocap speed at recorded time mocap speed after moving it by the session offset

What it looks like on the video

Crops of the middle camera at full resolution during smooth, fast moves. The orange ring is the gel centre projected from mocap at the frame's recorded time. The blue dot is the same projection after moving the mocap by the session's median offset (+1.41 frames for training, -0.41 for val0909). The gel centre sits at a fixed spot on the handheld sensor, where the handle meets the loop. In training, the orange ring lags behind that spot along the direction of motion and the blue dot stays on it. In val0909 the two nearly coincide.

Training sessions (2026-09-11 and 09-12)

motherboard_0911_episode_000 frames around 1134 — motherboard_0911_episode_000 · left hand · frames 1132–1136 · gel centre moving about 30 mm per frame · the two markers are 34 px apart at frame 1134 (640×480)

motherboard_0911_episode_003_seg00 frames around 2861 — motherboard_0911_episode_003_seg00 · right hand · frames 2859–2863 · gel centre moving about 31 mm per frame · the two markers are 8 px apart at frame 2861 (640×480)

motherboard_0911_episode_007_seg01 frames around 761 — motherboard_0911_episode_007_seg01 · right hand · frames 759–763 · gel centre moving about 31 mm per frame · the two markers are 22 px apart at frame 761 (640×480)

motherboard_0912_episode_000_seg00 frames around 13889 — motherboard_0912_episode_000_seg00 · left hand · frames 13887–13891 · gel centre moving about 26 mm per frame · the two markers are 25 px apart at frame 13889 (640×480)

motherboard_0912_episode_003 frames around 3958 — motherboard_0912_episode_003 · left hand · frames 3956–3960 · gel centre moving about 30 mm per frame · the two markers are 31 px apart at frame 3958 (640×480)

Val0909 session (2026-09-09)

motherboard_0909_episode_000 frames around 2211 — motherboard_0909_episode_000 · left hand · frames 2209–2213 · gel centre moving about 31 mm per frame · the two markers are 9 px apart at frame 2211 (640×480)

motherboard_0909_episode_001 frames around 2576 — motherboard_0909_episode_001 · right hand · frames 2574–2578 · gel centre moving about 30 mm per frame · the two markers are 11 px apart at frame 2576 (640×480)

motherboard_0909_episode_002 frames around 516 — motherboard_0909_episode_002 · right hand · frames 514–518 · gel centre moving about 29 mm per frame · the two markers are 13 px apart at frame 516 (640×480)

What to do about it

Re-time the 2026-09-11 and 09-12 poses by about +1.4 fps15 frames, which is about +2.8 frames at the native 30 fps, and regenerate the action trees (sensor, view and force-informed actions). Then rerun this check; every session should peak near 0 ms.
Models trained on the current trees learned a 94 ms action delay that val0909 does not have. That is one more difference between training and val0909, next to the calibration epoch and the wrist cameras.
Check the force and tactile streams separately. The dataset notes say the tactile capture has no lag, so they may already line up with the video.
The fastest mocap rows include tracking glitches, where the pose jumps several centimetres for a frame or two while the video does not move. The examples above use only smooth moves.

Data table

Per-episode offsets (ms) and peak correlations

episode	session	left offset	left peak r	right offset	right peak r
motherboard_0911_episode_000	2026-09-11 train	-93	0.73	-90	0.59
motherboard_0911_episode_001_seg00	2026-09-11 train	-107	0.67	-90	0.78
motherboard_0911_episode_001_seg01	2026-09-11 train	-97	0.90	-89	0.61
motherboard_0911_episode_002_seg00	2026-09-11 train	-106	0.71	-105	0.63
motherboard_0911_episode_003_seg00	2026-09-11 train	-103	0.69	-118	0.58
motherboard_0911_episode_004_seg00	2026-09-11 train	-95	0.72	-86	0.38
motherboard_0911_episode_004_seg01	2026-09-11 train	-91	0.76	-87	0.67
motherboard_0911_episode_005_seg00	2026-09-11 train	-81	0.79	-76	0.40
motherboard_0911_episode_005_seg01	2026-09-11 train	-88	0.69	-81	0.39
motherboard_0911_episode_005_seg02	2026-09-11 train	-98	0.71	-82	0.60
motherboard_0911_episode_005_seg03	2026-09-11 train	-89	0.75	-76	0.74
motherboard_0911_episode_005_seg04	2026-09-11 train	-91	0.71	-79	0.68
motherboard_0911_episode_005_seg05	2026-09-11 train	-91	0.73	-77	0.61
motherboard_0911_episode_006_seg00	2026-09-11 train	-99	0.72	-92	0.37
motherboard_0911_episode_006_seg01	2026-09-11 train	-103	0.72	-107	0.48
motherboard_0911_episode_007_seg00	2026-09-11 train	-83	0.67	-68	0.70
motherboard_0911_episode_007_seg01	2026-09-11 train	-98	0.64	-88	0.61
motherboard_0911_episode_007_seg02	2026-09-11 train	-90	0.69	-92	0.61
motherboard_0911_episode_007_seg03	2026-09-11 train	-90	0.67	-98	0.62
motherboard_0911_episode_007_seg04	2026-09-11 train	-104	0.66	-98	0.72
motherboard_0911_episode_008_seg00	2026-09-11 train	-106	0.60	-85	0.72
motherboard_0912_episode_000_seg00	2026-09-12 train	-103	0.69	-103	0.67
motherboard_0912_episode_000_seg01	2026-09-12 train	-94	0.73	-107	0.71
motherboard_0912_episode_002	2026-09-12 train	-106	0.68	-105	0.66
motherboard_0912_episode_003	2026-09-12 train	-105	0.64	-101	0.64
motherboard_0909_episode_000	2026-09-09 val0909	+20	0.82	+36	0.74
motherboard_0909_episode_001	2026-09-09 val0909	+42	0.80	+27	0.73
motherboard_0909_episode_002	2026-09-09 val0909	+27	0.81	+25	0.63

Method: checkpoints/hf_spaces/build_react_sync_data.py. Image change is the mean absolute gray-level difference between consecutive fps15 frames (256 px, same crop as the latent trees) inside a 40 px box around the gel centre projected with the middle-camera calibration. Offsets are in fps15 frames; one frame is 66.7 ms.