Python · MuJoCo · PyTorch · PPO · Inverse Kinematics
GenMotion2Physics — Generated HHI to Physical Execution
GenMotion2Physics follows generated human-human interaction motion from kinematic output to articulated execution in physics, then analyzes where interaction structure is preserved, altered, or lost.
Research Context
I built G2P (GenMotion2Physics) to connect generated human-human interaction motion to physical execution.
- A generated interaction may look correct kinematically but fail when two agents affect each other through contact and coupled dynamics.
- Conversely, a physically stable rollout does not guarantee that the intended interaction was formed or preserved.
- For the paired study, I used Word-Motion Cross Attention (WMCA), my word-level conditioning extension of the InterMask HHI generator from my earlier first-author research, with InterMask itself as the generation baseline. Both were tested using the same handshake prompt and downstream G2P pipeline.
G2P pipeline. (a) A Word-Motion Cross Attention (WMCA)-based generator produces the Kinematic Source. (b) Pair-aware retargeting converts it into an Articulated Reference while preserving the pair’s relative position, facing, and timing. (c) Individually qualified controllers are executed together as Baseline Physics. (d) Cooperative Paired Control produces Cooperative Physics. (e) Cross-stage analysis compares interaction preservation, physical readiness, and physical execution across the four stages.
My Contribution
Retargeting Generated HHI to Physical Humanoids
- Built pair-aware retargeting that converts generated two-person motion into articulated MuJoCo references while preserving relative position, facing, and timing.
- Validated retargeting and individual tracking before paired execution, so representation or controller limitations were not automatically treated as interaction failures.
Executing Two Agents Under Coupled Physics
- After each agent passed individual tracking qualification, executed both controllers together as Baseline Physics, revealing failures that appeared only after the agents shared the same contact environment.
- Trained the pair with a shared tracking reward to produce Cooperative Physics, while keeping separate policies and self-only observations.
Diagnosing Interaction and Physical Execution
- Compared whether body-part relations and event timing were preserved (Interaction Preservation), whether foot-ground support remained available (Physical Readiness), and whether the pair stayed upright and tracked the motion to completion (Physical Execution).
- Built a synchronized diagnosis interface that connects these signals to the motion timeline and execution stage.
Result & Analysis
The two matched handshake cases exposed different failure locations under the same downstream pipeline.
- WMCA-generated case: hand proximity was present in the Kinematic Source and remained after retargeting. Baseline Physics produced real cross-human contact but failed with falls; Cooperative Physics completed all 232/232 frames while cross-human contact remained present.
- InterMask-generated case: the intended hand proximity was already absent in the Kinematic Source and remained absent after retargeting. The pair nevertheless completed physical execution stably with no cross-human contact, and cooperative training did not produce a better selected policy than the baseline.
- Together, the cases show that interaction formation/preservation and physical execution are distinct success criteria, and G2P can distinguish the stage at which the problem first becomes visible.
- These are matched case studies, not a model-level WMCA vs. InterMask performance comparison.