Unity Engine · C# · Azure Kinect · Body Tracking · Off-Axis Projection
3D Parallax Body Tracking
A body-tracked off-axis projection system that turns a normal display into a viewer-responsive 3D window.
Problem
This project explored how body tracking can make a normal 2D display behave more like a window into a 3D scene.
- A flat display can show perspective, but the view usually stays fixed even when the viewer moves.
- This weakens depth perception because the screen does not react to the viewer’s physical position.
- The goal was to track the viewer’s head position and update the Unity camera and projection so the scene would shift as if the viewer were looking through a window.
Approach & Implementation
I led the team and implemented the Unity-side body-tracked parallax system.
- Built the Unity input layer around Azure Kinect Body Tracking SDK, using the nearest tracked body and deriving the viewer position from the left and right eye joint positions.
- Converted the tracked viewer position into screen-relative coordinates, then used it to move the virtual camera and update an off-axis projection matrix for the window-like parallax effect, while adding input filter feature by implementing average filter, exponential filter to make the rendering stable from noise.
- Built the analysis pipeline for an arrow-direction selection experiment. The script compared the correct direction vector with the user-selected vector, then calculated angular error, response time, and coordinate-distribution plots for normal 2D and body-tracked parallax conditions.
Result
The project produced a Unity prototype and a user experiment comparing normal 2D viewing with body-tracked 3D parallax viewing.
- The 3D parallax condition reduced average angular error from 22.69° to 15.26°, with standard deviation decreasing from 18.26° to 15.04°.
- The 3D condition took longer on average, 3.60s compared with 1.12s in 2D, suggesting an accuracy-time trade-off where users spent more time making more precise spatial judgments.
- The result supports the project’s core claim that body-tracked parallax can improve spatial direction estimation on a 2D display, even if it does not make the task faster.