Adapting to the Body

examples frames of kinematic perception
This figure shows three snapshots of data from a sequence of activity monitored by the system and visualized using a hand-tuned kinematic model. The host reaches for a cup and drinks from it. The top row consists of images from the head-mounted camera. The bottom row shows depictions of the corresponding data provided by three of the four the Intersense devices.

The following is an excerpt from my thesis, which will soon be published in its entirety:

We would like to interpret the images from the camera and the orientations from the orientation sensors in a consistent way across different wearers and across sessions of use by the same wearer. A three dimensional kinematic model that represents the major axes of the limbs and the relative position and orientation of the camera serves as a well-grounded representation for the body of the wearer and its relationship to the sensors. As with standard motion capture systems, it can also serve as a useful representation for visualizing the output of the sensors. We have developed methods that use the streams of orientations and images to autonomously estimate a kinematic model that describes the wearer's body, the camera's relative orientation and position, and the relationship between the kinematic model and images captured from the camera. In addition, the system autonomously estimates the projected direction of gravity in the image for each pixel by using image motion to estimate the orientation of the head-mounted camera with respect to the head-mounted orientation sensor.

Ideally, we would like the wearer to don the equipment without worrying about the details of sensor placement, or calibration. Similarly, we would like to give the designer of the equipment flexibility in sensor placement, so that he can put more emphasis on factors such as comfort and ease of use. The more the system is able to autonomously adapt to changes in sensor position, sensor alignment, and the body of the wearer, the more easily the system will be used without error. The methods we describe impose few constraints on the wearer and designer since they only require the use of a camera with known intrinsic parameters, sensors that are in rigid alignment with the body parts, and the visibility of a portion of the hand's workspace from the camera. Moreover, in practice the constraint for rigid alignment with the body parts can be weakened.