3DHP and DAVE

3DHP (3-Dimensional Hand Positions) was the focus of my master's project from the computer science department at the University of Alaska Fairbanks. It consists of using two cameras to calculate the location of hand markers in 3D world coordinates. This project was funded by the Arctic Region Supercomputing Center (ARSC) for use in the Discovery lab (their virtual reality lab).

Stereo transformation

Humans use both of their eyes to perceive depth information in our world. Similarly, we can gather information about the location of specific objects if we have two slightly differing views of the object. This project searches for a single object that's visible in a pair of calibrated cameras (we know where the cameras are, along with how they're oriented and some other specifics). Once located, we can use some math transformations to determine it's location in the world. 3DHP uses the HALCON machine vision library to assist with the locating and transforming of these positions.

The calibration plate as seen through the (a) left and (b) right camera

 

Hand markers

To assist with the locating of the user's hands, special hand markers were created using ordinary light-emitting diodes (LEDs) and ping-pong balls. The ping-pong balls spread the colored light across their surface, which allows us to search for the desired color range. Since we want to use two hands, there are two markers -- red and green.

Output

When the system locates a specific hand marker, it sends a multicast packet out to a network address, specifying the location of the given marker. Multicast packets allow multiple computers to receive the data simultaneously, even outside the local network.

Location

The system is set up in the Discovery lab, which is ARSC's visualization and virtual reality laboratory. The cameras are set in the corners of the center display screen, which gives a view of the user standing on the floor screen. The user's hands are visible while standing on the center screen, which acts as a stage area.

DAVE

An application called DAVE (Digital Audio-Visual Environment) was created to be used with the 3DHP input device. DAVE consists of a group of sound samples that you can drag around the screen. Several parameters such as volume, playback speed, pitch, reverb, and delay/feedback can be manipulated for each sample, depending on their location on the screen. The different parameters are adjusted by simply dragging your hand up and down in the air.

Here's a quick video of DAVE being played in the lab:

DAVE was created using GLUT for the user interface and max/MSP for the audio processing. DAVE was created in the summer of 2005 with the help of Quinton Harris (a student employee at ARSC) and Sean Waite (ARSC summer intern).

Future Work

Future work for 3DHP and DAVE include migrating some of the code to make use of open source alternatives, such as the OpenCV computer vision library and PureData for the audio processing.