DAVE2

DAVE2 (Digital Audio-Visual Environment pt.2) is a blanket term for the ongoing research that I have been doing with single-camera computer vision applications for the Arctic Region Supercomputing Center (ARSC). The primary goal of these applications is to use the body as an input device, like a mouse or keyboard. This is done by subtracting a background image from new video frames and drawing a box around what's left -- people in the scene. Certain information about this box, such as its width, height, and location, are then used as input for various audio applications.

The information gathered from the cameras is basic numerical data, such as the width of a box, height, and location in the viewport. It is sent out to the network as OpenSoundControl packets, which can be used to control a multitude of applications, including graphical and audio applications. In this case, the information is sent to an audio application written with Max/MSP, a visual audio programming environment.

The primary iteration of DAVE2 consists of four different regions, with objects detected in these regions affecting the audio processing in different ways. You can drag your hand in the left-hand region to pick from a selection of sound samples. Dragging your hand in the top region changes the playback speed of the current sample, and dragging your hand in the right region affects the pitch. This allows users to interact with the environment and create different sounds by simply moving their hands around.

DAVE2.1 (part one) from Dave Krnavek on Vimeo.

DAVE2.1 (part two) from Dave Krnavek on Vimeo.

The current version of DAVE2 allows for multiple pages of these region boxes. A separate region, called a page slider, is used to swap between the different pages. In the current implementation, each page consists of different types of effects that can be assigned to the looping sound sample. This lets the user cycle between a multitude of different effects, such as volume, panning, pitch, playback speed, and a delay-feedback loop. It is currently being developed with input from Dr. Morris Palter, percussionist and music professor at UAF.

 

DAVE2 was developed using the OpenCV computer vision library. This library works with many platforms, which allows the application to be run on different operating systems, including Windows and Mac OS X. The bounding box information is sent using OpenSoundControl packets to the audio processing program, built with Max/MSP.

Future work in this area includes detecting multiple persons in the scene, enhanced graphics, and research into other uses for this kind of information.