Tele-Immersion

The capture of sensory data in three spatial dimensions in real time from multiple physically separate spaces, the projection of the data into shared virtual environments, and the projection of the virtual environments into immersive physical environments, constitute the emerging technology of tele-immersion.

This enterprise has engaged the skills of researchers in a variety of disciplines, including graphics, vision, and networking. We currently have several projects underway to improve the existing technology, including segmentation algorithms to recognize human beings and segment their data into individual limbs, mathematical models of human movement in order to identify actions, refinement of stereo algorithms to better discern depth information in less time, customized hardware architectures to optimize performance of existing algorithms, and ongoing experiments and refinements in the sharing of real-time video and audio data over long distances.

The current apparatus includes 48 cameras in 12 clusters. The images from each cluster are processed by stereo programs running in parallel on 12 computers, and the outputs of the stereo programs can be sent via Gigabit Internet II connection to another computer to be rendered into a 3-dimensional scene. The received data is displayed dichoptically to evoke the perception of depth. We also have hardware capable of capturing and playing sound from 4 microphones and 4 speakers, and 8 infrared lights that project patterns that facilitate depth detection.

Intended applications of tele-immersion include teleconferencing, distance learning, archiving and distribution of physical performances, and training for complex and hazardous tasks where physical visual interactions are the key component.

We have collaborated with researchers in other departments, such as Dance and Psychology, as well as researchers at other schools such as the University of Illinois, Urbana-Champaign.

This project is supported by NSF grant SCI-0549231.