Multi Touch Interaction on Stereoscopic Displays

„Analysis of effects of 2D and (stereo) 3D mixing“

  • „Multi-Resolution-Display System for Virtual Reality Setups“, Jürgen Grüninger, Hilko Hoffmann and Uwe Kloos, to be published

New 3D display devices, such as 3D televisions, auto stereoscopic displays as well as Nvidia’s 3D Vision technology, are becoming available to consumers these days and the up to separated worlds of the Internet, traditional television and computer games are merging more and more to one all encompassing information and entertainment environments. It is expected that the 3D Internet will be an important technical platform for that development. More and more real 3D content will be available combined with traditional 2D web pages. Stereoscopic display devices will be used to display and interact with this content but little research on appropriate, intuitive user interfaces and interaction techniques for combined 2D and 3D content has been done so far.

This project investigates possible, intuitive combinations of traditional 2D and 3D user interfaces as well as dedicated interaction techniques suitable for the intuitive interaction in mixed 2D and immersive 3D environments. In particular the project examines the use of the widely accepted multi-touch interaction combined with 2D and 3D user interfaces. The key issues are the reduced readability of text and small details on 3D displays, the need to match a 2D interaction surface within the perceived 3D space on an immersive display, and the potential visual mismatches of combined 2D / 3D content within one application.

Monoscopic MT table using the DI principle

Depending on the MT devices different protocols for touch detection are used. Rear projection MT tables with build in infrared cameras need to detect touches out of the infrared images of the camera. We realised a monoscopic MT table using the diffuse illumination (DI) principle and a stereoscopic installation with the Laser Light Plane (LLP – see The DI as well as the LLP are designed as MT tables users are standing in front of to interact with it. Additionally a monoscopic setup with a commercially available MT screen from PQLabs ( has been realised which is designed as system users can operate sitting in front of it.

Stereoscopic MT table using the LLP principle

An analysis of multiple MT frameworks and libraries such as Touchlib, CCV, mt4j, reactivision as well as the recently built in MT capabilities of Microsoft Windows 7 and some Linux derivates has been performed. For our demonstrators we have chosen CCV in the MT table setups and the standard Windows 7 MT functionality for the MT screen setup. An evaluation on how fast and reliable touches are identified by the different touch recognition technologies is still ongoing.

We implemented direct as well as indirect interaction. Direct interaction means that objects or interface elements are touched directly without using pointers or other user interfaces. In case of stereoscopic display that type of interaction leads to visual distraction and undesired side effects if virtual objects are touched that are located visually above the screen surface. The immersion and stereoscopic perception of the object might get completely destroyed because the fingers are forced to point on the surface whereas the object is above the surface. In order to avoid distraction we are implementing indirect interaction for the stereoscopic case.

User interface concept for indirect interaction and a blending zone. Navigation and object manipulation is done with the interface elements on the lower side of the MT surface.
User interface concept for direct interaction with a combined 2D frame and a 3D inlay. Object manipulation is done directly in the 3D inlay.

Object manipulation and navigation are performed with dedicated interaction areas so that there is no need to point directly on virtual objects. On other strategy would be to push virtual objects below the view plane, i.e. the screen surface, while interacting. The disadvantage is the steady change of user’s perspective on the displayed scenery that also might lead to nausea or even sickness.

General system architecture


Project team

Principal Investigator
Dr. Hilko Hoffmann

Jürgen Grüninger, M.Sc.