Magic Torch: Designing for Mobile Interaction with Augmented Objects
About
Have you ever been in a bookshop and pulled out your phone to check its online reviews? What about when you’re shopping for wine, or admiring a museum piece?
“Designing for Mobile Interaction with Augmented Objects” is a research project that initiated by Dr Martin Tomitsch that involved creating, testing, and evolving interactive prototypes of an augmented reality interface, evaluating a range of user gestures.
The findings were turned into a paper for the proceedings of the 2012 International Symposium on Pervasive Displays.
Team & Role
As a Creative Coder, Interaction Designer & Animator, I prototyped different live gestural interfaces throughout the project in addition to the planning and setup of the technological environment for user tests. Meanwhile, Mark C Mitchell iteratively worked on the beautiful visuals. Martin Tomitsch mentored us with his experience in the field, and covered the academic aspects of the project.
Process
With the idea of a mobile phone communicating wirelessly with a projection powered by another computer, the first thing done was to build a basic feasibility test.
After a few trials, this was accomplished with OSC Messages sent from the phone via the Touch OSC app, to the computer where it was parsed via a simple Processing sketch (translating continuous to discrete values), which were then forwarded to a projected Flash Animation - also via OSC. The Computer was also later connected to a Kinect for additional gestures.
[Phone:TouchOSC] -osc-> [Computer:Kinect+Processing] -osc-> [Flash_Animation]
With the viability of the concept ticked off, we moved on to working on the visuals. Mark worked on a few different visual concpets for feedback from us. In iterative cycles, we would animate the visuals, incorporate them into the system, evaluate them, and re-iterate.
Multiple forms of mobile interactions were attempted including buttons, sliders, twisting & rotating the phone, and other air-gesture-based interactions.
A selection of interactions were chosen for user testing and are published in the following paper.
Paper
Abstract
Handheld projectors represent an emerging type of pervasive display that can provide situated, personalised access to digital information. Beyond the projection of content such as photos or websites they can be used to augment physical everyday objects with digital information. This approach has several advantages compared to other augmented reality (AR) technologies. For example handheld projectors eliminate the need to wear head-mounted displays and provide a true integration of the virtual overlay with the physical environment. Yet little is known regarding user interaction with projection-based AR interfaces. The work described in this paper aims to fill this gap through an explorative comparison of four interaction techniques. The techniques were evaluated with two iterations of a projection-based interface that augments physical books with digital information. The interface was prototyped with a fixed projector and a depth-camera sensor to simulate handheld projection. The paper provides a discussion of the techniques along with insights regarding the interface design of projection-based AR interfaces.
