This demo shows an experimental system allowing a slew of interactions using a mobile device in tandem with a display. It is extraordinary and one of the few new technologies that seem like a legitimate surprise—I think we all knew things like the Nest thermostat would come along at some point.

While the effect is jaw-dropping, the technology used to achieve this effect doesn’t seem too far-fetched. From the video’s description:

Our system enables accurate position tracking of a smartphone placed on or over any screen by displaying a 2D color pattern that is captured using the smartphone’s back-facing camera. The proposed technique can be implemented on existing devices without the need for additional hardware.

Gaming is an obvious application, but this could work with nearly anything. Imagine using your phone as a palette while creating a digital painting, or to collect scavenger hunt items or even to drop a few bitcoins into a donation jar—Very promising.

Check the video out here: http://vimeo.com/105950126