IT S THE INTERFACE, STUPID! FJORD S CHRISTIAN LINDHOLM AND 13TH LAB S PETTER IVMARK OFFER THOUGHTS ON WHY AUGMENTED REALITY STILL SUCKS

Transcription

1 IT S THE INTERFACE, STUPID! FJORD S CHRISTIAN LINDHOLM AND 13TH LAB S PETTER IVMARK OFFER THOUGHTS ON WHY AUGMENTED REALITY STILL SUCKS

2 OVERVIEW This Fjord report takes a look at responses to AR... Augmented Reality (AR) has been around for a while now, and despite great advances in technology, it is usually a disappointing experience for the user. So far it looks like there is yet to be any kind of true utility or value added by AR, and its implementation is largely confined to a few simple games and labelled as a gimmick. Yet everyone, young and old, technical or not, somehow instinctively feels that this technology is part of the future, and will eventually shape how we interact with the world around us. So why the gap between AR vision and reality? And what solutions can we find to understand and adapt this underachieving technology?...and how we can formulate a new way of managing and navigating the crucial layer between virtual and real worlds. In particular, we examine how an early breakthrough could be made by bringing understanding, as well as augmenting, of reality to the mobile web browser and all the millions of sites that already exist on the Internet, instead of trying to build a new Internet using use case dedicated apps and proprietary content solutions We also question whether the term AR is even right for this area as augmenting reality is only a part of the bigger shift, which is actually understanding the reality around you. Computer Vision may be a better term for the area. IT S THE INTERFACE, STUPID 2

3 WHAT REALLY HAPPENED WHEN MOBILE EXPLODED, AND WHAT CAN WE LEARN FROM IT? There was never one killer application that cracked mobile. Instead, the mobile explosion arrived when users could connect to all the already-existing Internet services, such as Facebook, Twitter, Google, and media sites whether through apps or the browser. It took off when developers (and largely Apple) actually managed to crack the user interface to interact smoothly with the existing web content within the constraints of the mobile form factor. Fjord believes we may be in a very similar situation regarding AR today. Innovators in AR are trying to create new content universes (be they layers, channels or other metaphors) in proprietary augmented reality languages, we think this is the wrong approach. It is not about creating a new Internet for reality, it is about connecting the existing one to reality. Just as most Internet properties now have a mobile version of their web page and user interface, we believe they will eventually have a reality version of web page adapted to be rendered on top of reality. IT S THE INTERFACE, STUPID 3

4 CONNECTING VIA THE INVISIBLE GLOVE In his keynote speech at the 3GSM conference in 2001, writer Douglas Adams outlined his vision of an Invisible Glove. He described it like this: If every piece of information knew the when of itself and the where of itself, so that the virtual world we created fitted over the real world like an invisible glove, these devices, which we currently think of as telephones or PDA s, would be the devices that made that invisible world visible to us. The things which both generate the model and make it appear to us, they become windows from the real world into the virtual world which is everywhere around us. Fjord has collaborated with 13th Lab to explore exactly this type of approach to understanding the link between real and virtual worlds. We found that to realise Douglas Adams vision users require not a new glove, but a type of lining between humans, machines and the environment. This way of looking at the relationship also exposes the inadequacies of AR, and the limited options offered by tagging approaches such as QR codes and Near Field Communication (NFC). IT S THE INTERFACE, STUPID 4

5 THE LIMITATIONS OF CURRENT APPROACHES TO TAGGING Physical tags, whether visual such as QR codes or radiobased such as NFC, are great ways to connect and align digital information into the physical world. However, current technologies do not support spontaneous discovery of this digital information. QR codes are very poor from a user interface point of view. When a user sees a code, he or she first has to scan it, which in turn opens a web page, and only once that page is fully loaded do they learn if the content connected to the code/physical object was even interesting. This is the equivalent of a user interface in which the lag between click and response may be seconds: no one in 2012 would use a web page with that kind of response time, so why do we expect them to do that on a mobile device? With NFC, we have another possible way to embed virtual tags into real objects. These command physical space, but are invisible to the eye, making them more versatile, and the only thing needed is a common icon or interface indication to users. But like other codes, they are not human readable and do not provide useful and physically compact hints as to where the tag will take the user. There is no doubt a place for QR codes and NFC models, but right now they look like an unsatisfactory solution to connecting the physical and the digital worlds. IT S THE INTERFACE, STUPID 5

6 GIVING THE WEB EYES: WHY IT S COMPUTER VISION, NOT AUGMENTED REALITY THAT IS THE BIG DEAL Fjord wonders if innovators will find that AR is really just a subset of a much wider and more interesting concept and technology: computer vision. This is technology designed around the concept of actually making a computer see What if a web page knew what it was looking at? Once a computer can see, recognize and have limited understanding of something in front of it or it s surroundings, it can do a lot of things, one of which is to augment that reality with computer graphics. But the interesting thing is actually that the computer understands what is in front of it and can act on it, not whether it chooses to augment the view with an image or not. IT S NOT ABOUT AUGMENTING REALITY - IT S ABOUT UNDERSTANDING IT To investigate these options, Fjord has been working with 13th Lab on a new type of web browser. This would let an ordinary HTML/JavaScript web page tell the device to notify it of things that appear in front of the camera. As an example, you could load the web page of a museum you visit on your phone s browser. On the surface it would look and work just like an ordinary web page, but as you hold the camera up towards a painting in the museum, the web page will automatically understand what it is looking at and show you all the information about a painting. 13th Lab s technology can therefore recognise and put a user interface on both 2D surfaces and 3D environments but this alone is not enough. Putting the interface in the digital world instead of the physical ultimately means something subtle yet revolutionary: the physical object or environment becomes simply the context and model through which you browse and navigate the digital content attached to it. Taking this vision a step further opened new and intriguing questions. What if it could react to things in front of the camera, much as it can already adapt its content based on the device s GPS position? This would open the door to a smarter web and likely change the Internet forever. IT S THE INTERFACE, STUPID 6

7 This may seem backwards, but consider the example of scanning the front page of a large newspaper or a print magazine with your eyes. It s still a vastly superior experience compared to browsing the same content on a small mobile screen. However, interacting with that physical content, for example seeing likes, comments, videos etc, is best done through a digital interface. Realising this vision will require an improved web browser with computer vision technology and JavaScript hooks to make it work. Anyone who can create a web page can then create content, without any approval or centralised hosting system, creating whatever content or user interface they want. We suggest this will be successful for the same reasons that the original open web overtook various walled gardens driven by a corporate need for control. OVERALL, THIS WORK POINTS DIRECTLY TO PIONEERING WAYS IN WHICH DIGITAL INNOVATORS WILL NEED TO RENDER THE UI OF THE WEB ON TOP OF REALITY. THOSE THAT DO WILL BE REACHING FOR THE KIND OF PARADIGM SHIFT THAT TOOK PLACE WHEN APPLE RENDERED THE UI OF THE WEB ON A TOUCHSCREEN PHONE DEVICE. IT S THE INTERFACE, STUPID 7

8 ABOUT FJORD Fjord is a service design consultancy. We create useful, effective, and desirable digital services that people love. We work with companies to design breakthrough experiences that make complex systems simple and elegant. Founded in 2001, Fjord employs a diverse group of over 180 design experts in eight global creative hubs including Berlin, Helsinki, London, Madrid, New York, Paris, San Francisco, and Stockholm. Clients include BBC, Citibank, ESPN, Flickr, Foursquare, MSN, Nokia, Qualcomm and Telefonica, among others. Find out more about Fjord at or follow Fjord on ABOUT 13TH LAB 13th Lab gives the web eyes. Wouldn t it be great if a mobile web page could understand and automatically adapt to what is actually in front of the device s camera? 13th Lab s technology extends standard browsers to let an ordinary web page see and react to reality. It lets developers extend their existing web user interface to physical surfaces, rendering HTML and JavaScript on top of physical things. The web is finally connected to the real world! Download the PointCloud browser from the AppStore or visit for more information.