NEXT-GENERATION MATRIX 3D IMMERSIVE USER INTERFACE [ M3D-IUI ] H Raghavendra Swamy AMD Senior Software Engineer

NEXT-GENERATION MATRIX 3D IMMERSIVE USER INTERFACE [ M3D-IUI ] H Raghavendra Swamy AMD Senior Software Engineer

SESSION AGENDA Quick Keywords Abstract and Scope Introduction Current User Interface [ UI ] and its limitations Current trends in 3D display space and compute Proposed Idea The Matrix 3D Immersive User Interface [ M3D-IUI ], pronounced as " I-Uee " The M3D-IUI compute engine The swarm algorithm Essence of swarm engine in M3D-IUI Role of OpenCL TM and AMD APU [ Accelerated processing Unit ] 3 Next-Generation Matrix 3D Immersive User Interface June 2011

SESSION AGENDA Sample applications The 3D icons / objects floating in 3D space The M3D-IUI desktop The context based windowless Matrix 3D-IUI The 2D canvas of Microsoft Paint expanded into 3D The FILE browser as the MATRIX FILE VAULT The Winamp TM visualization in true 3D The system tools and utilities Google search bar moved floating into 3D space The 3D web browser and 3D Markup language The true 3D video chatting Co-existence of existing 2D application windows 4 Next-Generation Matrix 3D Immersive User Interface June 2011

SESSION AGENDA Stack overview for 3D compute M3D-IUI software stack overview Conclusion Q and A 5 Next-Generation Matrix 3D Immersive User Interface June 2011

QUICK KEYWORDS 6 Next-Generation Matrix 3D Immersive User Interface June 2011

QUICK KEYWORDS APU : Accelerated Processing Unit OpenCL TM 3DMLW : 3D Markup Language for Web 3D Display Unit UI : User Interface 7 Next-Generation Matrix 3D Immersive User Interface June 2011

ABSTRACT AND SCOPE 8 Next-Generation Matrix 3D Immersive User Interface June 2011

ABSTRACT AND SCOPE A context based windowless Matrix 3D Immersive User Interface [ M3D-IUI ] built on swarm engine using OpenCL TM on Accelerated Processing Unit [ APU ] based platform Here scope is to show or highlight the essence of next generation M3D-IUI Note : Matrix 3D Immersive User Interface [ M3D-IUI ] is pronounced as M3D I-Uee Third person camera view immersive experience. 9 Next-Generation Matrix 3D Immersive User Interface June 2011

INTRODUCTION 10 Next-Generation Matrix 3D Immersive User Interface June 2011

INTRODUCTION The UI plays an important part in interfacing the compute unit with the outside world It has evolved from Simple switching / toggle lights Character based command consoles 2D Graphical Interfaces 3D Graphical Interfaces 11 Next-Generation Matrix 3D Immersive User Interface June 2011

CURRENT UI AND ITS LIMITATIONS 12 Next-Generation Matrix 3D Immersive User Interface June 2011

CURRENT UI AND ITS LIMITATIONS Historical trend: The rectangular, resizable graphical window object was introduced to align with the 2D display unit available at that time. The current user interfaces are 2D in nature; the interfaces are made to look 3D, but they are mapped onto 2D window display units like conventional flat panel display units. Hence, they are not truly 3D. The 3D graphical objects mapped onto the 2D display unit form the limitations. 13 Next-Generation Matrix 3D Immersive User Interface June 2011

CURRENT TRENDS IN 3D DISPLAY AND COMPUTE 14 Next-Generation Matrix 3D Immersive User Interface June 2011

CURRENT TRENDS IN 3D DISPLAY AND COMPUTE Date: CES 2011 [ Consumer Electronics Show 2011 ] Toshiba's glass-free 3D laptop prototype Link : http://ces.cnet.com/8301-32254_1-20026948-283.html Sony makes a push for 3D Link : http://ces.cnet.com/8301-32254_1-20027515-283.html AMD Fusion APU series debuts 15 Next-Generation Matrix 3D Immersive User Interface June 2011

THE MATRIX 3D IMMERSIVE USER INTERFACE [M3D-IUI] 16 Next-Generation Matrix 3D Immersive User Interface June 2011

PROPOSED IDEA The Matrix 3D Immersive User Interface [ M3D-IUI ] All Applications currently are not truly 3D in nature as they get mapped onto conventional 2D display units. The M3D-IUI incorporates an out-of-the-box approach by moving into windowless 3D environment, which is similar to real world. The idea is to break free from the window-bounded region and create a true 3D interface. Note : The term window is currently used as a generic representation of a rectangular resizable bounded graphical object, and should not be confused with or interpreted as the Microsoft Windows operating system NAME series. The inspiration being the real world, where objects of all shapes, sizes, colors seamlessly integrate into the same environment without the concept of window, except for cases like the paper on which this idea was written or the window on the wall 17 Next-Generation Matrix 3D Immersive User Interface June 2011

THE M3D-IUI COMPUTE ENGINE 18 Next-Generation Matrix 3D Immersive User Interface June 2011

THE M3D-IUI COMPUTE ENGINE The M3D-IUI is based on the following The swarm algorithm Example: Bee, Ants,. Essence of swarm engine in 3D-IUI Example: 3D Agents Role of OpenCL TM and AMD APU [ Accelerated Processing Unit ] M3D-IUI compute engine : Harness APU s array of SIMD computing power using OpenCL TM by mapping low level 3D computing agents to synthesize complex 3D objects and environment using swarm based algorithm. 19 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS 20 Next-Generation Matrix 3D Immersive User Interface June 2011

LIST OF SAMPLE APPLICATIONS THAT MAY BE MOVED INTO M3D-IUI The 3D icons / objects floating in 3D space The M3D-IUI desktop The context-based windowless Matrix 3D-IUI The 2D canvas of Microsoft Paint expanded into 3D The FILE browser as the MATRIX FILE vault The Winamp visualization in true 3D The system tools and utilities The Google search bar moved floating into 3D space The 3D web browser and 3D Markup language The true 3D video chatting The co-existence of existing 2D application windows 21 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS The 3D Icons / Objects Floating in 3D Space The matrix 3D-IUI proposes the 3D icon which gets mapped into the matrix 3D-IUI space It can be viewed 360 o and moved across in 3D space Current scenario 3D Icon in 2D display device The notion of full 3D feature is achievable: i.e., even grabbing and moving 3D objects, which gives the user more flexibility in interacting and manipulating the 3D content in true 3D style These graphical objects can also be made governed by the physical laws to give a more realistic feel to the end user Proposed 3D icon in true 3D space 22 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS The M3D-IUI Desktop P XYZ a Matrix 3D-IUI desktop objects with content tagging 23 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS The M3D-IUI Desktop Opps!!! Now ROOM Existing solution: The BumTop Desktop The sample 3D desktop mapped to 2D display can be seen in the BumpTop desktop by Dynamics Graphics Projects now part of Google Inc. Here bumptop provides a varied 3D-based command interface like pile, fish layout view, etc... It has shortfalls: The desktop is not fully 3D, but made to look 3D on 2D space, by having an inclined spatial position To have full-fledged 3D objects on desktop and not just 3D documents and images Source: http://www.ted.com/talks/anand_agarawala_demos_his_bumptop_desktop.html Proposed solution: The M3D-IUI Room The M3D-IUI room provides true 3D interface with third-person camera view It provides full-fledged 3D objects in 3D space Content tags can be added to give easier identification of document and arrangement This can be extended to audio and video tag pop-ups 24 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS The Context Based Windowless Matrix 3D-IUI Firefox Internet Explorer Common graphical objects moved outside the bounded window region into M3D-IUI space The M3D-IUI part is equipped with the context sensitivity / awareness, which can change the behavior of the UI based on the context. As a simple example explained below, the graphics objects are selectively computed rather than repeated computing depending on the context. Example 1: The current user interface in PowerPoint has different kinds of menus, icon, other GUI components that are common to other applications like Firefox, Internet Explorer, etc...gets regenerated or computed when switched back and forth. Here each GUI component or object is bound to the individual application window. These common menu items, icons and other GUI components can be moved outside the window so that the same components can be reused by changing the behavior based on the context thereby increasing efficiency. 25 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS The 2D Canvas of Microsoft Paint Expanded Into 3D The current Paint software has a 2D canvas space where 3D objects gets mapped. The 2D canvas space can be expanded to 3D space canvas where the 3D objects are moved, created and manipulated using 3D objects/tools located in 3D space and not in the current 2D layout inside the Paint application. This can be a boon for the 3D animation field and 3D movie post-production, as users will be viewing in true 3D and manipulating in a true 3D environment. 26 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS The FILE Browser as the MATRIX FILE Vault Current File Browsers C:Vault Matrix File vault 3D-IUI x: Drive partition X: 3D vault folder / directory Files Matrix File Vault M3D-IUI Racks/rows similar to book racks 3D arrangement of file on racks The matrix file vault M3D-IUI file management tool has the following features: The 3D vault Racks / rows 3D arrangement of files Content management The 3D vault is the 3D storage compartment which is the equivalent of a drive partition. The racks or rows are equivalent of folder or directory structure arrangement; the files are arranged in 3D layout on racks. The content management UI provides facilities to tag the file and file contents in hierarchical 3D space. 27 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS The WINAMP VISUALIZATION in True 3D Winamp player s audio synchronized visualization display, which is currently 3D on 2D displays, becomes full 3D giving the user a new kind of experience. The 3D audio equalization settings and the 3D Winamp skins/interfaces will give an ultimate new way to the user to interact with the audio player. This can be also extended to other audio players. 28 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS The System Tools and Utilities The system tools and utilities can be given a true 3D touch with some of the examples listed below: Example A The hard disk management utility can now show hard disk in full 3D view for easier management. Example B The matrix file vault 3D can be used to view and search old files and folders in a better way, so that they can be moved to part of the hard disk where access time is slow, and move the active part of the content where the access/seek time is fast. The network admin utility with the 3D touch provides an complete 3D view of network configuration for easier management. 29 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS Google Search Bar Moved Floating Into 3D Space The Google search bar gets suspended in 3D space and can be rotated 360 degrees. It can be made to always point to the user, using third-person camera effect. The search result can fill up the 3D space. Google Google search bar in 3D space 30 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS The 3D Web Browser and 3D Markup Language for Web The simple 2D graphics manipulation inside the browsers is controlled through the most popular scripting language HTML. These scripts have tags which describe 2D. The main intention was to reduce the load of graphics data on the network, but still have a graphics experience at the client end. Along the same lines, 3D Markup Language for Web [ 3DMLW ] can be used to minimize the 3D graphics load. This is an open source scripting language for 3D computations and manipulation on 3D-enabled browsers. Most of the popular browsers currently support the 3DMLW. This 3DMLW is a major advancement for easier transition into true 3D web interfaces and 3D user experiences. Since internet applications are gearing up for an increase in 3D support adoption, it would be ideal if the M3D-IUI is adopted so that websites and online stores can be searched in true 3D. 31 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS True 3D Video Chatting Adding the 3D touch to current video chatting will take user experience to the next level. 32 Next-Generation Matrix 3D Immersive User Interface June 2011

SAMPLE APPLICATIONS Co-existence of Existing 2D Application Windows The topic of discussion until now was all about 3D spaces. So what happens to the existing 2D-window based UI applications? Hmmm...so these applications can be layered on page layouts and can be folded, twisted, bent on the lines of "Dynamics Graphics Projects. It will be floating paper in 3D space, and hence the existing 2D window applications can still be used without any changes maintaining backward compatibility. 33 Next-Generation Matrix 3D Immersive User Interface June 2011

STACK OVERVIEW FOR 3D COMPUTE 34 Next-Generation Matrix 3D Immersive User Interface June 2011

STACK OVERVIEW FOR 3D COMPUTE 3D-Enabled Browser / Desktop M3D-IUI Swarm Engine OS APIs / Library OpenCL APIs and Runtime Layer The top module is the M3D-IUI desktop or application or 3Denabled browser which is compliant with 3D mark-up language for web natively through the support of 3D mark-up language plug-ins. The next layer forms the M3D-IUI swarm engine based API libraries. The next layer forms the OpenCL APIs and runtime layer, which finally gets mapped to the APU. CPU and GPU Component of APU overview stack for 3D compute 35 Next-Generation Matrix 3D Immersive User Interface June 2011

MATRIX M3D-IUI SOFTWARE STACK OVERVIEW 36 Next-Generation Matrix 3D Immersive User Interface June 2011

MATRIX M3D-IUI SOFTWARE STACK OVERVIEW True 3D-Based Application Space 3DML Browser 3D-UI Room 3D Vault 3D Audio Player Other 2D mapped applications 3DML Plug-in 3D UI component APIS like 3D Icons, 3D floating text, etc Low level APIs for stereo matrix 3D primitives implemented through OpenCL based swarm algorithm OpenCL runtime platform OS APIs Low Level drivers GPU, Mouse, USB, Touch pad, etc APU (CPU and GPU) Platform Hardware Accelerated M3D-IUI software stack on APU 37 Next-Generation Matrix 3D Immersive User Interface June 2011

38 Next-Generation Matrix 3D Immersive User Interface June 2011 CONCLUSION

CONCLUSION The idea here focuses on harnessing APU processing power using OpenCL to provide the user nextlevel 3D immersive experience by Matrix 3D-IUI, which is built on context-based swarm engine using open technologies like 3DMLW by leveraging the 3D display units in the space of laptop, netbook, desktop segments operating system user interface and also the internet applications which are moving towards the 3D space. 39 Next-Generation Matrix 3D Immersive User Interface June 2011

QUESTIONS

Disclaimer & Attribution The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. There is no obligation to update or otherwise correct or revise this information. However, we reserve the right to revise this information and to make changes from time to time to the content hereof without obligation to notify any person of such revisions or changes. NO REPRESENTATIONS OR WARRANTIES ARE MADE WITH RESPECT TO THE CONTENTS HEREOF AND NO RESPONSIBILITY IS ASSUMED FOR ANY INACCURACIES, ERRORS OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. ALL IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE ARE EXPRESSLY DISCLAIMED. IN NO EVENT WILL ANY LIABILITY TO ANY PERSON BE INCURRED FOR ANY DIRECT, INDIRECT, SPECIAL OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD arrow logo, and combinations thereof are trademarks of Advanced Micro Devices, Inc. All other names used in this presentation are for informational purposes only and may be trademarks of their respective owners. 2011 Advanced Micro Devices, Inc. All rights reserved. Microsoft, Windows, PowerPoint, and DirectX are registered trademarks of Microsoft Corporation in the United States and/or other jurisdictions. Google, BumpTop are the registered trademarks of Google Inc. in the United States and/or other jurisdictions. 41 Next-Generation Matrix 3D Immersive User Interface June 2011

Disclaimer & Attribution Toshiba is the registered trademarks of Toshiba Corporation in the Japan and/or other jurisdictions. Sony is the registered trademarks of Sony Corporation in the Japan and/or other jurisdictions. Winamp is the registered trademarks of NULL soft / AOL Inc. in the United States and/or other jurisdictions 3DMLW is developed and distributed under GNU GPL V3 license by the 3D technologies R & D. 42 Next-Generation Matrix 3D Immersive User Interface June 2011