Creating User Interfaces Using Web-based Technologies to Support Rapid Prototyping in a Desktop Astrovisualization Software

Transcription

1 LiU-ITN-TEK-A--17/062--SE Creating User Interfaces Using Web-based Technologies to Support Rapid Prototyping in a Desktop Astrovisualization Software Klas Eskilson Department of Science and Technology Linköping University SE Norrköping, Sweden Institutionen för teknik och naturvetenskap Linköpings universitet Norrköping

2 LiU-ITN-TEK-A--17/062--SE Creating User Interfaces Using Web-based Technologies to Support Rapid Prototyping in a Desktop Astrovisualization Software Examensarbete utfört i Datateknik vid Tekniska högskolan vid Linköpings universitet Klas Eskilson Handledare Emil Axelsson Examinator Anders Ynnerman Norrköping

3 Upphovsrätt Detta dokument hålls tillgängligt på Internet eller dess framtida ersättare under en längre tid från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida Copyright The publishers will keep this document online on the Internet - or its possible replacement - for a considerable time from the date of publication barring exceptional circumstances. The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its WWW home page: Klas Eskilson

4 Linköping University Department of Science and Technology Master thesis, 30 ECTS Datateknik LIU-ITN/LITH-EX-A--2017/001--SE Creating User Interfaces Using Web-based Technologies to Support Rapid Prototyping in a Desktop Astrovisualization Software Skapande av användargränssnitt med webbtekniker för snabbt prototypande i en rymdvisualiseringsmjukvara i skrivbordsmiljö Klas Eskilson Supervisor : Emil Axelsson Examiner : Anders Ynnerman External supervisor : Alexander Bock Linköpings universitet SE Linköping ,

5 Upphovsrätt Detta dokument hålls tillgängligt på Internet eller dess framtida ersättare under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannenslitterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida Copyright The publishers will keep this document online on the Internet or its possible replacement for a period of 25 years starting from the date of publication barring exceptional circumstances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: c Klas Eskilson

6 Abstract In this report, the development and implementation of a desktop user interface framework is presented. It is built using web technologies and the Javascript framework React together with a web socket server to render the graphical user interface in an OpenGL environment. This is done by using the open-source framework Chromium Embedded Framework (CEF). The resulting framework and implementation has proven successful, with promising results both from a performance perspective and from a development rapidness perspective.

7 Acknowledgments First of all, I would like to thank everyone involved at Linköping University and New York University for giving me the opportunity to do this work, especially Anders Ynnerman and Claudio T. Silva. Working on this project in New York was an incredible experience. Secondly, I would like to thank Alexander Bock for being an excellent friend, supervisor and life saver. Furthermore, Emil Axelsson for providing excellent help. Also, everyone I met during my years in Norrköping friends, class mates and colleagues, both during school hours and after. During early mornings with too much coffee or during late nights with too many beers, I had a great time thanks to all the amazing people. Last but not least, I would like to thank my family for their support and help throughout my studies. Klas Eskilson Stockholm, October 2017 iv

8 Contents Abstract Acknowledgments Contents List of Figures List of Tables iii iv v vii viii 1 Introduction OpenSpace Motivation Aim Research questions Delimitations Related work 3 3 Theory Web browsers and CEF Web sockets Javascript UI frameworks Transpiling Data serialization: JSON Web workers CSS prepocessors Implementation Web browser Socket server and simulation control GUI Results 27 6 Discussion Results Implementation Source criticism The work in a wider context Conclusion Research questions Future work v

9 Bibliography 35 vi

10 List of Figures 3.1 An overview of the two-executable structure with processes and threads The screen space browser with controls. The browser window is showing a video from the OpenSpace Youtube page Screen shot showing some information about the project, using the GUI browser The layer interaction mask Overview of the mask update and user interaction process Overview of the rendering process Overview of the property sending mechanism Design mockups with different alternatives of numeric inputs The On screen GUI, immediately after application start. Overlooking Mars The On screen GUI with sidebar open. A loading animation is shown where the scene graph properties will be loaded Popovers on the bottom of the screen The calendar component Different states of a numeric input Alternative selection view with an expanded drop down menu The system menu, providing the user with some extra tools and links to information The filter list being used Loading indicators Different variations of properties Top of a movable window Scroll issue shown. How can the user be informed that there are elements outside the visible area? Content of a scrollable list fading out Visible scroll bar to the right Tree view using the collapsible component Error message when the GUI cannot reach the simulation FPS meter in the top right corner Final implemented GUI vii

11 List of Tables 3.1 Table showing developer satisfaction Implemented property types in the GUI, and which corresponding input mechanisms that is used viii

12 1 Introduction When building an graphical user interface, being able to iterate fast is key. This makes it possible to have user feedback tightly integrated into the process. Using a set of tools that allows for these iterations and simultaneously allows the developer to make quick changes is therefore a good step in the direction of building a well functioning user interface. In this report, the development and implementation of a desktop user interface framework is presented. The goal with this framework is to be able to quickly create rich user interface components that can be improved over multiple iterations. 1.1 OpenSpace OpenSpace is a astrovisualization software [1]. It is designed to visualize the intire known universe and uses dynamic data to allow for an interactive experience in order to explore the cosmos. It has been in development since 2014 and the project is a collaboration between Linköping University, NASA Goddard s Community Coordinated Modeling Center, American Museum of Natural History, New York University s Tandon School of Engineering and University of Utah s Scientific Computing and Imaging Institute. OpenSpace is an open source, non-commercial and freely available application. The data that is visualized comes from a number of different sources. It can be observations, simulations, space mission planning and operations. OpenSpace is a C++ application that uses C++ s close connection to the computer hardware to use as rapid data througput as possible, allowing for more powerful visualizations. Some of the visualizations that is possible today are a globe browsing mode, where planets surfaces can be explored, and a solar radiation visualization, where solar events can be visualized. OpenSpace supports both a regular desktop screen as well as dome planetariums, which allows for immersive experiences. 1.2 Motivation Having a framework that allows for graphical user interface (GUI) components to be designed, developed and improved is a step into making OpenSpace a software application that may be used by a user with little or no previous experience. The goal with this project is to improve the pace at which the graphical user interface may be developed and improved. This, in turn, is to be able to improve the overall experience of using OpenSpace. The idea is to make OpenSpace more accessible and easier to understand for both experienced and new users. The GUI that exists today is C++-based. It is created using a framework called Dear ImGui. This has proven useful for developers to give easy access to the configuration of the OpenSpace simulation, but it is limited in terms of changing its looks. While it might be possible to achieve a satisfying look, the lack in flexibility might still be an issue. This framework is, just like the suggested solution in this report, supporting cross-platform develop- 1

13 1.3. Aim ment. This means that instead of using platform-specific application programming interfaces (APIs) for GUI creation, the GUI is developed for all platforms using a unified method. 1.3 Aim In this project, a C++ framework called Chromium Embedded Framework (CEF) is investigated and added to the OpenSpace application. CEF is a web browser framework, allowing its host application to browse, render and interact with an arbitrary web site. Using CEF and web technologies, this project is about creating a foundation and an initial implementation of a GUI framework. Javascript, HyperText Markup Language (HTML) and Cascading Style Sheets (CSS) is used to built the user interface itself, and communication between the simulation and the user interface is done through web sockets [2]. Evaluating this from a perspective of performance and ease of development may tell if these technologies are wellsuited for cross-platform development in a C++ based application. Topics that will be investigated are communication, development pace, performance (in relationship to the other parts of the application and the GUI s impact on the performance in total) and extendibility. This means, can the GUI easily be improved and extended, so that control of future functionality may reach the user? 1.4 Research questions In order to find an answer to the above mentioned broad purpose and aim, answers to the following questions will be sought. 1. What are the drawbacks and advantages of building a graphical user interface using web technologies in a C++ application? 2. What are the drawbacks and advantages of communicating internally and externally using web sockets? 3. How can graphical user interface development affect the future of a software product? 1.5 Delimitations In order to limit the scope of the project, user testing will not be conducted. The project will, in other words, not be evaluated from a usability perspective. This ties together with the design and user experience. These two aspects of the project will be mentioned and presented briefly, mostly to not leave any big blank areas open. The components built and discussed here is assumed to be used on a laptop or desktop computer. Neither hand held devices, touch interfaces nor immersive environments, such as dome theatres or virtual reality devices, are the target of the user interface. While some parts of the project works across several platforms, Microsoft Windows 10 has been used as the primary development platform. No work has been put into making the project work on Linux or Macos, which OpenSpace normally supports. 2

14 2 Related work This project is not the first of its kind. On several previous occasions, web technologies has been used to create a graphical user interface for a computer application. Some of these applications include the music streaming service Spotify s desktop application, the messaging platform Slack and the text editor Visual Studio Code. As of September 2017, all of these use the framework CEF by some extent. Slack and Visual Studio Code use uses the CEF abstraction framework Electron and Spotify uses CEF in a non-disclosed way [3]. That said, finding research on CEF and its use is not easy. Neither is it easy to find research on one of CEF s popular abstractions, Electron, as it was only announced as recently as 2015 [4]. Using web technologies for desktop applications, however widespread of a phenomena it is, can be seen as uncharted territory from a research perspective. Inter-process communication (IPC), however, is widely researched. Internal communication between processes and environments can be considered an older and more fundamental problem than the richness of the GUI. By some extent, the Internet can be considered a broad-purpose IPC mechanism, where data is communicated between units and processes to send and receive instructions and information. CEF and Chromium has an IPC protocol that allows for communication between the different processes of the application [5]. This, combined with the custom Javascript integration of CEF, would make it possible to handle commmunication between the GUI and the simulation [6]. The IPC of Chromium and CEF has for instance been used to combine eye tracking technologies together with CEF to create web browser extensions to use eye tracking mechanisms [7]. Some web related communication resarch, for instance, are more focused on communication between different web contexts [8]. Not as much internal communication between a web-based GUI and the underlying web browser and its host application, which is what the main focus is here. There is also Electron s solution, which allows communication between the web browser window and a window-less server-like structure [9]. This, however, was not a viable option in the OpenSpace case, as the goal was to implement a web-based GUI into an already existing C++ application and Electron is an all-javascript environment using Node.js [10]. 3

15 3 Theory In order to explain some implementations and solutions, some terms and concepts needs to be presented and explained. 3.1 Web browsers and CEF Later, the implementation of a web browser within OpenSpace will be presented. But first, what a web browser is, the purpose of a web browser, a web browser s structure and some challenges that web browsers handles needs to be described. For the sake of clarity, a web browser as discussed in this paper is a software application, or a part of one, that receives an arbitrary web resource in the form of an Uniform Resource Identifier (URI) [11]. It then downloads this resource and presents it in an appropriate fashion Structure I order to meet the requirements of a web browser and handling all the content that is should be supported according to the HTML 5 standard, a web browser is an application with a complicated structure [12]. Among several other things, media download, web rendering and user interaction should all happen in the same application. All this preferably with little to no delay and without disrupting any of the other tasks. In the CEF and Chromium projects, the solution is handling most tasks through a mutliprocesses, multi threaded approach [13]. This allows multiple tasks to be handled simultaneously. A user s interactions should ideally not disrupt the playback of a video or a sound, for instance Executables in CEF Figure 3.1: An overview of the two-executable structure with processes and threads. 4

16 3.1. Web browsers and CEF CEF has support for two different executable models. The implementation uses either one or two separate executables for its sub processes. An overview can be seen in figure 3.1. The processes are launched with different command line arguments that determine the purpose of the launched process. These arguments get sent to either the CefExecuteProcess, which then takes care of determining what needs to be done. The single-executable structure is supported on Windows and Linux systems, but not on Macos systems. [13] In a single-executable implementation of CEF, the return code of CefExecuteProcess is used to determine wether or not execution of that process should be terminated. In a twoexecutable implementation, the main executable (that also runs the rest of the application) initializes CEF with an option parameter declaring the location of the sub-process executable. In this case, the sub-executable is a single-purpose executable that does little else than calling CefExecuteProcess to allow CEF to handle the incoming task. By using dynamic, shared libraries, the double-executable approach does not necessarily increase the application size, as the two executables share the CEF library. A potential drawback of a single-executable implementation is that the implementation becomes sensitive to where CefExecuteProcess gets called. If it is late in the application s start cycle, this may cause delays in CEF s sub-process spawning Processes in CEF In CEF, each different thread and process have separate purposes. They handle different tasks. Although the there might be similarities between the tasks, they are distinct. The main, browser, process handles window management (towards the host operating system), painting and network access. This process is the same as the host application, where the rest of the application s logic is run. Another process is called the render process. This is where Blink rendering and Javascript execution happens. Blink is Chromium s web rendering engine - it takes formatting information such as CSS style sheets and transform it into the visual result that eventually will be shown [14]. For safety and robustness, each unique combination of URL scheme and origin will trigger a new render process to be spawned. This separation and reusability of processes is considered to give a good balance of safety and computing resource usage at the same time as allowing instances of different web pages from the same web site to access each other according to the enforced security policy. [15] The other processes that may be spawned by CEF are related to so called plugins. These might be if a web site wants to show content that are developed by a third party, for instance. This might be Flash content, Java aplets, or similar. GPU powered content is also being handled by a separate process. An overview of the different processes together with its executables and threads can be seen in figure Threads in CEF Each process in CEF is also multi threaded. This is where the multi-dimensional parallelism structure of CEF comes in. The different threads handle different aspects of constructing the requested web page. There are many different threads used by different processes [13]. The most common ones are: The UI thread. This is the main thread of the browser process. It is the thread where the functioncefinitialize is called to initialize the CEF framework. The IO thread, where inter-process communication is handled and where network messages are processed. The render thread, which is the main thread of the render process. 5

17 3.2. Web sockets 3.2 Web sockets The Web socket technology is an extension of TCP sockets made for two-way communication between a client and a host. The protocol uses TCP as the transport layer [16]. By using a handshaking process, a persistent full-duplex connection between the peers may be established. This means that no further handshaking is required, since the TCP connection remains open. Web sockets are essentially a layer on top of TCP that adds web targeted security model, an addressing and naming mechanism to support several services on one port, a framing mechanism to describe the messages and other handshaking related mechanisms. [2] Arbitrary UTF-8 text messages or binary data may be sent over the connection [17]. These are interpreted as standalone messages, and the peers should treat them as such. This differs from raw sockets where the messaging back and forth can be seen as streams, and where some sort of breaking character or sequence must be used to distinguish messages from each other. The web socket frame header contains details of the message size. This tells the clients how to interpret the received frames and where the frames end Establishing and closing connections In order to establish a web socket connection, a handshake process using HTTP is used. The client requests a protocol upgrade to the web socket protocol. The successful response to the client-initiated request is a HTTP 101 response, stating that the used protocol should be switched [18, section ]. The server also has mechanisms to prove the validity of the response. This is done by combining two pieces of information; the Sec-WebSocket-Key value from the client s request and a fixed string, assumed to be unique to the web socket protocol. By concatenating these two and Base64-encoding [19] the hash of the combination generated using SHA-1 [20], the client may verify the server s response. The fixed string is assumed to not be used by any other protocol. This should validate the connection, so that the client knows that the server is compatible. To close a connection, another handshaking mechanism is in place. This is on top of the TCP connection closing mechanism, as that is not always end-to-end reliable and connections may be in a half-closed state with messages not yet fully transmitted [21, section ]. The web socket closing mechanism relies on a confirmation, where the first peer initiates the closing by sending a so-called closing frame and the second peer sends a closing frame in response. This way, the number of cases where data is lost in a half-closed state is reduced. Both peers may initiate the close handshake simultaneously. When closing the connection, an optional reason code may be provided to explain why the connection is being closed. This can be used to allow the other peer to make a good decision in how to act after the closing. 3.3 Javascript UI frameworks Web application development has over the years gotten more powerful as the available browser technologies and computing power has advanced. It involves more steps and the web browser takes care of more than only rendering HTML and CSS. There are multiple different web application frameworks today that helps the developer by abstracting away complexity React React is an open source javascript framework for building user interfaces. It manipulates the web browser s Document Object Model (DOM) to provide the developer with an applica- 6

18 3.3. Javascript UI frameworks Framework Developers Satisfaction React % Angular % Angular % Ember % Vue % Backbone % Table 3.1: Table showing developer satisfaction. tion programming interface (API) to build the interface. It uses a declarative paradigm to allow the developer to create components. These components may be reused and are highly configurable for the developer. [22] In React, the user interface can be described as a function of the the application s state. The state of a React component is either owned by itself, in which case it is called state, or it may be passed down from the parent component, in which case it is called props, or it may be a combination of the two. React components may be written is different ways. Either using the ES5-style [23] React.createReactClass, using ES6-style class extending or as so-called pure functional components, which are functions returning valid React elements [24] JSX How React elements are written may vary, too. With React, a XML-like Javascript superset called JSX was developed. See Listing 3.1 for a simple example. The second option is using the React API with thereact.createelement method, see Listing const element = <h1>hello, world!</h1 >; Listing 3.1: Simple JSX example snippet 1 // React. createelement ( component, props, c h i l dren ) ; 2 const element = React. createelement ( h1, null, Hello World ) ; Listing 3.2: Usage of React.createElement By removing the need for advanced API usage, such as React.createElement, and replacing it with the XML-like syntax, JSX allows the developer to understand what the end component will look like faster. JSX also comes with other features. One is HTML injection prevention to ensure no malicious HTML gets injected to the page. Another is a simplified syntax for dynamic element attribute assignment, see Listing const u r l = image. png ; 2 const element = <img s r c ={ u r l } / >; Listing 3.3: Element attributes in JSX Other front end frameworks Angular, Vue, Backbone and Ember are some of the most popular web development frameworks available today, alongside React. [25] These frameworks solve similar issues to React, but with varying philosophies and solutions. What also varies is the satisfaction among the developers, see table 3.1. The developers number show number of developer who said they have experience in the given framework. Satisfaction are the percentage of developers that said they would use the framework again. 7

19 3.4. Transpiling React and Vue are the most popular frameworks among their respective developers with satisfaction of 92% and 89%. Angular is the most used framework, but also with the second to lowest satisfaction rate at 47%. Angular 2 has the third highest satisfaction rate at 65%, and is a complete rewrite from Angular [26]. Backbone is the framework with the lowest satisfaction at 32%. 3.4 Transpiling In order to use modern javascript features without loosing support for older web browsers and servers that does not yet support these features, a technique called transpiling (combination of transformation and compilation) can be used. [27, p. 3-4] These features are amongst others classes and inheritance, modules, arrow functions and object destructuring. The technique is also used for JSX (see section ). This technique transforms the newer syntax to an older, more well-supported syntax automatically. This is often done through adding a build-step into the development workflow, similar to linting and other operations. This process is not needed for all new Javascript features, as some can be added to the programming environment by the developer through another technique called polyfills. [27, p. 4-5] 3.5 Data serialization: JSON JavaScript Object Notation (JSON) is a data format useful for serializing arbitrary data types into a transport-friendly format. It is a text format, meaning it is readable. In it, strings, numbers, booleans, and null-values can be stored. Structural types are arrays of objects. The syntax is derived from the object and array syntax of Javascript. [28] 3.6 Web workers In modern web browsers, a standard called web workers exist. It is a method of running multithreaded tasks in the browser. It is invoked by providing the URL to the script that should run and a set of options. In this separate worker, different tasks can be handled without blocking the main thread that takes care of the website s rendering and other Javascript execution. [29, 30] 3.7 CSS prepocessors Cascading style sheets (CSS) is how the design and look of web sites are described and controlled. CSS uses a global namespace of so-called classes and IDs to link HTML elements to style rules. While this is a powerful language that gives the developer access to GPU powered animations, compositioning and a vast number of styling rules, CSS is not flawless. This global namespace can become hard to handle as the number of style rules increase. CSS Variable management is not widely spread, and as of writing this globally supported by approximately 75% of the world s web browsers [31]. Support for functions and loops is non-existing, making code reusability harder. Consequently, the configurability and reusability in CSS is lacking. This is where CSS preprocessors come into the play. They can be seen as a superset of CSS, providing syntax, feature and tooling improvements to the language. Variables, functions, mixins and loops are common features of a CSS preprocessor. Sass [32], Less [33] and Stylus [34] are three examples gaining popularity over the years. Using these tools, CSS becomes more manageble for the developer and might improve how a web site can be configured and styling adjusted. [35] These prepocessor does, as the name suggest, process CSS before it reaches the browser. They can bundle several files into one and thereby reduce the number of HTTP requests that 8

20 3.7. CSS prepocessors is needed for a web site while keeping a high maintainability. They may be inserted into a web application s build system, or by using the preprocessor s standalone binaries. 9

21 4 Implementation The project can be divided into three major parts; the web browser, the socket server and simulation control, and the GUI. Presented here are these three components. 4.1 Web browser Initially in the project, the requirements and structure of CEF was investigated. This was to as quick as possible determine wether or not including CEF in OpenSpace and its OpenGL structure would work. It was found that both OpenSpace and CEF uses Cmake as the build system-of-choice. CEF provided Cmake bootstraping helpers, to download the binary library files for the framework. Depending on what platform and architecture that the developer is using, the helper automatically detects this and downloads the needed binaries from CEF s web page. By combining this with OpenSpace s Cmake module system, a web browser module was added to the build process. This downloads, unpacks and includes the needed CEF files. Some compiler flags that CEF s Cmake instructions inserted into the build process were not supported by OpenSpace and created issues at build time. Manually removing these from the downloaded CEF structure resolved that issue. The resulting web browser is implemented in two ways. The first as a so called screen space browser. This is a browser window within the OpenSpace window that can be moved around and placed on the screen where the user wants it. See figure 4.1. Figure 4.1: The screen space browser with controls. The browser window is showing a video from the OpenSpace Youtube page. The second is the screen-covering GUI browser. This is a transparent browser layer that covers the entire OpenSpace window. This is used to render the GUI on screen. See figure 4.2 for an early screen shot in the process of creating the GUI. While the screen space browser 10

22 4.1. Web browser has a dynamic window size, allowing it to be changed as the user wants, the GUI browser s dimensions change as the window is resized. Figure 4.2: Screen shot showing some information about the project, using the GUI browser Event handling and interaction To handle events and interactions from the user, an event handler was created. This is a C++ class that holds a reference to a single active browser window within OpenSpace. This can be either a screen space window or the GUI window. This event handler listens to the events that gets captured by OpenSpace, which include keyboard button clicks, mouse movement, mouse button clicks and mouse scroll wheel movement. These get sent to the active browser windows using corresponding public methods. To determine whether or not a mouse event should be captured, and blocked from triggering other parts of OpenSpace, a transparency layer mask is stored by each browser window. This is based on the assumption that an interaction over a transparent area will not trigger anything on the web page. See figure 4.3a for an example view in the GUI and figure 4.3b for its corresponding layer mask. (a) The GUI as it looks in OpenSpace, overlooking the surface of Mars. (b) The layer mask. The black area will capture mouse clicks and mouse wheel movement. Figure 4.3: The layer interaction mask The layer mask is stored as a one-dimensional C++ array, and is updated every time the GUI is re-rendered. See more about the rendering in section When a user clicks inside the area marked as non-transparent, this will be sent to the GUI, and the event is captured. If the user clicks outside the area, the event will be ignored by the GUI, and the event trickles on within OpenSpace. 11

23 4.1. Web browser To get the index i in the one-dimensional layer mask array from the coordinates x and y, equation 4.1 is used. Here, W is the width of the browser window. This equation is used to decide whether or not the user s interaction is within an active area of the browser or not, as the position is given as coordinates in OpenSpace s event handlers. i = x+y W (4.1) Keyboard button events, typing, is always sent to the GUI and never blocked. The phenomena caused by this, such as keyboard shortcuts getting unintentionally triggered, will later be discussed. Figure 4.4: Overview of the mask update and user interaction process. When a browser window gets repainted, a buffer stored in the web renderer gets updated, see At this time, the layer mask also gets updated. To do this, the alpha channel of the RGBA values are extracted. If the alpha value of a pixel is non-zero, meaning that it is opaque, this is considered a pixel that will capture the mouse events. See the life cycle of the interaction decision process in figure 4.4. The modifier keys, such as control, shift and option, needed to be translated from how the window management tool GLFW interprets them to how CEF interprets them in order to CEF to understand them. This was done using a translation table Rendering Figure 4.5: Overview of the rendering process. The rendering is a two-part process. First, CEF renders the web page to a buffer. This buffer is then copied on the working memory from CEF to the web browser implementation in OpenSpace. The buffer is then copied, each frame, to the GPU in the OpenGL pipeline in order to be drawn on the screen. The copying from CEF to OpenSpace is triggered only when the web page needs to be rendered, and the stored buffer is used until it gets updated and replaced by a new copy. See figure 4.5 for an overview of the steps in the rendering process. This technique is called Off Screen Rendering in CEF terms, meaning that no window is created by CEF itself. Window management happens separately instead. 12

24 4.2. Socket server and simulation control To reduce the computing power needed, CEF only updates the buffer when something on the web page gets changed and a repaint of the web page is needed. This can be a colour changing, text updating, boxes moving or anything similar. Regardless of the magnitude of the change or the cause of this change, the whole buffer gets replaced. This affects both the transparency mask mentioned in section and the buffer that gets copied to the GPU for rendering. Depending on the internal bandwidth of the computer, this copying from the RAM to the GPU might be a time consuming task, affecting the performance of the simulation. The implications of this will be discussed later. 4.2 Socket server and simulation control To make sure that the simulation actually was controllable from the GUI, ways of communicating between the simulation and the GUI needed to be put into place. This was done by implementing a web socket server within OpenSpace. CEF and Chromium has a fully working web socket client built in, which is why web socket was considered a good option for communication. The GUI and socket server send JSON messages back and forth with instructions and information. An example request can be seen in listing { 2 " type " : " example ", 3 " t o p i c " : 1, 4 " payload " : { } 5 } Listing 4.1: Example request sent by the GUI Two different web socket libraries were considered: Libwebsockets and Websocket++. Libwebsockets is, however, single threaded and written in C, not C++ like OpenSpace [36]. Websocket++ is written in C++ and has built-in multi-threaded support. By using the already existing TCP socket abstraction that existed in OpenSpace, web socket support was added. The existing abstraction allowed for easy access to the platform-dependant APIs that adds socket access. There are some element in this JSON body that are needed for each request: type is a string that describes what type of message this is, and should correspond to one of the implemented Topics (more on this later), topic is a required numeric identifier for this message and its possible responses, and payload is an object containing the message details and has further instructions on what to do. A response from the socket server can be seen in listing 4.2. Here, thetype is no longer needed. The client should be aware of the topic, as there are no messages sent on the server s initiative. 1 { 2 " t o p i c " : 1, 3 " payload " : { } 4 } Listing 4.2: Example request sent by the GUI 13

25 4.2. Socket server and simulation control Types of requests and responses The different types of requests are called topics. There are a number of different types, each with a separate purpose. The implemented topics are: authorize, used to authorize a connection, bounce, a ping-like mechanism, that simply bounces back a response, get, get the value of a given property, lua script, run arbitrary Lua scripts in the OpenSpace environment, set, set the value of a given property, subscription, subscribe to the value of a given property, time, various special-case time related operations, and trigger, trigger a trigger property. The properties mentioned in the list above are OpenSpace properties that may be attached to an object. These objects can for instance be groups of settings, a planet in the scene or the stars in the background. Each property has a URI-like key parameter, that may describe which object it belongs to and what this specific property is named. There are also different types of properties; string, numeric, boolean, trigger, vector and matrix. An overview of the property sending mechanism can be seen in figure 4.6. Figure 4.6: Overview of the property sending mechanism. The payloads of each topic varies. Every topic have different purposes, but some of them have similarities. To use the topics get, set, subscribe or trigger, the payload always contain the key-value-pairproperty. This should contain the URI-like key of the wanted property. See listing 4.3 for a get topic request. In this example, get could be changed to any of subscribe or trigger and still be a valid request. The purpose of that request would simply be different. 1 { 2 " type " : " get ", 3 " t o p i c " : 1, 4 " payload " : { 5 " property " : " example " 6 } 7 } Listing 4.3: Example get request sent by the GUI On a get request, the client asks for the value of the named property. The server receives the request, and if the named property can be found, a message is sent back to the client with the property in a JSON formatted structure. The serialization will be further explained later. On a subscription request, the client asks to be notified each time the value of the given property has changed. If the server finds the given property, a callback that gets called each 14

26 4.2. Socket server and simulation control time the property changes it s value gets inserted. This callback sends the message to the client, just like the get requests. A copy of the current value also gets sent immediately. The trigger requests work differently. Here, no message gets sent back to the client, as the trigger property itself has no value. It can be thought of as a button. Pressing the button has no value itself - it is the underlying mechanisms that cause something to happen. This, in turn, could potentially be subscribed to or retrieved. To use the set property, the client also needs to provide a new value to the property. See listing 4.4 for an example set request, where a three dimensional vector property gets set to[1, 2, 3]. 1 { 2 " type " : " s e t ", 3 " t o p i c " : 1, 4 " payload " : { 5 " property " : " example ", 6 " value " : [ 1, 2, 3] 7 } 8 } Listing 4.4: Example get request sent by the GUI To verify that the client actually should get access to the web socket server, the authorize topic exists. This can be used to prove the client s validity by providing a password. Whether or not this is required can be set in the OpenSpace configuration file, along with the password and an optional list of whitelisted IP addresses. The clients connecting from an IP address in the whitelist will not need authorization. If the client is not from one of the white listed addresses, however, all the topic requests but authorize are disregarded until the client is considered authorized. See listing 4.5 for an example request. In listing 4.6, a message telling the client it is authorized is shown. Listing 4.7 shows the message sent when the client could not be authorized. The code entry in the responses mimics HTTP status codes [18, ch. 10]. 1 { 2 " type " : " authorize ", 3 " t o p i c " : 1, 4 " payload " : { 5 " key " : " example password " 6 } 7 } Listing 4.5: Example authorization request sent by the GUI 1 { 2 " message " : " Authorization OK. ", 3 " code " : } Listing 4.6: Example authorization granted response sent by the server 1 { 2 " message " : " I n v a l i d key. ", 3 " code " : } Listing 4.7: Example authorization denial response sent by the server The bounce topic provides a ping-like mechanism. Immediately when the server receives a bounce request, it sends back a bounce response. Thepayload sent may contain anything and is returned to the client. 15

27 4.2. Socket server and simulation control The lua script topic allows the client to send an arbitrary lua script that will be executed by OpenSpace. This can for instance be shutting down the application or similar. There is no responses being sent back. An example request can be seen in listing { 2 " type " : " l u a s c r i p t ", 3 " t o p i c " : 1, 4 " payload " : { 5 " s c r i p t " : " openspace. toggleshutdown ( ) " 6 } 7 } Listing 4.8: Example Lua script request sent by the GUI, toggling the shut down process in OpenSpace. The time topic handles everything related to the simulation time. This covers both the speed of the simulation and the actual simulation date and time. The client can subscribe to these items, and change them using the set topic. Neither the simulation speed or the date are stored as properties, so to subscribe to them a custom method is used. By inserting a post render callback into the OpenSpace engine, the server can see if a value has been changed since it was last sent. The structure of this request looks much like the get request, as seen in listing 4.3. However, the property entry has the value currenttime or deltatime, depending on what callback you want. An example message sent from the server to the client can be seen in listing { 2 " t o p i c " : 1, 3 " payload " : " T10 : 5 7 : " 4 } Listing 4.9: Example response for the time topic, sending the current simulation time. The simulation time is expected to change at a high rate. To limit the burden of sending messages, the rate is throttled. This is done on a per-topic basis, requiring a given time to pass between each message JSON serialization As mentioned before, messages sent back and forth between the server and client are JSON formatted. On the server, this is done by using a JSON library that allows arbitrary C++ types to be serialized to JSON [37]. Using this method, nested data structures may be serialized as the library uses recursion. A serialized OpenSpace property contains information that describes how it should be handled by the GUI. What type of data it contains, limits, name, description and its value are described. See listing 4.10 for a completely serialized property. 1 { 2 " Value " : 123, 3 " Description " : { 4 " d e s c r i p t i o n " : " Example description, explaining t h i s property to the user ", 5 " Type " : " The type of t h i s property ", 6 " I d e n t i f i e r " : " The URI to t h i s property ", 7 "Name " : " The name to show in the GUI ", 8 " MetaData " : { 9 " Group " : " ", 16

28 4.3. GUI 10 " V i s i b i l i t y " : " All ", // one of All, Developer, User and Hidden 11 " ReadOnly " : f a l s e 12 }, 13 " AdditionalData " : { } 14 } 15 } Listing 4.10: Example of a serialized property Message throttling In order to ease the load on the socket server, a message throttling constraint was implemented. This limited every topic from sending more than a given amount of requests per second. This was done by storing the timestamp of latest sent message and requiring a certain wait period between the last message and the next one. 4.3 GUI The created GUI is a web page written using React. It has been created in such a way that different versions of the GUI may be created. This could for instance be the GUI that exist within OpenSpace, or a GUI that an be accessed from another unit. The idea behind this is to allow OpenSpace to be controlled remotely, for instance during a presentation or similar. However, only the on screen version of the GUI has been created as of this writing. Mainly two different javascript frameworks for GUI development was considered. React, as mentioned, and Angular. Due to previous experience, the selection leaned in favour of React. While continuing creating design mockups and gathering feedback, the GUI was simultaneously developed Design Initially in the GUI process, a design was created in the design tool Figma. The design was shared with the rest of the OpenSpace team, so that early thoughts and ideas could be collected. The design process was not the focus of this project. However, due to the nature of design in web and GUI development, the design played a big role in the end result. As mentioned in section 1.5, this will not be thoroughly presented nor discussed. In figures 4.7a and 4.7b, design mockups for numeric inputs can be seen. These images were shared with the OpenSpace team to gather feedback and see which alternative that was the most popular one. (a) Different variations of numeric input. (b) Even more variations of numeric input ideas Figure 4.7: Design mockups with different alternatives of numeric inputs. 17

29 4.3. GUI Data management To handle the connection from the GUI to the server, a data management layer was created. This is capable of handling every task supported by the web socket server. This is a singleton class, having a web socket connection open to the server. Instead of manually creating calls to the socket and creating the needed payload and message structure, this data manager can be used by the developer. This proved to be useful during the development process, when the structure of the requests changed and got improved. This data manager has a number of methods corresponding to the topics on the server. For instance, instead of creating an object such as the one seen in listing 4.3 when a property was needed from the server, the developer can use DataManager.getValue(property, callback). An example of this can be seen in listing Here, the property example.property is fetched and the callback defined in the variable callback handles the data when it arrives from the server. As can be seen, no JSON payload is created. Neither is any topic identifier nor topic type needed. The data manager keeps track of the topics, its incoming messages and the callbacks. 1 // c a l l b a c k f o r the get request 2 const c a l l b a c k = ( value ) => { 3 // do something 4 console. log ( The value received from the server i s :, value ) ; 5 } 6 // get the property 7 DataManager. getvalue ( example. property, c a l l b a c k ) ; Listing 4.11: Example get request using the DataManager. Subscribing works in similar ways. Given a property name and a callback, the data manager makes sure the subscription is created and receives its needed data, given no server error is generated. The data manager also manages several subscriptions to the same property, and reuses the incoming update message instead of creating a new one. This decreases the burden on the socket connection. Unsubscribing works in a similar way. Provided the same property name and callback, the data manager removes this callback from that property. 1 // c a l l b a c k f o r the s u b s c r i p t i o n 2 const c a l l b a c k = ( value ) => { 3 // do something 4 console. log ( The value received from the server i s :, value ) ; 5 } 6 // c r e a t e s u b s c r i p t i o n 7 DataManager. subscribe ( example. property, c a l l b a c k ) ; 8 // unsubscribe a f t e r 3 seconds 9 window. settimeout ( 10 ( ) => DataManager. unsubscribe ( example. property, c a l l b a c k ), ) ; Listing 4.12: Example subscription creation and removal using thedatamanager. In the case of loss of connection between the GUI and the server, the data manager has mechanisms to recreate the subscriptions. This is done using the callbacks that the WebSocket API provide and by storing the subscriptions that needs to be recreated [38]. Setting values are done in a similar way, too. See listing 4.13 for an example. Here, the methoddatamanager.setvalue is used, and the callback is replaced by the new value. 1 // s e t the property value 2 DataManager. setvalue ( example. property, newvalue ) ; Listing 4.13: Example set request using the DataManager. 18

30 4.3. GUI Finally, the data manager supports trigger topics and Lua script topics. These can be accessed using the methods DataManager.trigger(propertyName) and DataManager.runScript(theScript) On screen GUI The GUI that is rendered as the fore-most layer in the OpenSpace window is called the On screen GUI. This is created with the idea of being as non-intrusive as possible, so that the user can focus on the simulation. In figure 4.8 the GUI as it looks after application start can be seen. Figure 4.8: The On screen GUI, immediately after application start. Overlooking Mars. The on screen GUI uses two major components to divide its content. The first is the sidebar to the left, see figure 4.9. Here are the most simulation properties and settings of the application placed. The second is the popover windows on the bottom of the screen. See figure 4.10a for the focus popover and figure 4.10b for the time and date control popover. The purpose of the popover windows are to provide powerful access to some more complex phenomena such as time, or common-use features such as focus selection. More tools can be added to the bar at the bottom Reusable components Throughout the GUI, components are being reused for different use cases some more general-purpose than others. To pick a few, there is the calendar, the input fields, the sidebar panes and the popover windows. These reusable components have been developed to be as close to native HTML elements as possible in how they are used by the developer. Some components also come with logic for sending and receiving properties attached to it. To make development as easy as possible, some of the components were developed in a tool called Storybook. This is a development environment that can be used to isolate the creation of React components. It allows the developer to create a collection of components and presents these components in an interface. This helps in creating stand-alone components with a API suitable for usage in varying context Calendar A calendar was created to give a more intuitive way of selecting the current active date in the simulation. It highlights the current date as well as the currently selected date. The user can go back and forth between months and also change year by typing. See figure 4.11 for a 19

31 4.3. GUI Figure 4.9: The On screen GUI with sidebar open. A loading animation is shown where the scene graph properties will be loaded. (a) The popover showing focus selection. (b) The popover showing the calendar and time settings. Figure 4.10: Popovers on the bottom of the screen screen shot of the calendar. Here, the day highlighted by the blue line is the selected day. The other highlighted day is the current date. The faint days are days of another month. To avoid flickering experiences and prevent the calendar from resizing, the calendar always shows six weeks Numeric input fields To allow the user to quickly change a numeric value, the numeric input field was developed with care. The idea was to allow the user to change the value both in a precise way by typing and in a fast way by dragging with the cursor. This was done by switching between a HTML5 range based input and a number based input [39]. The input field could be configured in different ways, either typing only, range only or both. When both is the case, switching from range to typing is done by double clicking the field or single clicking the value of the field. Switching from typing to range is done when the focus on the input field is lost. See figure 4.12a, the range input can be seen. In figure 4.12b, the same input is being edited by the user. In figure 4.12c, the same input can be seen in type-mode. 20

32 4.3. GUI Figure 4.11: The calendar component. (a) Range input. (b) Editing a numeric value. (c) Typing numeric input. Figure 4.12: Different states of a numeric input. To provide the user with hints about what the purpose of the input field is and it s current value proportionally to its minimum and maximum value, there is a progress-bar like look on the range input. The size of this gets filled depending on the value of the field and it s limits. All the numeric values have a maximum or a minimum value. There is also, as can be seen in figure 4.12b, a similar bar that follows the user s movement. This is paired with a small tooltip showing the estimate of the value that will be set if the user changes the value Other types of input fields There are not only numeric input fields in the GUI. Text, alternative selection, boolean and multi-component values like vectors and matrices, all needs to be editable. This is supported, too. Using these inputs allows the developer to create a consistent experience throughout the application. The boolean input values result in a check box, and the alternative selection in a drop down (see figure Text values use an input field similar to the one used for numeric values, but with the typetext instead. Figure 4.13: Alternative selection view with an expanded drop down menu. 21

33 4.3. GUI Matrices and vectors use the numeric input fields, put side by side or in a grid. This way, components gets reused without duplicated logic. Using a custom button component adds a consistent look and feel to the buttons that exist throughout the application. With a certain amount of configurability, these inputs and buttons can be used throughout the application and provide familiarity for the user and saves time for the developer Popovers To show information or similar elements on an on request-basis, popovers were implemented. These are windows that can be toggled to be shown by the user. It can be used to reveal details or explain things, or to hide complex tools, like in the case of the simulation speed and time case. This, too, is used on several occasions. The focus selection, simulation speed and time control and the system menu, as seen in figure 4.14, use the popover with varying configurations. Figure 4.14: The system menu, providing the user with some extra tools and links to information Tooltips Similar to popovers, tooltips where implemented to be provide the developer with a method of hiding information until the user needed it. This works in similar ways to the popovers, but are configured to be used with less information in an non-interactive way. One example is the estimated new value seen in figure 4.12b Filter lists In order to allow the user to quickly search through an arbitrary long list of items, a component called filter list was created. Given a list of items and a component to use to display these items, the filter list adds a search input to the list. As the user is typing their search, the content of the list gets filtered. This allows the user to quickly find what it is looking for, even as the size of the list increases. This is used in several locations. Both side panel views uses it, as well as the focus selector. See figure 4.15 for an example. The filter list is configurable in the way that the function used to match the list object with the search string is changeable. By default, it uses a simple sub-string match. For every value stored in the list, can a match on the query string be found?f If yes, it is a match and this object gets included in the list. If not, then this object is not shown in the search result. This substring matching function is currently the only implemented matching function. However, to 22

34 4.3. GUI Figure 4.15: The filter list being used. implement more matches, all that is needed is a function that accepts an object and a search string, and returns a boolean value true if it is a match,false if is not Loading indicators While the main target of the GUI discussed here is running locally, using web technologies can lead to situations where loading times are to be expected. It is simply not something that can be completely disregarded. Therefore, loading indicators were created for the GUI. These consist of boxes with an animated gradient background. See figure 4.16a and 4.16b for two examples. Here, the loading indicators are used in two different cases. First, we see an example where the sidebar has been opened, and placeholder loading indicators are in place, waiting for the content to be loaded. Second, we have the focus selector and the current simulation date waiting to be loaded. (a) Loading content in sidebar. (b) Loading content at bottom of screen. Figure 4.16: Loading indicators. In the first example above, from the sidebar, the loading indicators are of random count and with random size. This is to mimic the diverse values that might be shown once the data is loaded. In the second example, the loading indicator replaces placeholder text Properties To let the user achieve total control of the simulation, there are support for the majority of OpenSpace property types. In table 4.1, a list of all the implemented simulation properties and its corresponding GUI component and input type are listed. These were all the property types required by the OpenSpace simulation today. The vast majority of the properties are in the sidebar. Here, the GUI asks the socket server for a list of all the properties that control the simulation. This is done either by asking for the scene graph objects or by the system properties. The scene graph objects and its properties gets returned and rendered to the GUI. In figures 4.17a, 4.17b and 4.17c, different variations of the properties are shown. In 4.17b, a tooltip with information provided by the server can also be seen. 23

35 4.3. GUI Simulation Property type GUI Property type Input type BoolProperty BoolProperty Checkbox OptionProperty OptionProperty Select TriggerProperty TriggerProperty Button StringProperty Property Text input NumericProperty NumericProperty Numeric input FloatProperty NumericProperty Numeric input IntProperty NumericProperty Numeric input Vec2Property VecProperty Numeric vector input Vec3Property VecProperty Numeric vector input Vec4Property VecProperty Numeric vector input MatrixProperty MatrixProperty Numeric matrix input DMat4Property MatrixProperty Numeric matrix input default Property Text input Table 4.1: Implemented property types in the GUI, and which corresponding input mechanisms that is used. (a) Collection of properties controlling the look of the on-screen console. (b) Collection of properties with active information tooltip. (c) Nested collection of properties, where the property collection renderable for SaturnTrail is shown. Figure 4.17: Different variations of properties Windows To be able to show information in a less disruptive way than using a fixed-position popover, a movable window component has been implemented. This supports any content and allows the window to me moved around freely. This makes it possible for the user to adjust the GUI to fit their specific needs. See figure 4.18 for an example, where the date selection popover has been detached. Figure 4.18: Top of a movable window. 24

36 4.3. GUI Scroll hiding Given that the screen space available to the GUI is limited, showing a big list in a limited area is useful. In this situation, scrollable areas provide a well-known way of hiding and showing information. This is not, however, completely without issues. See figure 4.19 for an illustration of an issue where a list of elements are shown, but align in such way that the faint blocks outside the current view are completely invisible. Figure 4.19: Scroll issue shown. How can the user be informed that there are elements outside the visible area? To solve this, a wrapping component has been created. This wraps any content in a limited height, and allows the developer to give a hint to the user that there is more content hidden. This is done by letting the content of the scrolling box to become more and more transparent the closer it gets to the bottom or the top. This can be seen in figure Figure 4.20: Content of a scrollable list fading out. Another option for this component is to use visible scroll bars. Instead of the fading effect, the idea here is to allow the user s experience of scroll bars to make it clear that there are more objects outside the current view. These scroll bars use a brighter color than the regular scroll bars in the GUI. See figure 4.21, where the scroll bar is visible to the right. Figure 4.21: Visible scroll bar to the right Collapsible view To support nested objects and hiding and showing details, an collapsible view component was created. Given a title and an arbitrary list of elements, this component can be used to build a tree view, as can be seen in figure

37 4.3. GUI Figure 4.22: Tree view using the collapsible component Warnings and error messages Given the complexity of OpenSpace s simulations, there are occasions where errors occur. Explaining this to the user might be important, especially when it is a fatal error. The error management that is implemented in the GUI handles the loss of connection between the simulation and the GUI. See figure As this completely removes the GUI s capabilities to control the simulation, this error covers the entire screen. Immediately when the connection is lost, the GUI tries to reconnect. If this is without success, the next reconnection attempt is delayed. The delay increases for every attempt, following the equation 2 n, where n is the number of the current attempt starting at 0. The first attempt is delayed one second, the second two seconds, the third attempt four seconds, the fourth eight seconds and so on. Figure 4.23: Error message when the GUI cannot reach the simulation. 26

38 5 Results The implemented web browser handles mouse and keyboard input, and supports rendering web pages to the screen. By using Chromium s built-in performance measurement tool that allows the user to see the achieved frames per second (FPS), an estimate of the browser s performance can be found. In figure 5.1 the FPS meter is seen. On a page with continuous animations, a stable frame rate of 50 to 60 FPS was achieved for the web rendering separately. This was on a machine with 32 GB of RAM, a GeForce GTX 1070 graphics card and a fourcore Intel i7-7700k processor. In some cases with a lot of user interactions (such as clicks and dragging), together with other computationally intense events, observations where the frame rate dropped to 30 FPS was found. It was, however, difficult to find a recurring pattern to this, as there are a lot of factors affecting the FPS. Figure 5.1: FPS meter in the top right corner. The implemented GUI gives control of the simulation and allows the user to change every property of the simulation. The resulting GUI can be seen in figure 5.2. The implemented message throttling reduced the load on both the server and the GUI, but no suitable measurement to show this was found. A noticeable stuttering followed by a short freeze was, however, no longer apparent. Whether this behaviour was caused by the client (GUI) handling the incoming messages or the server producing these messages is unclear. 27

39 Figure 5.2: Final implemented GUI. 28