Project Number 611411 D4.5 OS Allocation and Migration Support Final Release Version 1.01 Final Public Distribution University of York, aicas, Bosch and University of Stuttgart Project Partners: aicas, Bosch, CNRS, Rheon Media, The Open Group, University of Stuttgart, University of York Every effort has been made to ensure that all statements and information contained herein are accurate, however the DreamCloud Project Partners accept no liability for any error or omission in the same. 2016 Copyright in this document remains vested in the DreamCloud Project Partners.
Project Partner Contact Information aicas Bosch Fridtjof Siebert Devendra Rai Haid-und-Neue Strasse 18 Robert-Bosch-Strasse 2 76131 Karlsruhe 71701 Schwieberdingen Germany Germany Tel: +49 721 66396823 Tel: +49 711 811 24517 E-mail: siebert@aicas.com E-mail: devendra.rai@de.bosch.com CNRS Rheon Media Gilles Sassatelli Raj Patel Rue Ada 161 20 Leighton Avenue 34392 Montpellier Pinner Middlesex HA5 3BW France United Kingdom Tel: +33 4 674 18690 Tel: +44 7547 162920 E-mail: sassatelli@lirmm.fr E-mail: raj@rheonmedia.com The Open Group University of Stuttgart Scott Hansen Bastian Koller Avenue du Parc de Woluwe 56 Nobelstrasse 19 1160 Brussels 70569 Stuttgart Belgium Germany Tel: +32 2 675 1136 Tel: +49 711 68565891 E-mail: s.hansen@opengroup.org E-mail: koller@hlrs.de University of York Leandro Indrusiak Deramore Lane York YO10 5GH United Kingdom Tel: +44 1904 325 570 E-mail: leandro.indrusiak@cs.york.ac.uk Page ii Version 1.01
Contents List of Figures iv 1 Executive Summary 2 2 Overview of Prototype Release 3 3 OS Support for Migration 5 3.1 Migration of Bundles Between Nodes......................... 6 3.2 Test of Migrating Bundles Between Nodes....................... 7 4 Conclusion 10 5 References 11 Version 1.01 Page iii
List of Figures 1 Conceptual DreamCloud migration architecture..................... 3 2 Xilinx Zynq SoC block diagram [5]........................... 3 3 Conceptual architecture with Linux OS running on each node............. 5 4 Conceptual architecture with Linux OS and JVM running on each node........ 5 5 Applications (Ts) running on different nodes within JVM................ 6 6 Migration process to migrate bundles between nodes.................. 6 7 Counter bundle is running when a migration request is received............ 8 8 Counter bundle starts counting from last state of the freeze bundle........... 8 9 Management webpage of the Openfire server showing established connections.... 9 10 Spark XMPP client. The left hand side shows sending chat message text window, and the right hand side shows the connected users available to chat........... 9 Page iv Version 1.01
Document Control Version Status Date 0.1 Initial Draft 4 March 2016 0.2 Version after addressing 7 March 2016 project partners reviews 1.0 Final version 9 March 2016 1.01 Added link to the demonstration video Version 1.01 Page 1
1 Executive Summary This document constitutes deliverable D4.5 OS Allocation and Migration Support Final Release of work package 4 of the DreamCloud project. To fulfill the requirement of the deliverable, allocation and migration support including software and documentation is presented along with a brief review of the prototype deliverable. The prototype architecture used field-programmable gate arrays (FPGAs) as they support easy and rapid prototyping. A brief review of the prototype deliverable is provided in next section. This document also provides an elaboration of the achievements for the final release. Page 2 Version 1.01
2 Overview of Prototype Release D4.5 OS Allocation and Migration Support Final Release The DreamCloud conceptual migration architecture proposed in the prototype release and adopted in the final release is shown in Figure 1. The architecture contains a set of 4 nodes connected via Ethernet/Fibre Channel, where each node is a Xilinx Zynq-7000 System-on-Chip (SoC) board [1]. The board has a FPGA which contains the processor system and reconfigurable region. The board also contains other peripherals such as memories and inputs/outputs. Data streams Zynq-7000 SoC Zynq-7000 SoC Zynq-7000 SoC Zynq-7000 SoC Processor System Processor System Processor System Processor System Interconnect Interconnect Interconnect Interconnect IP IP IP IP Ethernet/Fibre Channel Figure 1: Conceptual DreamCloud migration architecture. Figure 2 shows the structure of the Zynq FPGA (figure with more details can be found in deliverable D4.3). The upper part of the figure shows the processing system which is a dual core 32-bits ARM Cortex-A9 processor. The bottom part shows the programmable logic, which can be configured with a set of pre-compiled accelerators (s). The accelerators are connected by an interconnection network that could be a bus or Network-on-Chip (NoC) depending upon the number of s to support [4]. The accelerators and the CPU cores communicate through the AMBA (ARM Advanced Microcontroller Bus Architecture) interconnection, which is an open standard, on-chip intereconnect specification for the connection and management of functional blocks in system-on-chip (SoC) designs. Figure 2: Xilinx Zynq SoC block diagram [5]. Version 1.01 Page 3
In the industrial use-case to be evaluated in deliverable D6.4, this conceptual architecture is expected to process the streams received by a satellite feed or stored by prior streaming. The main reason behind choosing this architecture is that it can be used where data streams are received continuously and sent for processing to various nodes at run-time. Further, it provides the opportunity to balance the load over the nodes and accelerate some heavy computations by using reconfigurable hardware. The DreamCloud project aims to address the load balancing and acceleration requirements, which could arise to process a heavy computation request or multiple requests for the same video content. The industrial use-case will exploit this architecture mainly to process the Micro Cloud (video processing) workload (from the transcoding applications) to be provided by the industrial partner Rheon Media. The nodes in the architecture could be located in different flats/houses and could act as DreamCloud media servers 1. For various users requesting different kinds of streams to different devices (e.g., laptop, phone, etc.) the processing can be distributed to various nodes such that the loads of the nodes are balanced. Such distribution for balanced load processing is to be achieved by the resource allocation approaches developed in work package 2 (D2.1, D2.2 and D2.3). In case of performance bottleneck, identified compute intensive components can be offloaded to the reconfigurable area. These components can be compiled in advance in the form of s so that they can be used to configure the FPGAs whenever acceleration is required. In the prototype release, the OS allocation and migration infrastructure was developed for one node of the architecture shown in Figure 1. The deliverable D4.3 provides the details of the node, where Linux OS runs on the dual-core ARM Cortex-A9 processor of the node in order to facilitate efficient allocation and migration. A simple example application specified in high level language was run on a real-time Java Virtual Machine (VM) instantiated under the Linux running on the ARM processor. The OS support for various kinds of migrations was introduced in D4.3. Additionally, the acceleration approach to be followed to offload the compute intensive components on the reconfigurable area was introduced. 1 There maybe one or more servers depending on the reachability and latency between sets of nodes Page 4 Version 1.01
3 OS Support for Migration D4.5 OS Allocation and Migration Support Final Release For our conceptual migration architecture, OS support is developed for each node, where the OS can reserve an appropriate amount of resources, bandwidth and prioritize processes to access the resources (e.g., processors and networks). We choose Linux OS as it can cope with the diversity in the hardware architecture in foreseeable future. Additionally, it offers several advantages such as ability to modify it for our needs as it is open source, its availability for most of the hardware platforms and the existence of a big list of libraries and tools that could be used for the development. Linux on nodes (done) Figure 3 provides an overview of the conceptual architecture, where Linux OS is running on each node. We chose PetaLinux as it is a full Linux distribution which includes the Linux OS as well as a complete configuration, Linux working build and deploy on environment different fornodes, Xilinx silicon, i.e. provided each by Xilinx [3]. Further, it is fully node featured, acts integrated, as atested, PC and with documented, its IPandaddress, thus recommended to be used. By running Linux on each node, the nodes act as PCs with their IP addresses, enabling communication enabling communication between nodes between nodes via simple Linux commands for communication. Linux Linux Linux Linux JVM on each node of microcloud architecture Figure 3: Conceptual (ongoing) architecture with Linux OS running on each node. Communication between the 4 FPGA boards over the fibre as well Trying now, hopefully will be done soon As mentioned earlier in the overview of the prototype release, the application/task is executed on a real-time Java Virtual Machine (JVM) instantiated under the Linux running on the ARM processor. Therefore, JVM is installed on each node as shown in Figure 4. This JVM is provided by the Project Review 6 industrial partner Aicas. JVM JVM JVM JVM Linux Linux Linux Linux Figure 4: Conceptual architecture with Linux OS and JVM running on each node. The applications/tasks are to be executed on the JVM and these applications are modeled as Open Services Gateway Initiative (OSGi) bundles. The OSGi defines an infrastructure for developing and deploying Project modular Review applications and libraries. This infrastructure 8 allows us to break a complex application into multiple modules that can be managed more easily. These modules are also referred to as Version 1.01 Page 5
OSGi bundles, as mentioned earlier. They hide their internals from other bundles and communicate through well defined services. This provides more freedom for changing at later stages, reduces the number of bugs and makes the development of bundles simpler, which implements a piece of computation (functionality) through well defined interfaces. The OS s offers several other advantages as mentioned in D4.3. Figure 5 shows some example applications modeled as OSGi bundles (T) and installed and running on JVMs on each node within an OSGi container. The OSGi container can contain and execute multiple bundles as shown in the figure. Figure 5: Applications (Ts) running on different nodes within JVM. 3.1 Migration of Bundles Between Nodes To support migration of bundles between nodes, freeze and resume techniques are employed, where the bundles are frozen, migrated to a destination JVM and resumed on the destination. Figure 6 shows the process of migrating a bundle (highlighted with distinct color) from one node to another. For the bundle, once the migration request is received or a need of migration arises due to load balancing requirement, the bundle is freezed on the currently running node, migrated to the destination node, and resumed on the destination node to start the processing from the current state. Such required techniques are developed by the industrial project partner and detailed into deliverable D4.4. During the execution of a bundle at run-time, monitoring of the bundle and system infrastructure takes place in order to identify the need for migration of the bundle to some other node, which is possibly less loaded. Figure 6: Migration process to migrate bundles between nodes. Page 6 Version 1.01
In order to keep the migration overhead low, the bundles that might need migrations are installed on each node so that only the state of a bundle needs to be transferred. Then, the bundle on the destination node can be started with the current received state. This facilitates to achieve executions without long delays that might incur to send the whole program code of the bundles from one node to another. In order to explain the migration, we can consider as an example the migration of a simple counter application. In this example, the migration is started when the board running a bundle receives the request to migrate the running bundle to another board. That request is a text message with the structure migrate <bundle-id> 2 <destination-board>. It is sent and received through an XMPP (Extensible Messaging and Presence Protocol) server. The board freezes the bundle and sends the current state and a bundle activation request as an XMPP message to the destination board. In this example, the migration order is given to the XMPP shell extension, which parses the command and then uses the migration API to trigger the migration. In Deliverable 6.4 (validation in video domain), the heuristics will be communicating with the JavaDispatcher service to trigger migration. The JavaDispatcher will in its turn use the migration API to execute the orders coming from the heuristics. The decision to use XMPP lies in the fact that it is a well known established standard, and it is a realtime communication protocol which includes chat. It has been known as Jabber; however, as Jabber was not the only software relying on XMPP, it has been renamed to XMPP. We use the Spark as XMPP client and Openfire 3 as XMPP server, both installed in one of the boards. We use these client and server software when testing on the boards, because our partner aicas has been using them when developing the bundles due to aforementioned benefits. Notice that we are running, during the test of migrating bundles, the Openfire server in one of the boards, and Spark-XMPP- Client for sending the migration request message, but in the industrial use-case the XMPP server can run in a remote server, and there is not need for any XMPP-Client since the OSGi containers will establish their own XMPP connections. The detailed exploitation of these migration mechanisms and tools for industrial partner use-case application will take place in work package 6. In the next section, we show the exploitation of the migration mechanisms for a simple counter application. 3.2 Test of Migrating Bundles Between Nodes The migration process described earlier has been tested by a counter application that is migrated from one node to another after sending manually a migration request [2]. Figures 7 and 8 shows such process, where the counter application executing on one node was counting when the migration request is received. In Figure 8, a different counting bundle resumes the counting from the last value counted before the migration. As mentioned earlier, the counter bundle is installed on each node so that migration overhead is minimized. 2 Each bundle gets an unique identification number, this value is used as <bundle-id>. 3 Openfire is a real time collaboration (RTC) server licensed under the Open Source Apache License. It uses the XMPP protocol and can be managed via a web interface. It is easy to setup and configure, but has a high level of security and performance. It runs using Java. It should be noted that with Openfire, no chat is possible yet. A client is needed: Openfire cannot be used alone, just like web servers need a browser. Version 1.01 Page 7
Figure 7: Counter bundle is running when a migration request is received. Figure 8: Counter bundle starts counting from last state of the freeze bundle. In order to perform the migration example just described, it was required to start a Openfire server. In this test, we run the Openfire in one of the boards. Figure 9 shows the configuration webpage of Openfire server, on which we can see 3 users connected. The users corresponds to the XMPP bundles connected to the server, whose names come from the last 4 characters of the MAC address 4 of the board where they are running. It was decided to define names in such way to avoid confusions or future errors. The 3 XMPP bundles connected appear as 0194, 0197, and C595. 4 In order for a network device to be able to communicate, the MAC Address it is using must be unique. No other device on that local network subnet can use that MAC address. Page 8 Version 1.01
Figure 9: Management webpage of the Openfire server showing established connections. The migration message is generated manually in this test. It is sent from a Spark XMPP client which allows to send chat text messages. We were logged in the Openfire with Spark using the account admin. The figure 10 shows on the right the list of possible chat destinations when logged as admin. On the left, there is the chat window showing the last message sent to user 0194, the content of the message was migrate 20 0197@localhost/Smack. The message request that XMPP bundle logged as 0194 migrates the bundle number 20 running on same board, to the board where the XMPP bundle logged as 0197 is running. Figure 10: Spark XMPP client. The left hand side shows sending chat message text window, and the right hand side shows the connected users available to chat. Version 1.01 Page 9
4 Conclusion This document has introduced the OS allocation and migration support in terms of tools and documentation. The OS support has been tested for an example counter application in order to successfully migrate the counter bundle from one node to another and for offloading a simple function to FPGA. The followed migration techniques are introduced in D4.2 and D4.4. The technique adopted to offload a function to FPGA is introduced in D4.3. In the industrial use-case study (D6.4), multiple applications will need to be allocated on the nodes. To accomplish the same, we plan to exploit the resource allocation approached reported in work package 2 (D2.1, D2.2 and D2.3). This will be realized after receiving the transcoder and other required bundles from the industrial partner. The transcoder bundles will be installed on the nodes in order to observe the capacity of the nodes, and the resource allocation will be performed such that load on the nodes is balanced. The load balanced allocation is expected to lead to high performance. To support multiple applications on a node, the hierarchical scheduling approach introduced in D4.3 will be exploited. Page 10 Version 1.01
5 References [1] Zynq-7000 All Programmable SoC, 2014. http://www.xilinx.com/products/silicondevices/soc/zynq-7000.html. [2] Migration of bundles in the JamaicaVM framework, 2016. https://www.youtube.com/watch?v=oinzc6zvhts. [3] Petalinux full linux distribution, 2016. http://www.wiki.xilinx.com/linux (Last visited: 3 March, 2016). [4] Luca Benini and Giovanni De Micheli. Networks on chips: a new SoC paradigm. Computer, (1):70 78, 2002. [5] Xilinx Kees Vissers. Programming novel recognition algorithms on heterogeneous architectures. In Embedded vision submit, 2014. Version 1.01 Page 11