Heterogeneous Software Architecture with OpenAMP

Transcription

1 Heterogeneous Software Architecture with OpenAMP Shaun Purvis, Xilinx

2 Agenda Heterogeneous SoCs Linux and OpenAMP OpenAMP for HSA

3 Heterogeneous SoCs A System-on-Chip that integrates multiple processor architectures Also called an MPSoC (multi-processor SoC) Example combinations include Cortex-A MPCore + Cortex-M (e.g., A7+M4) Cortex-A MPCore + Cortex-R (e.g., A53+R5) Probably integrates other processing units (PUs) as well e.g. GPU, DSP, FPU Vendor typically provides an interface for these more specialized PUs Developers must decide on a heterogeneous software architecture (HSA) for generic PUs

4 Example: big.little SoCs big.little combines two Cortex-A processor units Maybe other PUs as well (e.g. GPU) Some SoCs using a big.little configuration Samsung Exynos Octa Apple A10 Fusion Renesas Mobile MP6530 MediaTek MT6595 Qualcomm Snapdragon 808/810/820/821 Nvidia Tegra X1 Image Source: (ARM Ltd.)

5 But Not The Best Example We want to discuss heterogeneous software architecture (HSA) Different software platforms on different processors Asymmetric Multi-Processing (AMP) as opposed to Symmetric Multi- Processing (SMP) Big.LITTLE similar to Multi-core SoCs in terms of software architecture Linux runs SMP across both Cortex-A PUs Libraries/APIs used to leverage other PUs (e.g. GPU) So there s really only one software platform: Linux Predominant in mobile market But not very accessible to other embedded markets

6 Better Example: Xilinx Zynq UltraScale+ MPSoC Image Source: (Xilinx Inc.)

7 HSA Implied by ZynqUS+ MPSoC Not possible to run Linux across Cortex-A and Cortex-R AMP implied by differing PUs: APU and RPU GPU still abstracted through Libraries/API APU a good candidate for Linux RPU a good candidate for an RTOS But how is this so different from AMP on a multi-core CPU?

8 AMP on Multi-core CPU All cores are within the same processor architecture Architecture specifies a well defined interface between cores Can abstract SGIs into an AMP framework and reuse on other SoCs with a similar APU Core1 Core3 APU SGI Core2 B-M Core4 RTOS SoC D/C GPU

9 AMP on Heterogeneous SoC Interface between APU and RPU will be device-specific Abstraction becomes more complicated Openly documented framework that different vendors could leverage to abstract device-specific interfaces would be ideal Core1 Core3 APU SGI Core2 B-M Core4 RTOS SoC? Core1 B-M RPU SGI Core2 RTOS

10 Agenda Heterogeneous SoCs Linux and OpenAMP OpenAMP for HSA

11 Background: Linux AMP Heterogeneous SoCs are by no means a new concept But we re seeing more that give developers access to raw firmware and that deploy multiple ARM architectures A Linux framework called rpmsg and remoteproc is proof of this Introduced into the Linux kernel around 2011 remoteproc Remote Processor Framework that allows a Linux master to control/manage remote processors (power on/off, reset, load firmware) rpmsg Remote Processor Messaging Messaging framework that provides inter-processor communication (IPC) between kernel drivers and remote processors

12 Linux Remoteproc AMP framework was introduced to Linux due to an increasing number of heterogeneous hardware platforms Introduced in 2011 Available as of Linux The key components of the framework are based on two responsibilities Management The remoteproc component is a mechanism that allows the Linux master to start software on a remote processor Messaging The rpmsg component is remote processor messaging that provides inter-processor communication (IPC) Linux AMP framework was limited in scope Masters expected to be Linux No framework provided for firmware on remote processors

13 The OpenAMP framework was introduced to expand the scope of the original Linux AMP framework Provides a software framework for remote processors (for example, RTOS or bare-metal) Adopts the same conventions as with Linux (remoteproc and rpmsg) Master no longer needs to be Linuxbased Introduced by Mentor Graphics in collaboration with Xilinx in 2014 Now a working group under Multicore Association (MCA) Helping formalize and standardize OpenAMP Framework

14 Reminder: Unsupervised OpenAMP offers unsupervised AMP Framework is embedded into application and/or OS code No natural isolation benefits Consequently designer has more responsibilities Partitioning memory Partitioning devices Failure recovery

15 Remoteproc Startup Flowchart Assumes master is already running and remote processor in some sort of standby or powered down state Master loads firmware for remote processor into memory based on configuration information embedded into firmware Master then starts remote processor and waits for it to initialize For example, wake-up, release from reset, power-on, etc. Master is notified when initialization is complete and then establishes a communication channel to the remote processor

16 Accessing Remote Firmware When Linux is the master, the file system is the obvious place to store remote firmware Linux has a built-in firmware loading mechanism to handle this (userspace firmware loading conf must be enabled) But what about for standalone code? A couple options are supported 1. Add basic file-system support Developer must fill-in get_firmware function based on selected file-system 2. Embed firmware into master firmware Use objcpy to add to master ELF file

17 OpenAMP Remoteproc API Actual API is pretty straightforward API calls for master: remoteproc_init remoteproc_deinit remoteproc_boot remoteproc_shutdown API calls for remote: remoteproc_resource_init remoteproc_resource_deinit Header file supplied for high-level configuration For example, how long a master should wait after starting a remote Remember: simple on the surface, but architecture-specific code must be filled in under the hood

18 Packets contain source and destination IDs as part of the packet header Rpmsg Illustration Endpoints allow for the binding of multiple callbacks to the same channel For example, use different endpoints for data destined for different applications/threads

19 Also pretty straightforward OpenAMP Rpmsg API rpmsg_send rpmsg_sendto rpmsg_send_offchannel rpmsg_trysend rpmsg_trysendto rpmsg_trysendoffchannel rpmsg_get_buffer_size rpmsg_create_ept rpmsg_destroy_ept rpmsg_chnl_cb_t rpmsg_rx_cb_t

20 How Do We Use It? Framework must be ported to a specific architecture and/or device Special structures must be populated and embedded into user code Remote firmware must be accessible to master (Linux only) Linux rpmsg driver must be ported to support remote processors

21 Completing The API OpenAMP framework includes a Hardware Interface Layer (HIL)...think HAL, but with an I HIL declares functions and structures that must be filled in For example, hil_platform_ops structure in hil.h declares platform operations/functions Platform specific functions and structures are kept separate from baseline framework Given a specific location in source structure For example, lib/system/... Or incorporated into remote application For example, rsc_table.c

22 Resource Table A resource table must be included with the remote application C structure declared by HIL Application supplies information (e.g., rsc_table.c) Resource table includes some essential information for the master Firmware carve-out requirements VirtIO device information Framework includes a mechanism for the master to retrieve this information from the remote FW

23 System Partitioning Example

24 MCA Working Group Multicore Association hosts several working groups that provide solutions to various needs in a multi-core/processor system Working groups include Multicore Communications API (MCAPI) Message Passing / IPC OpenAMP Complete AMP framework Multicore Resource-Management API (MRAPI) Managing Shared Resources Multicore Task Management (MTAPI) Task Parallelism / Scheduling These various working groups address different problems/needs Each can be used independently but one could compliment another e.g. OpenAMP provides built-in IPC but is also compatible with MCAPI

25 Agenda Heterogeneous SoCs Linux and OpenAMP OpenAMP for HSA

26 Common Commercial Use-cases SLAVE MASTER Linux RTOS Bare-Metal Linux - Y* N RTOS Y Y N Bare-Metal Y N N NOTE: N use-cases are supported by OpenAMP framework but not much commercial interest

27 Xilinx Supplied Solutions SLAVE MASTER Linux (on APU) RTOS Bare-Metal Linux - Y N RTOS (on RPU) Y Y N Bare-Metal (on RPU) Y N N Xilinx supports most common use case: Linux master on APU Support for other use-cases available through 3 rd parties e.g. Mentor Graphics, Micrium (Silicon Labs)

28 Linux Master Kernel Space user kernel code User Space user application Kernel Space user kernel driver User Space user application OpenAMP remoteproc rpmsg LibMetal OpenAMP OpenAMP Kernel Space remoteproc rpmsg remoteproc rpmsg UIO/VFIO driver Increased Flexibility

29 LibMetal LibMetal provides a common API for Device access Interrupt handling Memory management Synchronization primitives Exposes these low-level functions to user-space in a controlled manner But this is not just limited to Linux Application Space LibMetal (OS Adaption) Atomics Locks Shmem I/O Mem IRQ Bus Device OS Device

30 LibMetal Across the Board LibMetal allows some low-level mechanisms to be used across different software platforms Offers better separation between platform and nonplatform specific components of OpenAMP Now available as a separate repo within the OpenAMP project: github.com/openamp/open-amp github.com/openamp/libmetal RTOS App user application OpenAMP remote proc LibMetal RTOS rpmsg virtio Bare-metal user application OpenAMP remote proc LibMetal BSP rpmsg virtio

31 Stacks MASTER User Space user application OpenAMP remote proc LibMetal rpmsg virtio Kernel Space UIO/VFIO driver REMOTE Bare-metal user application OpenAMP remote proc LibMetal BSP rpmsg virtio

32 Stacks MASTER User Space user application OpenAMP remote proc LibMetal rpmsg virtio Kernel Space UIO/VFIO driver Bare-metal user application OpenAMP remote proc REMOTE LibMetal BSP rpmsg virtio

33 Stacks MASTER User Space REMOTE user application Bare-metal OpenAMP remote proc LibMetal rpmsg virtio Kernel Space UIO/VFIO driver user application OpenAMP remote proc LibMetal BSP rpmsg virtio MASTER User Space user application Kernel Space OpenAMP remoteproc REMOTE Bare-metal user application BSP

34 Dependent Remote Firmware APU crashes APU starts remote APU Master Running APU/RPU linked via OpenAMP link broken RPU Remote Running APU restarts APU must kill remote APU Master Running APU starts remote link reestablished RPU Remote Running Master always spawns remote firmware Master also establishes communication with remote processor Communication link is lost if master goes down Restarting remote firmware may be undesirable

35 Independent Remote Firmware APU crashes APU Master Running APU restarts APU Master Running APU starts remote APU/RPU linked via OpenAMP link broken link reestablished RPU Remote Running Work is in progress to allow remote to continue running Management of the remote firmware can be re-established Already implemented for simpler situations e.g. remoteproc only

36 What About the Other Use-Cases? SLAVE MASTER Linux RTOS Bare-Metal Linux - Y* N RTOS Y Y N Bare-Metal Y N N Things get more interesting Generally left to the expertise of commercial RTOS vendors Note that master/slave no longer limited to APU/RPU

37 AMP on APU SoC APU RPU Core1 Core2 Core1 Core2 Core3 OS1 Core4 OS2 Could do this

38 Supervised AMP on APU SoC APU RPU OS1 OS2 Hypervisor Core1 Core2 PE1 PE2 PE3 PE4 but more likely to find this.

39 Benefits Of Supervised AMP Hypervisor handles low-level complexities associated with AMP such as Inter-OS communication Sharing resources between OSs Devices Memory Processors OS management (resetting, spawning, dynamic re-configuration) Isolation between OSs OS crash or corruption will not bring down or impact other OSs Inherit support on APU boosts performance e.g. EL2, virtual interrupts, interrupt routing, etc.

40 Resource Pinning (Xen Example) Devices can be passed through directly to a VM VM gets exclusive control No dependency on Dom0 CPUs can be pinned to a VM CPU no longer pooled as a shared resource Separate Linux instance still recommended Mitigates stability and security risks associated with using Dom0

41 Supervised AMP + OpenAMP SoC APU rpmsg RPU RTOS remoteproc Master OpenAMP Master rpmsg Core1 OpenAMP Remote Core2 OpenAMP Remote Hypervisor PE1 PE2 PE3 PE4

42 Learn More About Xilinx Zynq SoCs Go to to see Hardent s full training schedule Contact training@hardent.com to find out more about customized training packages for your team

43 Reference Material & Speaker Details github.com/openamp Shaun Purvis Embedded/Processor Specialist, Eastern Canada XILINX, Inc.

44 Thank You!