Using Virtual Platforms To Improve Software Verification and Validation Efficiency

Transcription

1 Using Virtual Platforms To Improve Software Verification and Validation Efficiency Odin Shen Staff FAE Arm Arm Tech Symposia Taiwan 2017

2 Software complexity and best practices

3 Software Costs Increasing Primary Design Costs By Process Node The cost of developing and qualifying software increases rapidly with higher complexity IC designs High-performance SoCs include multiple processor cores and 100s of IP blocks Source: IBS

4 Software Verification and Validation Traditional flows involve development followed by testing. Modern practices such as Continuous Integration and Extreme Programming call for continuous development & test cycles. Develop -> Commit -> Test cycles daily, or several times a day Frequent commits help reduce the number of conflicting changes. Frequent testing gives clues about where and when the error was introduced. 4

5 Continuous Integration Automation Server, e.g. Jenkins Commit in a version control system Test run scheduling Triggered after the other builds in the queue have completed Scripting Configure target Connect to target Initiate tests Stop target and query state of memory, registers, etc Control and Reporting Reset, results presentation, etc Target 5

6 Automation is Key Efficient Continuous Integration programs require good automation. Integration with common automation platforms such as Travis, Jenkins and AppVeyor Kick off jobs as soon as a commit is made Overnight testing Results collection and presentation Robust, stable, automatable target platforms 6

7 Targets For Your CI System Continuous software integration enforces fail-fast, but you have to have the right infrastructure. Key component of testing software on Arm is a fast, available, cheap, automatable target. Let s look at target options for a CI system 7

8 Choosing a Target for your Software Testing

9 Software Test Platform Options Emulator FPGA SoC Virtual Platform 9

10 Variables to Consider Accuracy How much accuracy do you need? What is more important, accuracy or speed? Performance What kind of dependency do you have on performance? Can your testing finish within the desired time window? Debug and Analysis How stable is your design? How much debug do you expect? How much visibility into hardware/software interface do you need? Availability Are the targets available? Are there sufficient quantity to scale out to a large test environment? Capacity How big is your design? Will it fit within the target s capacity? Does cost scale with the capacity required? Cost Have you considered scaled out cost? Infrastructure cost? Setup cost? Maintenance cost? Flexibility What is turnaround time for changes? 10

11 Platforms Based on Simulation If you don t need cycle accuracy, most will agree that a simulated target is preferable. Minimal infrastructure Easy to maintain Easy to deploy No capacity issues No expensive probes Easily automated Excellent debug capabilities Some testing will require cycle accuracy, but testing functional correctness, software-software and software-hardware correctness don t. 11

12 Hardware Software Co-design Standard Project Flow Without Virtual Platform Spec Freeze Tape Out Silicon Project Finished Arch Design SoC Hardware Development Manufacturing Software Development, HW/SW Integration & System Validation Standard Project Flow With Virtual Platform Spec Freeze Tape Out Silicon Project Finished Arch Design SoC Hardware Development Manufacturing Software Development, Integration & System Validation Gained TTM 12

13 Virtual Prototype Availability Armv8 Juno test chip bring-up timeline 2 years 10 days Development on virtual platform before silicon Development on hardware Fast Models Arm DS-5 Arm TrustZone Full software stack and tools validated on hardware 13

14 The Real Value of Simulation Platforms Better debug capabilities. Why? Visibility and control. Non-intrusive, deterministic debug Control, visibility and determinism are guaranteed Immediate halt of entire system on watchpoint or breakpoint No expiring watchdogs or transient states Capture state for testing, then carry on running as if the stop never occurred VP Debug callback Checkpointing Reverse execution Built in or 3rd party tools 14

15 Scripting Example Model connection and configuration Set breakpoints Run the application Get status of variables, registers, memory, etc. Variable status determines test pass/fail 15

16 Debug Example Virtual Platforms provide almost unlimited tracing capabilities. Virtually every signal, register, memory or CPU program flow can be traced. No need for special hardware or software instrumentation A user can define his/her own monitors and query information about OS state and threads to give more context when printing out results. A watchpoint can be set on a signal between a core cluster and a subsystem. Callbacks from the model can non-intrusively report system state. Stop on access 16

17 Virtual Prototype Availability Cycle Accurate Debug Interface (CADI) defines debug interface to model. Model Trace Interface (MTI) defines trace interface to model. CADI and MTI are open and freely available. Enables a debug user experience similar to hardware debug. Scripting interface for advanced capability. Source Level Debugger 17

18 Biggest Challenge with Virtual Platforms Where s the model? Companies like Synopsys, Arm, Intel have large modelling teams who create models of their IP But what about the rest of the system: UART, keyboard, display, speaker, etc? Someone has to model them, or, you exclude those peripherals from your testing As hardware evolves, the models must also evolve 18

19 Arm Models Arm Models come from various places. Fixed Virtual Platforms (FVP) Roll your own with Fast Models, OVP, QEMU Ecosystem partners Silicon partners The closer to the actual system, the better. What if I don t have a full system simulation? Test what you can Stub out code 19

20 Regression Testing Regression farms also benefit from models. Automated defect and performance regression testing. Same advantages as CI scenario. Easy to setup and deploy Highly automatable Superior debug-ability Easy to maintain Performance Regression Testing Android Boot 20

21 Success Stories Linaro System Guidance Armv8-A FVP Customers Intel, Qualcomm, Hewlett Packard, Texas Instruments, Cirrus, Nokia, Ericsson, ST, HiSilicon, Samsung, and many more Arm internal usage (compiler, models, system software, IP development and testing) Arm System Guidance Platform 21

22 Case Study AI Inference on Cortex-A

23 NN on Arm? Deploy NN inference on Cortex v8.2 platform w/o real hardware How to select solution? Accuracy Performance Debug and Analysis Availability Capacity Cost Flexibility 23

24 Use MNIST as Example The MNIST database of handwritten digits, available from this page, has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image

25 How To Deploy on Arm? Purpose: Prototyping on Arm platform Early exploration NN engine design SW framework profiling Script Load image Multicore Debug and Trace Load parameter Automation Profiling Debug Trace Streamline Trace and Debug SCU Cortex-A75 ARMv8.2-A 32b/64b CPU L1 cache Private L2 cache Snoop Filter Core Peripheral Port Cortex-A55 ARMv8.2-A 32b/64b CPU L1 cache Private L2 cache Core 5 6 Async Bridges ACP AMBA 4 ACE or AMBA5 CHI Shared L3 cache 25

26 MNIST at FVP Debugging: Result: NN#1 NN#2 26

27 Let s See DS5 live demo Variables to Consider: Accuracy V Performance V Debug and Analysis Availability V Capacity V Cost V Flexibility V V Debug Trace Streamline Trace and Debug SCU Cortex-A75 ARMv8.2-A 32b/64b CPU L1 cache Private L2 cache Snoop Filter Multicore Debug and Trace Core Peripheral Port Cortex-A55 ARMv8.2-A 32b/64b CPU L1 cache Private L2 cache Core 5 6 Async Bridges ACP AMBA 4 ACE or AMBA5 CHI Shared L3 cache 27

28 Summary

29 Summary Growing software complexity is creating the need for new test strategies. Continuous Integration enables fail-fast, greatly reducing software integration efforts. Models are easy to automate and provide extended access and visibility into the system, are more reliable, and easier to maintain than hardware. Freely available Arm system models at with software stacks available from Linaro ( Looking for more about the demo? Check the git and free feel to contact for the real demo. 29

30 Thank You! Danke! Merci! 谢谢! ありがとう! Gracias! Kiitos! 30

31 The Arm trademarks featured in this presentation are registered trademarks or trademarks of Arm Limited (or its subsidiaries) in the US and/or elsewhere. All rights reserved. All other marks featured may be trademarks of their respective owners. 31