Benchmarking: Classic DSPs vs. Microcontrollers

Transcription

1 Benchmarking: Classic DSPs vs. Microcontrollers Thomas STOLZE # ; Klaus-Dietrich KRAMER # ; Wolfgang FENGLER * # Department of Automation and Computer Science, Harz University Wernigerode Wernigerode, D-38855, Germany * Department of Computer Science and Automation, Technical University of Ilmenau Ilmenau, D-98693, Germany ABSTRACT When building a new product, developers have to decide, which hardware to use. The right choice is essential for a cost-efficient and successful product. But in most cases the performance of the desired new hardware is not known. In order to get an overview and to estimate performance, benchmarks are used to compare different systems. Nowadays, many Microcontrollers claim to have improved performance on DSP-like algorithms, using specialized architecture features like multiply-andaccumulate (MAC) units or vector calculation. But is this enough to form a real alternative to common DSPs? This paper deals with the pros and cons using Microcontrollers in DSP-dominated fields of application and gives hints which architecture to choose under certain circumstances. It also includes a detailed look at the algorithms used to measure performance. Keywords: benchmarking, microcontroller, digital signal processor, performance, analysis Three types of MP-/MC-/DSP-benchmarks are known: - Synthetic benchmarks (e.g. Whetstone Benchmark, Dhrystone Benchmark [1], [2], etc.) developed to measure system specific parameters (by CPU, Compiler, and so on) - Application based benchmarks ( real world benchmarks) developed to compare different system architectures in the same real fields of application - Algorithm based benchmarks (a compromise between the first and the second type) developed to compare different system architectures in special (synthetic) fields of application. The benchmarks described in this paper are benchmarks (HSH-benchmarks) of the third type. 1. INTRODUCTION / MOTIVATION Benchmarks are necessary to compare and to assess different systems (in this case Microcontroller- or DSParchitectures) in well defined fields of application. Developers need to know how desired target systems behave under conditions that come close to the algorithms used in the target application. Choosing the correct system saves time and money and minimizes the risk of a too late time to market. Furthermore, the target system can be selected in order to keep enough room for developing future extensions. But that can only be achieved using fair benchmark comparisons. 2. SYSTEM DEMANDS In order to create benchmark algorithms, developers have to define demands for the target application, referring to special use cases in digital control or signal processing applications. In many cases, a set of different key-components is investigated: - processor core (architecture, instruction set, clock, ) - special system features, e.g. DSPinstructions, MAC-units, an so on)

2 - program memory location (Flash-ROM, ROM, RAM, Cache, ) - integrated development environments (editor, assembler, compiler, debugger, ) When it comes to evaluating performance of these components, different measuring methods are used. For example, regarding to performance, execution-time is measured to calculate comparable values like instructions per second. Other demands may address the size of required memory often depending on chosen optimizations, the compiler that is used and the instruction set of the tested Microcontroller/DSP or the used library algorithms. These demands can only be checked using special test algorithms. Regarding to the program memory location it is necessary to use the same memory structure for all tested target systems. 3. BENCHMARKS Overview The HSH-benchmark code used to test the processor architecture and compilers can be separated into eight different modules: - M1: fixed-point math algorithms - M2: floating-point math algorithms - M3: logic calculations - M4: digital control - M5: Fast Fourier Transform - M6: field processing - M7: loops and conditional jumps - M8: recursion and stack tests Each module is a unique piece of code for testing special parts of the controller and the compiler functions. M1 performs basic fixed-point calculations. The native processing width is used for each target, but all calculations can be done using 16 bit arithmetics. M2 does similar calculations with regard to double precision floating point numbers. In M3 logic calculations like and, or and xor are processed, while in M4 a complete digital control algorithm is calculated. M5 is used to evaluate performance on common FFT algorithms, using functions like sin(), too. M6 deals with double precision field operations while M7 focuses on loops and conditional jumps in state machines. Finally, M8 is used to compare performance when executing recursive algorithms. Architecture For testing selected features of a target s architecture different HSH-benchmark modules are used. These modules are written in order to test the performance of the core of the target system. Developed with regard to prior works (see [4], [5], [6]), each module represents common algorithms used in real applications. Tool Chain Complex applications require sophisticated development environments in order to support the development process. A very important component of integrated development environments (so-called IDEs) is the compiler. Translating the user program to machine code the compiler is elementary influencing the speed and size of the code. Different optimizations may be selected in order to improve the machine code even more. When there is more than one compiler available for the target system, a close look at the features and performance may reveal pros and cons, so the best compiler can be selected for compiling the user code. In order to compare the performance of compilers, identical code is used to measure speed and size on the same target. Since the results have to be fair and comparable, the settings of the compiler flags are essential. Because real world applications will commonly use most of the optimization features the compiler offers, the benchmarks are run having optimization technologies generally turned on. But there is a great risk of producing code that is not running as expected, e.g. optimizing out variables or loops. So the test runs are verified using the debugger that such cases do not appear. For example, a benchmark score for a loop created not running that loop is quite useless. In connection with the compiler, the linker takes the pieces of program code and locates that code in memory. Depending on where the program is located, RAM or flash memory, internal or external, the linker affects the execution speed, too. In addition to these performance considerations, a comparison of supported features like support of special instruction sets (DSP, SIMD, ) may be evaluated, too. A multiply and accumulate support may help executing typical DSP algorithms, while a floating-point-unit accelerates calculations with floating-point numbers. To get all these things working together, an integrated development environment is needed in order to develop complex applications. Furthermore, IDEs commonly include a debugger to solve problems and to test the code. That is why IDEs also contribute to acceptance and costs of embedded devices and their components directly and indirectly affect the performance of the device and application. Additionally, they support the developer writing smart code by providing tools like powerful editors, code profilers, debuggers, etc.

3 When running performance tests, IDEs are essential to set up hardware and to manage different configurations, e.g. memory configurations or compiler optimizations. In order to get comparable results, equivalent setups for different systems can be created where the tests are run on. 4. RESULTS Although the Microcontroller Application Center at Harz University performs test runs with various targets, only a subset of common Microcontrollers and DSPs was selected to be tested in this context and at this point of our research. The test setup contains three common Microcontrollers challenging two Digital Signal Processors. The tested hardware is in particular: Module M5 mainly makes use of addition, subtraction, multiplication and sinus calculations. The FFT used here is based on an algorithm by Danielson / Lanzcos [3]. As in M4, most of the values are also double precision floating points. In addition, some field-accessing work has to be done. Running the tests requires the code to be compiled for each target device. That is why different compilers have to be used. All tests are run having optimizations turned on, but the generated assembler code was checked in order to prove that all loops used still remain while executing the code. Furthermore, the expected results of the calculations are checked, too, to ensure there are no errors. The measurements taken by running the benchmarks represent a combined performance of hardware and compiler, even if the code is written with the focus on testing hardware. - Infineon XC167ci (20MHz) - NXP LPC2138 (60MHz) - Infineon Tricore TC1796B (150MHz) - Texas Instruments TMS320C6711 (150MHz) - Texas Instruments TMS320C5402 (100MHz) That means the three Microcontrollers are up against two well-known DSPs. In order to represent common fields of applications and to give a detailed look at the HSHbenchmarks, modules 4 and 5 were chosen to test the hardware, although all modules are run on all platforms and additional results are to be presented at the conference. But taking a look at all modules would go beyond the scope of this paper. In order to take a closer look at the algorithms used for the tests presented in the paper, here is a short code section taken from M4: //Digital Controller (PID): e = sollwertregler[i] - istwertregler[i]; esum += e; y = Kp * e + Ki * Ts * esum + Kd * (e - ealt) / Ta; Fig. 2: Digital Control Algorithms (execution time) Figure 2 shows the absolute execution times for module 4, showing how the 5 different targets perform. The Infineon Tricore marks the lowest time, which means this system is the fastest one, closely followed by the TMS320C6711. Both systems are able to make use of built-in hardware floating point support, where the Tricore can also use SIMD extensions in its instruction set when appropriate. The third-fastest chip is the second DSP, the TMS320C5402 without hardware floating point support. The remaining Microcontrollers are slower, especially the LPC2138. ealt = e; stellwertregler[i]=y; Fig. 1: C-code from M4 (excerpt) Module 4 represents digital control algorithms like controllers, saturations and gains. Most variables used in this module are double precision floating point values. So, the target has to calculate complex formulas in order to get the output value of the control algorithm. Fig. 3: Fast Fourier Transform (execution time) Looking at figure 3 which shows the results from module 5 the results are similar to the ones above, except that the

4 TMS320C6711 and the Tricore change places. So, the TMS320C6711 is the fastest device here, having the lowest execution time. The order of the other devices does not change, the LPC2138 is still the slowest of all tested chips, although it does not have the lowest clock speed. That point is very interesting when looking at the normalized results in the following figures. interesting when normalizing the results not only to time, but also to clock speed, as the following figures show. Figure 6: Digital Control Algorithms (time and speed normalized) Fig. 4: Digital Control Algorithms (time normalized) Figure 4 represents a different point of view, examining the results in thousand runs per second (kruns/s). That means the highest result is the best. So, the Tricore achieves the most runs in one second, followed by the TMS320C6711. The gap to the other DSP is not large, remembering that the TMS320C5402 does not have hardware floating point support. The Microcontrollers XC167ci and LPC2138 have the lowest scores. Both do not have floating point support by hardware, too, but the XC167ci has a special MAC unit helping to calculate the control algorithms a tick faster than the NXP device. Figure 6 shows the results of the digital control algorithms normalized to time and clock speed. That means, the lower a device is clocked and the lower the execution time for the benchmark run is, the more clockefficient is the chip, performing more runs per second and per MHz. In other words, figure 6 shows the runs per second having all devices clocked theoretically equal at 1MHz. That is why the XC167ci gets the highest score. This Microcontroller only has the lowest clock speed, but in relation to that, it performs the most runs per second. That means this device is more clock-efficient than the Infineon Tricore. Moreover, the gap between the XC167ci and the other devices is quite high. The higher clocked DSPs and the Tricore deliver the highest absolute speed, where the XC167ci delivers the highest relative speed. Fig. 5: Fast Fourier Transform (time normalized) The results from figure 3 become even more clearly when looking at figure 5. The TMS320C6711 manages to perform more than twice the runs of the Tricore. That is an impressive result since both devices run at the same clock speed of 150MHz. As in figure 3, the TMS320C5402 performs a bit slower, but is also running at a clock speed of only 100MHz. The both lowerclocked Microcontrollers achieve the lowest scores. But looking at the clock speed of only 20MHz for the XC167ci, it seems that this device is not that far away from the TMS320C5402, being clocked five times higher than the Infineon controller. This thought becomes Figure 7: Fast Fourier Transform (time and speed normalized) Figure number 7 shows the results of the FFT-test, showing that the XC167ci is the fastest one from this point of view, too. But in contrast to figure 6 the TMS320C6711 is very close, showing that the DSP has a very good clock-efficiency here, too. That means the TMS320C6711 has a clear advantage in FFT-calculation, one of the common fields of application DSPs are designed for. The Tricore is not that clock-efficient as the TMS320C6711 under these circumstances, but its MACcapabilities, the SIMD instruction-set extensions and the hardware floating point support may help accelerating the

5 executed code so it can manage to deliver a better score than the TMS320C5402. usability of Microcontrollers combined with superior calculation power of a DSP. 5. CONCLUSION Summarizing the achieved results it seems that when choosing an appropriate device, DSP-like applications may be performed, but are not limited to Digital Signal Processors. Especially the Infineon Tricore shows that a quite sophisticated and high-clocked Microcontroller can perform quite well and may be the better choice under certain circumstances. That can be proved referring to the digital control algorithms, where this device is the fastest one. But it is also well-suited for applications utilizing FFT-like algorithms. On the other hand, the Tricore has to let the C6711 pass by in M5, although it runs at the same clock speed. One reason may be that the algorithm massively makes use of double precision floating point calculations, were the floating point support of the Tricore focuses on single precision operations. However, there are devices like the PLC2138 that are not as fast as the Tricore. That is why they fit in more traditional Microcontroller applications. The XC167ci takes advantage from the built-in MAC unit and is able to compete with other devices with the help of a good compiler support. Its lower clock speed surely limits the power of the device, especially when looking at figures 5 and 6. Unfortunately, when the user needs more power on an XC167ci device, the maximum clock speed is currently limited to 40MHz. So, the user has to look for alternatives if possible, such as the XC2000 family for example. Generally said, modern Microcontrollers are not limited to simple control applications and have enough power to calculate complex algorithms on their own instead of passing these tasks by to other devices. Additionally, they bring with a rich set of peripherals, normally missing when looking at DSPs. So, by using such a Microcontroller, software development can achieve more integration which leads to reduced costs and less overhead. On the other hand, DSPs still deliver very high performance in special fields of application, despite they are limited referring to control possibilities of the whole system. That is why an ideal device may implement the ACKNOWLEDGMENT The authors wish to acknowledge the support of the Microcontroller Application Center by the KAT Network of Competence for Applied and Transfer-oriented Research in Saxony-Anhalt. REFERENCES [1] H. J. Curnow, B. A. Wichmann: A Synthetic Benchmark, Computer Journal Vol. 19, No [2] Weiss, Alan R.: Dhrystone Benchmark, History, Analysis, Scores and Recommendations, White Paper, ECL/LLC, 2002 [3] Danielson, G. C., and Lanczos, C.: "Some improvements in practical Fourier analysis and their application to X-ray scattering from liquids", J. Franklin Inst., pp. 233, and , 1942 [4] Kramer, K.-D., Banse, T.: Sinus als Standard Signal-Processing-Algorithmus als Benchmark, Design&Elektronik, p41-43, 08/2007 [5] Lojewski, D.: Benchmarking und Systemvergleich von Mikrocontrollern, Diploma-Thesis, Harz University, 2007 [6] Stolze, T., Kramer, K.-D.: Benchmarking: Klassische Mikrocontroller versus x86-prozessoren, Proceedings of ESE-Kongress, pp , 2008 All names and brands are property of their respective owners.