BLAST on Intel EM64T Architecture

Transcription

1 Performance Characterization of BLAST on Intel EM64T Architecture Intel Extended Memory 64 Technology (EM64T) is designed to provide enhancements to IA-32 architecture that can support 64-bit address extensions. This article introduces EM64T, including a discussion of different features and modes of operation. In addition, this article examines the performance of the basic local alignment search tool (BLAST) in EM64T s various operating modes on a Dell PowerEdge 1850 server. BY GARIMA KOCHHAR, KALYANA CHADALAVADA, AMINA SAIFY, AND RIZWAN ALI Several high-performance computing (HPC) applications from the domains of simulation, modeling, and image rendering can benefit from a larger addressable memory space than is currently available on 32-bit systems. HPC applications from the fields of cryptography and scientific computing may also benefit from a 64-bit processor architecture. Intel Extended Memory 64 Technology (EM64T) 1 architecture can help provide a way for such applications to take advantage of a 64-bit address space without requiring the applications to be ported to an entirely new 64-bit architecture an effort that typically involves recoding and retesting, which can potentially absorb significant resources. 2 In this article, the basic local alignment search tool (BLAST) is used as an example application to introduce the performance characteristics of EM64T architecture. BLAST is an application used in biological research that provides a method to rapidly search nucleotide and protein databases. Overview of EM64T Introduced in early 2004, Intel EM64T is a 64-bit extension architecture that was previously referred to as Clackamas Technology (CT) or IA-32 extension (IA32e). The idea behind the EM64T design is to allow applications to address larger amounts of memory than 32-bit processors and to support the coexistence of 32-bit and 64-bit applications in a 64-bit address space. A processor that is EM64T-capable requires an EM64T-aware BIOS. The EM64T specification calls for no other platform changes. An EM64T-capable processor is designed to support both 32-bit and 64-bit 3 operating systems, although only one operating system (OS) can be active on a physical server at a time. On a server running an EM64T OS, both 32-bit and 64-bit applications can be active at the same time. Apart from the capability to address a larger memory space natively than 32-bit processors and support 64-bit applications, an EM64T processor also includes features and enhancements such as Intel Hyper-Threading Technology, the Streaming SIMD (single instruction, multiple data) Extensions 3 (SSE3) instruction set, an 800 MHz frontside bus (FSB), and Demand-Based Switching (DBS) power management (see Figure 1). Compared to previous-generation Intel Xeon processors, which were based on 0.13 micron process technology, 1 For more information about Intel EM64T, visit and 2,3 With the advent of the Intel EM64T architecture, Intel now provides two distinct paths for 64-bit computing: EM64T and Itanium (IA-64). Because the actual chip architectures and instruction sets are different, the two processor chips require different versions of operating systems. This article discusses the EM64T version of operating system. 46 POWER SOLUTIONS Reprinted from Dell Power Solutions, October Copyright 2004 Dell Inc. All rights reserved. October 2004

2 Function Feature Description 64-bit processing EM64T EM64T includes extended memory addressability (that is, 64-bit pointers and registers); eight additional registers for SSE3; eight additional general-purpose registers; and double-precision integer support. IA32e mode (discussed in the section Different operating modes of an EM64T processor in this article) will support both 32-bit and 64-bit applications side by side under a 64-bit operating system. Bus bandwidth 800 MHz The FSB provides the processor with more bandwidth FSB and faster access to memory than in previousgeneration Intel Xeon processors, which offered a 533 MHz FSB. Instruction set SSE3 SIMD programming processes more data with fewer instructions compared to the traditional single instruction, single data (SISD) processors, thus helping to improve performance. SSE3 enables accelerated processing of complex arithmetic and video decoding. Power DBS DBS minimizes energy consumption and heat by management changing processor performance or power state (that is, voltage or clock frequency) in response to processor utilization. EM64T-capable Intel Xeon processors are based on a smaller, 90 nm line geometry allowing designers to fit denser circuitry on the same size processor to help enable enhanced clock frequencies. Furthermore, Intel has doubled the level 1 (L1) and level 2 (L2) cache sizes on EM64T compared to the previous-generation Intel Xeon processors. EM64T s L1 data cache has been increased to 16 KB and its L2 cache has been increased to 1 MB up from the 8 KB L1 data cache and 512 KB L2 cache on previous-generation Intel Xeon processors. Different operating modes of an EM64T processor Installing a 32-bit OS on a server with 32-bit drivers and 32-bit compilers enables support only for 32-bit applications. This setup, known as Legacy mode, represents an IA-32 environment that does not support 64-bit applications. Installing the 64-bit EM64T OS with 64-bit drivers and both 32-bit and 64-bit compilers enables support for 32-bit applications as well as 64-bit applications simultaneously on the same server. This is IA32e mode, also known as Long mode. IA32e mode is an extension of Legacy mode, and it supports two sub-modes: Compatibility mode and 64-bit mode. A 32-bit application running on a 64-bit OS will execute in Compatibility mode. A 64-bit application running on a 64-bit OS will execute in 64-bit mode. The 32-bit application will continue to use 32-bit components such as libraries and compilers that are included in the 64-bit OS. Note: The 64-bit OS for EM64T is not the same as an IA-64 OS. EM64T and IA-64 are two entirely different architectures with different instruction sets. A 32-bit application compiled with a 32-bit compiler and a 64- bit application compiled with a 64-bit EM64T compiler can execute simultaneously in a 64-bit OS in their respective modes. These two modes are transparent to the user; the only caveat is that separate binaries must be generated for 32-bit and 64-bit applications using the appropriate compilers. These requirements are summarized in Figure 2, and further detail is provided in the following sections. 4 Legacy mode. Legacy mode includes three sub-modes: Protected mode, Real Address mode, and Virtual 8086 mode. Legacy mode is designed to execute a 32-bit application the same way the application would execute on a pure IA-32 processor. An EM64T processor running a 32-bit OS will operate in Protected mode. The processor will transition to IA32e mode once a bit in the IA-32 Extended Feature Enabled Register (IA32-EFER) is set and the Physical Address Extension (PAE) mode is enabled by the EM64T OS. IA32e mode. Both IA32e sub-modes 64-bit mode and Compatibility mode provide eight floating-point registers (in addition to the original eight in previous-generation Intel Xeon processors). However, 64-bit mode and Compatibility mode provide different numbers of general-purpose registers. 64-bit mode: This mode is enabled when a 64-bit application runs on a 64-bit OS. The 64-bit mode is enabled by the OS on a code-segment-by-code-segment basis and allows 32-bit and 64-bit applications to execute simultaneously. In 64-bit mode, all registers, including the instruction pointer, are 64 bits. This mode has a total of 16 SSE3 128-bit registers, and 13 additional SSE3 instructions. The default operand is 32 bits but can be extended to 64 bits by the compiler using the provided operation code prefixes. The 32-bit operands generate 32-bit results that are zero-extended to 64-bit results in the destination 64-bit registers. The EM64T Intel Xeon processor DP has 36 bits of address bus width designed to support up to 64 GB of physical IA32e mode/long mode IA-64 Legacy Compatibility 64-bit mode architecture Mode mode (bits) mode (bits) (bits) (bits) Default Address 32/16 32/ sizes Operand 32/16 32/ General Purpose Register (GPR) width OS needed Application recompile No No Yes Yes needed Figure 2. Operating modes of an EM64T processor 4 For more information about EM64T, refer to volumes 1 and 2 of the 64-bit Extension Technology Software Developer s Guide (visit 48 POWER SOLUTIONS October 2004

3 address space. In comparison, the EM64T Intel Xeon processor MP is expected to have 40 bits of address bus width designed to support up to 1 TB of physical address space. Both processors are designed to support 48 bits of virtual memory, which is designed to address up to 256 TB of memory. The limit on physical addressing is implementation-specific and not bounded by the technology. In addition, 64-bit mode can allow two kinds of paging mechanisms: three-level paging with 2 MB pages and four-level paging with 4 KB pages. Compatibility mode: Compatibility mode can permit legacy 16-bit and 32-bit applications to execute without recompilation under a 64-bit OS. Like 64-bit mode, Compatibility mode is enabled by the OS on an individual codesegment basis. In Compatibility mode, applications can access only up to the first 4 GB of linear address space, standard IA-32 instruction prefixes, and standard IA-32 registers. Compatibility mode must also use 16-bit and 32-bit address and operand sizes. Compatibility mode, like Protected mode, is designed to allow applications to access up to 64 GB of physical memory using PAE. However, some differences exist between Compatibility mode and Protected mode. Virtual 8086 mode, task switches, and stack parameter copy features are neither available nor supported in Compatibility mode. Thus, legacy applications that run in Virtual 8086 mode or use hardware task management will not work in Compatibility mode. Furthermore, from the viewpoint of the OS, 64-bit mechanisms are used to handle system data structures, address translation, and interrupt and exception handling in Compatibility mode (instead of the 32-bit mechanisms that are used in Protected mode). Advantages of EM64T EM64T is designed to provide investment protection by allowing administrators to revert to a standard IA-32 environment (using Protected mode or Compatibility mode) if their existing applications do not experience any benefit from executing under the 64-bit architecture. EM64T also helps provide a proving ground that enables organizations to develop and test 64-bit versions of applications using 64-bit mode from the vantage point of the familiar IA-32 environment. The main advantage of EM64T is its capability to allow legacy 32-bit applications as well as 64-bit applications to execute on the same platform and even on the same OS. This capability has the potential to lower total cost of ownership (TCO) because both 32-bit and 64-bit applications can coexist on an EM64T-capable processor without the need for extra investments such as servers that have 64-bit processors in addition to servers that have 32-bit processors. Instead, the EM64T architecture enables organizations to use a single server to run both 32-bit and 64-bit applications. A 32-bit application can be migrated as is or recompiled to help leverage architectural enhancements and features such as SSE3, a faster FSB, and Hyper- Threading Technology. Memory-constrained 32-bit applications have the potential to experience immediate performance improvements from the larger addressable memory space. Before organizations can benefit from the full range of EM64T features and functionality, they may be required to invest in porting the application code, recompiling, and testing performance in 64-bit mode. This investment in application porting can be made over a period of time because existing 32-bit applications can be run during the transition. EM64T-capable processors are priced below the IA-64 Intel Itanium processor, to help enable organizations to utilize the power of 64-bit computing at an affordable price. Overview of BLAST The BLAST family of sequence database-search algorithms serves as the foundation for much biological research. BLAST algorithms search for similarities between a short query sequence and a large, infrequently changing database of DNA or amino acid sequences. Newly discovered sequences are commonly searched against a database of known DNA or amino acid sequences. Similarities between the new sequence and a gene of known functions can help identify the function of the new sequence. The scores assigned in a BLAST search have a well-defined statistical interpretation, making real matches easier to distinguish from random background hits. BLAST uses a heuristic algorithm that seeks local as opposed to global alignments. Other uses of BLAST searches include phylogenetic profiling and pairwise genome alignment. NCBI BLAST Established in 1988 as a national resource for molecular biology information, the National Center for Biotechnology Information (NCBI) creates public databases, conducts research in computational biology, develops software tools for analyzing genome data, and disseminates biomedical information. NCBI hosts a public-domain implementation of the BLAST algorithms. This implementation supports uniprocessor and symmetric multiprocessing (SMP) computing systems and is popularly known as NCBI BLAST. 5 NCBI BLAST implements database fragmentation techniques when using multithreading on SMP architecture. 5 For more information about NCBI and NCBI BLAST, visit and refer to Basic local alignment search tool by Stephen F. Altschul, Warren Gish, Webb Miller, Eugene W. Myers, and David J. Lipman (1990) at POWER SOLUTIONS 49

4 Unfortunately, traditional approaches to sequence homology searches using BLAST have proven to be too slow to keep up with the current rate of sequence acquisition. Because BLAST is both computationally intensive and exceedingly parallel, it is an excellent candidate for parallel implementation. Several open source and commercial implementations are available that Because sequence comparison is an exceedingly parallel operation, the BLAST application can be an excellent choice for parallel processing. enable BLAST to execute on a cluster of standards-based servers. An example of one such open source parallel implementation, based on Message Passing Interface (MPI), is mpiblast. 6 BLAST characteristics The BLAST application is memory intensive and highly sensitive to memory bandwidth. BLAST uses the mmap function to map file system objects into virtual memory. The file system objects comprise formatted data sets that are usually very large in size potentially tens of gigabytes. Therefore the universal buffer cache (UBC) is heavily used to keep the data sets cached in memory. If the entire data set cannot be cached in memory because of the OS s limited UBC parameter settings or because of low memory caused either by inadequate physical memory to hold the data sets or by the processor architecture s limited memory-addressing capability the file system objects will be paged. This page swapping can significantly reduce runtime performance. However, the BLAST application does not involve any memory write-back operations on the database which alleviates the burden on CPU and memory and the fetched data can be deleted immediately after use. Hence the memory-to-cpu bandwidth is also an influential factor in the performance of BLAST. Because sequence comparison is an exceedingly parallel operation, the BLAST application can be an excellent choice for parallel processing. BLAST can be run on an SMP system as well as on a cluster of standards-based servers. In either case, a divide-andconquer technique is employed on the database. BLAST uses threads to achieve parallelism on an SMP system. In an SMP system, memory latency can become more important than in a parallel-processing cluster because the memory subsystem must cater to multiple processes. When all the processors share the same memory bus, unless the bus is capable of an adequate data-transfer rate it can become a bottleneck that cripples application performance. BLAST is an integer operation intensive application. This implies that standard benchmarks such as SPECint 7 can be useful in selecting the right hardware platform for processing the BLAST application. In addition, SPECint results can be used to cross-check the results from BLAST. Test bed details Although for pure performance it would be optimal to run BLAST in a cluster, for the purpose of the study conducted in June 2004 by Dell engineers for this article, BLAST was run on a single processor. The aim of the Dell study was to compare and understand the performance of the different modes of operation offered by the EM64T architecture. To understand how the performance of BLAST compares in the three modes of operation Protected mode, Compatibility mode, and 64-bit mode the following three test cases were performed: Test case 1: Compiling and executing BLAST on a 32-bit OS in Legacy mode. Test case 2: Executing the 32-bit binary compiled in test case 1 on a 64-bit EM64T OS in Compatibility mode. Test case 3: Compiling and executing BLAST on a 64-bit EM64T OS in 64-bit mode. For all three test cases, the BLAST database and queries were run on a Dell PowerEdge1850 server configured as detailed in the following sections. BLAST setup Two BLAST algorithms blastp and blastn were used to benchmark BLAST performance in all three test cases in this study. The blastp algorithm performs an amino-acid-versus-amino-acid search against a database of protein sequences, while the blastn algorithm performs a nucleotide-versus-nucleotide search against the database of protein sequences. The following databases were used: Protein: The nr data set comprised 2,715,099 sequences represented by 757,237,578 letters (1 GB). Nucleotide: The nt data set comprised 2,218,584 sequences represented by 10,347,811,295 letters (10 GB). Queries of following word sizes were used: Protein: 153,117; 206,848; 237,455 Nucleotide: 94,799; 143,357; 187,517 6 For more information about mpiblast, see The Design, Implementation, and Evaluation of mpiblast by Aaron Darling, Lucas Carey, and Wu-chun Feng (2003) at For information about MPI and MPICH, a portable implementation of MPI, visit and 7 For more information about SPECint, visit 50 POWER SOLUTIONS October 2004

5 Time (sec) 25,000 23,000 21,000 19,000 17,000 15,000 13,000 11,000 Test case 1: 32-bit OS and 32-bit binary in Legacy mode Test case 2: 64-bit EM64T OS and 32-bit binary in Compatibility mode Test case 3: 64-bit EM64T OS and 64-bit binary in 64-bit mode The servers used the following operating systems: Red Hat Enterprise Linux WS 3 for IA-32 applications Red Hat Enterprise Linux WS 3 (update 2) for EM64T applications The following compilers were used: Intel C and Fortran Compilers 8.0 for IA-32 applications Intel C and Fortran Compilers 8.1 (beta) for EM64T applications 9,000 The follow execution parameters were used for runs in all three test cases: All databases were uncompressed and were in FASTA format. All databases were formatted using the formatdb utility (provided by NCBI BLAST source code). All databases were staged on local disks before executing the query. All time measurements were accomplished using the Linux time command. Server setup 153, , ,455 Query word size Figure 3. Performance of blastp on a PowerEdge 1850 server in different modes of operation The testing environment was based on a single Dell PowerEdge 1850 server with the following specifications: Intel E7520 chip set Two EM64T-based Intel Xeon processors running at 3.6 GHz 1 MB of L2 cache An 800 MHz FSB 8 GB of double data rate 2 (DDR2) RAM at 400 MHz obtained by using four 2 GB dual in-line memory modules (DIMMs) Two integrated Intel 82541GI/PI Gigabit 8 Ethernet controllers Intel Hyper-Threading Technology turned off Note: The E7520 chip set is designed to accommodate up to six registered 400 MHz DDR 2 DIMMs and provide an option for dual 64-bit Peripheral Component Interconnect Extended (PCI-X) slots or dual PCI Express x4 slots for interconnects and appropriate network interface cards (NICs). Results and analysis The two BLAST algorithms blastp and blastn were executed with the relevant databases. The test results for blastp are shown in Figure 3. The results in Figure 3 show that the 64-bit binary executed the blastp algorithm on a 64-bit OS (test case 3) consistently better than a 32-bit binary (test cases 1 and 2). The 64-bit binary in test case 3 showed approximately 5 percent performance improvement (measured by taking the average improvement attained by the 64-bit binary across the three queries) over the 32-bit binary in test case 1, which can be attributed to the larger addressable memory space available to the application in 64-bit mode. In addition to a larger addressable memory space, 64-bit mode provides more registers than Legacy mode and also more registers than are available for 32-bit applications in Compatibility mode. In this study, the increased number of registers in 64-bit mode helped reduce the number of register memory operations, thereby improving performance in test case 3 versus test cases 1 and 2. Performance of a 32-bit binary on a 64-bit OS (test case 2) was comparable to that of a 32-bit binary on a 32-bit OS (test case 1). Similar behavior was observed when testing blastn (see Figure 4). When executing the blastn algorithm, the 64-bit binary on the 64-bit OS (test case 3) outperformed the 32-bit binaries (test cases 1 and 2) by 6 percent (measured by taking the average improvement attained by the 64-bit binary across the three queries) because the application was able to utilize all the available physical memory without being restricted by the 4 GB limit imposed by 32-bit architecture. This advantage is more relevant in the blastn example In this study, the increased number of registers in 64-bit mode helped reduce the number of register memory operations, thereby improving performance. 8 This term does not connote an actual operating speed of 1 Gbps. For high-speed transmission, connection to a Gigabit Ethernet server and network infrastructure is required. POWER SOLUTIONS 51

6 Time (sec) 1,250 1,150 1, Test case 1: 32-bit OS and 32-bit binary in Legacy mode Test case 2: 64-bit EM64T OS and 32-bit binary in Compatibility mode Test case 3: 64-bit EM64T OS and 64-bit binary in 64-bit mode 94, , ,517 Query word size Figure 4. Performance of blastn on a PowerEdge 1850 server in different modes of operation than in the blastp example, because the size of the database being used is more than 4 GB. (The nt data set is 10 GB in size, while the nr data set is only 1 GB.) Additional registers provided by 64-bit mode reduced the number of register memory copies, thus improving the performance of the application. The 32-bit binary on a 32-bit OS (test case 1) performed almost as well as the 32-bit binary on a 64-bit OS (test case 2) when executing both blastp and blastn. This behavior was expected because Compatibility mode helped to ensure that the differences in the platform were transparent to the application when the 32-bit binary was being executed on a 64-bit OS. In fact, there may be a slight degradation in the performance of a 32-bit application on a 64-bit OS because the remaining 32 bits have to be zero-padded that is, filled in with zeros to form 64-bit words for proper execution. However, in this study the degradation observed was not significant. The 5 to 6 percent performance improvement for both blastp and blastn in 64-bit mode (test case 3) was achieved only by recompiling the application code with 64-bit compilers. Porting the code was unnecessary to achieve this gain. An option for transitioning to 64-bit computing The Intel EM64T based processor architecture is designed to run either a 32-bit OS or a 64-bit OS, thereby enabling enterprises to support both 32-bit and 64-bit applications on the same hardware platform. The study discussed in this article showed that the performance of a 32-bit binary on the 64-bit EM64T OS was comparable to that of a 32-bit binary on a 32-bit IA-32 OS. By comparing the performance of BLAST on a PowerEdge 1850 server using three modes of operation Legacy mode, Compatibility mode, and 64-bit mode Dell engineers demonstrated in this study that a 64-bit binary can exploit the advantages of Intel EM64T extended memory and other processor enhancements to achieve higher levels of performance than an IA-32 OS can provide. To take advantage of benefits provided by the Intel EM64T processor s 64-bit mode, organizations need to recompile 32-bit application code with an EM64T compiler; recoding and retesting code should not be necessary in most cases. Re-certification of applications may be required depending on how much nonstandard coding the application uses. 9 By potentially saving organizations the time, effort, and resources required to port code, EM64T can help make the transition to 64-bit computing easy and affordable while providing 32-bit compatibility. Garima Kochhar is a systems engineer in the Scalable Systems Group at Dell. She has a B.S. in Computer Science and Physics from Birla Institute of Technology and Science (BITS) in Pilani, India. She has an M.S. from The Ohio State University, where she worked in the area of job scheduling. Kalyana Chadalavada is a senior engineer with the Dell Enterprise Solutions Engineering (ESE) Group at Bangalore Development Center (BDC). Kalyan has a bachelor of technology degree in Computer Sciences and Engineering from Nagarjuna University, India. His current interests include performance characterizations on HPC clusters and processor architectures. Amina Saify is a member of the Scalable Systems Group at Dell. Amina has a bachelor s degree in Computer Science from Devi Ahilya University (DAVV) in India, and a master s degree in Computer and Information Science from The Ohio State University. Rizwan Ali is a systems engineer in the Scalable Systems Group at Dell. His current research interests are performance benchmarking and high-speed interconnects. Rizwan has a B.S. in Electrical Engineering from the University of Minnesota. FOR MORE INFORMATION Intel EM64T home page: EM64T developer s guide: EM64T Intel Xeon processor MP: part2.html NCBI and NCBI BLAST: mpiblast: 9 For more information about porting 32-bit applications to the EM64T architecture, see 52 POWER SOLUTIONS October 2004