SUN FIRE X4140, X4240, AND X4440 SERVER ARCHITECTURE

Transcription

1 SUN FIRE X4140, X4240, AND X4440 SERVER ARCHITECTURE Compact, Dense, and Scalable Systems Based on Powerful Dual-Core and Quad-Core AMD Opteron Processors White Paper May 2008

2 Sun Microsystems, Inc. Table of Contents Executive Summary The Evolution of x64 Systems Sun Fire X4140, X4240, and X4440 Servers System Comparison A Choice of Operating Systems Innovative and Consistent System Design for the Datacenter AMD Opteron Processor Technology Dual-Core AMD Opteron Processors Quad-Core AMD Opteron Processors Server Architecture System-Level Architecture Sun Fire X4140 Server Sun Fire X4240 Server Sun Fire X4440 Server Enterprise-Class Operating System and Management Software The Solaris Operating System Integrated Lights Out Management (ILOM) Sun xvm Ops Center Conclusion Fore More Information

3 Executive Summary Sun Microsystems, Inc. Executive Summary Explosive growth in a wide range of computing disciplines is putting immense pressures on the datacenter. Applications from Web services to high performance computing (HPC) are calling for ever-increasing resources, even as they demand robust IT infrastructure that is truly mission critical. As servers grow in number, it becomes ever more important to deploy systems that consider datacenter needs as a fundamental aspect of design. Regrettably, deploying and administering dynamic compute and storage infrastructure can be fraught with complexity. Administrative, energy, and real estate costs continue to grow unabated, and sprawl within the datacenter can severely strain IT budgets. Many are responding with consolidation strategies that combine ever more powerful standard x64 systems with innovative new approaches to virtualization. Not only does consolidation offer to improve vital resource utilization, but it can also greatly enhance business agility by providing computational resources where they are needed most. Sun Fire X4140, X4120, and X4440 servers combine the latest high-performance AMD Opteron processors with Sun's proven datacenter-centric design focus. Ideal for highperformance and mission-critical applications alike, these systems deliver scalable and manageable x64 computing in dense and servicable rackmount enclosures. Using the latest Dual-Core and Quad-Core AMD Opteron processors, Sun Fire X4140, X4240, and X4440 servers are ideal for HPC, database, and Web infrastructure, as well as consolidation and virtualization initiatives. These systems offer compelling compute, memory, storage, and I/O density, as well as considerable energy efficiency to help organizations live within their means. Sun Fire X4140, X4120, and X4440 servers also share innovative rackmount packaging with Sun 's other volume x64 and SPARC servers With a choice of operating systems including the Solaris Operating System (Solaris OS), Linux, Microsoft Windows, and VMware these servers effectively combine Dual- Core and Quad-Core AMD Opteron processors with balanced system designs. Large memory support, large numbers of internal disks, large I/O bandwidth, and up to four processor sockets give these systems the capacity to handle the most demanding applications. With datacenter operation as a fundamental design assumption, these systems offer redundant and hot-swappable components, efficient front-to-back air flow, highly-efficient power supplies, and built-in system management tools. Sun Fire X4140, X4240, and X4440 servers are engineered for mission-critical application availability, conserve valuable energy resources, and lower operational expenses even as they deliver performance and agility in a dense and effective package. This document details the systems architecture of Sun Fire X4140, X4240, and X4440 servers, along with key software components.

4 2 The Evolution of x64 Systems Sun Microsystems, Inc. Chapter 1 The Evolution of x64 Systems Modern IT infrastructure is under near-constant strain. Compute and storage demands have caused datacenters to grow exponentially in recent years to support new Web services, high-performance computing (HPC), and other compute-hungry applications. To stay competitive, organizations must deploy applications quickly, delivering compute power where it is needed most, all with increasingly little margin for error. As a result, many datacenters have become sprawling and complex, and most are up against very real constraints in terms of power, cooling, and real estate. Consolidation through virtualization has emerged as an effective strategy for addressing the very real need for computing scalability while increasing the work that can be done in a given power, thermal, and physical footprint. Consolidation can improve resource utilization, reduce administrative complexity, and drive down IT costs. Consolidating many smaller legacy servers into fewer more powerful systems can also help to minimize administrative workloads while increasing capacity and conserving valuable datacenter floor space. Energy costs can be drastically reduced, vastly improving available performance relative to the amount of energy consumed. Perhaps more importantly, consolidation through virtualization often gives organizations the agility they need to apply key resources to their most important and business-critical applications. Sun Fire X4140, X4240, and X4440 Servers To help IT managers address the challenge of increasing capacity while managing datacenter growth, Sun Fire X4140, X4240, and X440 servers provide a range of dense and scalable systems based on the latest AMD Opteron processors. These systems feature high performance and unprecedented density in energy-efficient and compact 1U and 2U form-factors. With capabilities that complement the rest of the Sun server product line, the Sun Fire X4140, X4240, and X4440 servers raise the bar for 32- and 64- bit enterprise-class computing. These systems offer: Best-in-Class Performance The Sun Fire X4140, X4240, and X4440 servers feature Dual-Core and Quad-Core AMD Opteron processors. With a sophisticated cache hierarchy, and on-chip memory management, these processors offer high system performance and throughput compared with systems based on earlier-generation x86 chipsets. In high-end configurations, these servers can house multiple Quad-Core AMD Opteron processors for example, the Sun Fire X4440 server supports up to four Third-Generation Quad-Core AMD Opteron processors, for a maximum of 16 cores and impressive performance in a compact 2U chassis.

5 3 The Evolution of x64 Systems Sun Microsystems, Inc. Remarkable Density Density is the cornerstone of the Sun Fire X4140, X4240, and X4440 server designs. When populated in a 40-rack unit (RU) enclosure, these 1U and 2U servers facilitate a single rack with up to 320 cores, 640 DIMM slots, and 120 PCI Express slots. In addition, a single 1U or 2U system can house up to 2.3 TB internally using inch SAS drives (via a PCI Express Host Bus Adapter). Sun Fire X4140, X4240, and X4440 servers provide the densities needed to achieve consolidation and virtualization efficiencies. Such density facilitates the consolidation of many smaller servers, helping to conserve real estate, lower energy expense, and reduce costly administrative talent. In addition, these servers support multiple operating systems, which helps to simplify consolidation efforts and diminish server sprawl. Extensive System Expandability The ability to expand a server over time reduces the need for additional capital acquisitions and lowers application lifecycle costs. Sun Fire X4140, X4240, and X4440 servers feature either two or four processor sockets per system and provide maximum memory configurations of either 64 GB or 128 GB (using 4 GB DIMMs), along with a maximum of 2.3 TB of internal storage. Four Gigabit Ethernet ports are standard, as are five USB ports (two front, two back, one internal), and one video port (HD-15). Because of breakthrough system densities, these servers are designed to scale to support new users, more transactions, or new 32-bit or 64-bit applications, enhancing system longevity and increasing overall return on investment (ROI). Improved Energy Efficiency Sun offers a portfolio of eco-responsible products and computing solutions to address a broad range of infrastructure requirements. In Sun Fire X4140, X4240, and X4440 servers, AMD Opteron processors incorporate new technologies that minimize power use and enhance energy efficiency. AMD PowerNow! technology optimizes processor performance relative to the power consumed, allowing CPU clock frequency to be adjusted to the needs of applications. Enhanced AMD PowerNow! features in Quad-Core AMD Opteron processors offer features such as Independent Dynamic Core technology and Dual Dynamic Power Management. High-efficiency power supplies in the server chassis lessen overall power consumption. Variable-speed fans, effective disk carrier design, and front-to-back air flow in the chassis help to effectively cool the system and maintain appropriate ambient temperatures for both the processor and the system. Enterprise-Class High Availability Sun Fire X4140, X4240, and X4440 servers are designed with enterprise-class reliability, availability, and serviceability (RAS) features. To maximize uptime, systems include redundant hot-swappable fans and can be configured with redundant hot-swappable power supplies. Using a Sun StorageTek Host Bus

6 4 The Evolution of x64 Systems Sun Microsystems, Inc. Adapter (HBA), internal SAS disk drives can be configured for RAID 0, 1, 1E, 10, 5, 5EE, 50, 6, and 60. Disk drives are also hot-swappable. Four integrated Gigabit Ethernet ports enhance network availability and can be installed in failover configurations. On-board system management tools encourage proactive remote monitoring and intervention. Tightly-Integrated Management To support out-of-band management, Sun Fire X4140, X4240, and X4440 servers incorporate an Integrated Lights Out Management (ILOM) service processor. This built-in hardware-based management functionality allows administrators to monitor and manage systems remotely, letting them take corrective action as necessary to minimize unplanned downtime. Sun Fire X4140, X4240, and X4440 servers combine best-in-class performance with noteworthy compute, memory, and I/O capacities. As a result, these systems are designed to scale up, scale out, and scale within, enabling deployment in a wide range of application architectures: Scale-up architectures With up to 16 cores available, these servers are well suited to scale for growing workloads that deliver Web, database, and other key infrastructure services. Scale-out architectures With large memory capacities, significant internal storage, four Gigabit Ethernet ports, and high-bandwidth PCI Express expansion, these servers can scale to solve complex computing problems that demand intensive computing power and data bandwidth. Scale-within capabilities With the ability to support Solaris 10 virtualization technology and VMware, Sun Fire X4140, X4240, and X4440 servers are ideal systems to consolidate multiple applications within a single extensible platform.

7 5 The Evolution of x64 Systems Sun Microsystems, Inc. Figure 1 illustrates the 1U Sun Fire X4140 and the 2U Sun Fire X4240 and X4440 server enclosures. Sun Fire X4140 server Sun Fire X4240 server Figure 1. Sun Fire X4140, X4240, and X4440 servers Sun Fire X4440 server

8 6 The Evolution of x64 Systems Sun Microsystems, Inc. System Comparisons Table 1 compares the features of Sun Fire X4140, X4240, and X4440 servers. Table 1. Sun Fire X4140, X4240, and X4440 server features Feature Sun Fire X4140 Server Sun Fire X4240 Server Sun Fire X4440 Server Processors Memory capacity Maximum internal disk drives One or two Dual-Core or Quad- Core AMD Opteron series 2000 processors Up to 64 GB (1, 2, or 4 GB DDR2 DIMMs) Up to eight SFF 2.5-inch SAS 73 or 146 GB disk drives, choice of RAID HBAs a RAID Hardware RAID 0, 1, 1E, 10, 5, 5EE, 50, 6, and 60 with SAS RAID HBA Removable and pluggable I/O PCI Ethernet Power supplies Fans Slimline DVD+/-RW Five USB 2.0 ports (Two front, two rear, one internal) Two x8 PCI Express slots, One x16 PCI Express slot Four on-board Gigabit Ethernet ports (10/100/1000) Dual redundant hot-swappable AC 650 W power supply units (N+N redundancy) Redundant, hot-swappable fan modules (N+1 redundancy) One or two Dual-Core or Quad- Core AMD Opteron Series 2000 processors Up to 64 GB (1, 2, or 4 GB DDR2 DIMMs) Two or four Dual-Core or Quadcore AMD Opteron Series 8000 processors Up to 128 GB (1, 2, of 4 GB DDR2 DIMMs) Up to 16 SFF 2.5-inch SAS Up to eight SFF 2.5-inch SAS or 146 GB disk drives, choice of RAID HBAs a or 146 GB disk drives, choice of RAID HBAs a Hardware RAID 0, 1, 1E, 10, 5, 5EE, 50, 6, and 60 with SAS RAID HBA Slimline DVD-R Five USB 2.0 ports (Two front, two rear, one internal) One x4 PCI Express slot, Four x8 PCI Express slots One x16 PCI Express slot Four on-board Gigabit Ethernet ports (10/100/1000) Dual redundant hot-swappable AC 1050 W power supply units (N+N redundancy) Redundant, hot-swappable fan modules (N+1 redundancy) Hardware 0, 1, 1E, 10, 5, 5EE, 50, 6, and 60 with SAS RAID HBA Slimline DVD-R Five USB 2.0 ports (Two front, two rear, one internal) One x4 PCI Express slot, Four x8 PCI Express slots, One x16 PCI Express slot Four on-board Gigabit Ethernet ports (10/100/1000) Dual redundant hot-swappable AC 1050 W power supply units (N+N redundancy) Redundant, hot-swappable fan modules (N+1 redundancy) Form factor 1 rack unit (1U) 2 rack units (2U) 2 rack units (2U) a.the RAID HBA consumes a single PCI Express slot Features in all systems include: Support for multiple Dual-Core or Quad-Core AMD Opteron processors Large memory capacities (up to 128 GB in the Sun Fire X4440 server) Large-capacity internal storage, with support for either eight or 16 internal SAS disk drives RAID support provided with host bus adaptors PCI Express expandability Built-in quad Gigabit Ethernet support Integrated Lights Out Management (ILOM) service processor and firmware Support for multiple operating systems Multiple off-the-shelf configurations of each platform are available, along with a wide spectrum of options to tailor each system for specific workload requirements.

9 7 The Evolution of x64 Systems Sun Microsystems, Inc. A Choice of Operating Systems In order to provide maximum flexibility and investment protection, Sun Fire X4140, X4240, and X4440 servers support a choice of operating systems, including: The Solaris Operating System The Linux operating system (64-bit Red Hat or SuSE Linux) Microsoft Windows Server VMware ESX Server Table 2 lists supported OS releases supported by the Sun Fire X4140, X4240, and X4440 servers as of this writing. Please see sun.com/x64 for the latest supported operating systems and environments. Table 2. Supported OS releases for Sun Fire X4140, X4240, and X4440 servers Operating System Solaris OS Microsoft Windows Red Hat Linux SuSE Linux VMware Minimum Version Solaris 10 OS Update 4 (64-bit) Microsoft Windows Server 2003 Enterprise Edition, SP2, (32-bit/64-bit) Red Hat Enterprise Linux 4, Update 5 (32-bit/64-bit) Red Hat Enterprise Linux 5 (32-bit/64-bit) SuSE Linux Enterprise Server 10, SP1 (64-bit) VMware ESX (64-bit) or higher or ESX 3.5 Update 1 or higher (please see sun.com/software/vmware for the latest information on support of Quad-Core AMD Opteron processors) The Solaris Operating System Distributed under a commercial and open source licensing model, the Solaris 10 OS offers many innovative technologies that change the equation for organizations needing to reduce costs, minimize complexity, and help eliminate risk. The Solaris 10 OS is optimized for Sun systems and is supported on over one thousand third-party x86/ x64 systems. In addition, the Solaris 10 OS is free for download without requirement to purchase a support contract, providing an economic advantage over other communitybased operating system offerings. Taking advantage of the Solaris 10 OS can bring added flexibility and power to the enterprise. Supporting systems from laptops and single-board computers to datacenter and cluster installations, the Solaris 10 OS serves applications ranging from military command and control systems, to telecommunication switch gear, to stock trading. The Solaris 10 OS also includes more than 180 applications from the free and open source software (F/OSS) community, and thousands of others are freely available for download over the Internet. Provided on all Sun systems at no charge, the Solaris OS delivers performance, security, scalability, and reliability advantages for scale-out computing environments. Underlying technologies, such as a high-performance networking stack, advanced file

10 8 The Evolution of x64 Systems Sun Microsystems, Inc. system, and modern memory model combine to optimize the performance of hosted applications. A suite of security features previously only found in Sun s military-grade Trusted Solaris operating system are now included to fortify the commercial enterprise. The Solaris OS supports near linear scalability from 1 to 72 CPUs and addressability of up to 2 64 bytes of memory, well beyond the physical memory limits of even Sun s largest server. In addition, by providing the ability to automatically recover from hardware faults, the Solaris OS provides maximum data and application availability.systems. The Solaris 10 OS is free for download without requirement to purchase a support contract, offering an economic advantage over other communitybased operating system offerings. Linux Environments Sun offers and supports the leading Linux variants on Sun Fire x64 servers, including Red Hat Enterprise Linux and Novell SUSE Linux Enterprise Server. As the leader in enterprise services for UNIX, Sun brings decades of expertise to Linux environments. Sun support contracts for Linux provide all front-line support and transparent access to back-line support from Red Hat and Novell. Sun is one of the largest contributors to the open-source community. Areas of contribution include OpenOffice.org, Mozilla, GNOME, and X.org. In addition, Sun provides key software offerings for Linux including Lustre parallel file system Sun Ray Server Software Sun xvm software StarOffice productivity suite Java Desktop Powered Program Sun Studio, Sun Java Studio Creator, and NetBeans IDE software MySQL database Microsoft Windows Environments Organizations are constantly seeking to reduce the variety of platforms in the datacenter, even when a wide range of workloads are present. To help this effort, Sun Fire X4140, X4240, and X4440 servers can run the Microsoft Windows operating environment. These servers have passed stringent Microsoft compatibility test suites, achieving the Designed for Windows certification and a listing in Microsoft Windows catalogs. Support contracts for Microsoft Windows are also available from Sun. This certification and support demonstrates Sun s commitment to providing the best platforms to run not only the Solaris OS and Linux, but Microsoft Windows as well.

11 9 The Evolution of x64 Systems Sun Microsystems, Inc. VMware Ground-breaking virtualization solutions from VMware help improve asset utilization, operational efficiency, and business agility. Sun offers the VMware Infrastructure product suite on Sun hardware systems with full support from Sun. VMware virtualization technology also combines with key Solaris 10 OS features such as DTrace, Solaris Containers, and Solaris Predictive Self Healing software. As a result, organizations can create breakthrough approaches to virtualization. In fact, utilizing VMware virtual infrastructure software with the Solaris 10 OS for consolidation projects can increase system utilization by up to ten times. Innovative and Consistent System Design for the Datacenter Beyond the capabilities of individual systems, Sun understands that datacenters have unique and pressing needs that require attention on the part of system designers. Density, performance, and scalability are all essential considerations, but systems must also be serviceable and fit in with modern datacenter strategies that consider power, cooling, and serviceability. Sun Fire X4140, X4240, and X4440 servers share an innovative design philosophy that extends across Sun s volume x64 and SPARC server platforms. Principals of this philosophy include: Common chassis design Shared chassis design leverages key system innovations across multiple architectures, provides for common components and subassemblies, and greatly simplifies administration for those deploying multiple processor architectures. Maximum Compute Density Sun s volume servers provide leading density in terms of CPU cores, memory, storage and I/O. This focus on density often lets Sun s 1 rack unit (1 RU) rackmount servers replace competitive 2U rackmount servers, for a 50- percent space savings. Leading Storage Capacity Sun s volume servers provide leading density and flexible RAID options. Smaller disk drives and innovations in structure, airway, and carrier design allow more disk capacity in smaller spaces, while enhancing system airflow. Common, Shared Management Sun Fire X4140, X4240, and X4440 servers are designed for ease of management and serviceability with service processors shared by other Sun volume server platforms. Systems and components are designed for easy identification and hot-swap components facilitate on-line replacement. Continued Investment Protection Sun designs for maximum investment protection. Even with breakthrough x64 technology, Sun s Solaris Binary Compatibility Guarantee means that applications simply run without modification.

12 10 The Evolution of x64 Systems Sun Microsystems, Inc. Chassis Design Innovations Sun Fire X4140, X4240, and X4440 servers share basic chassis design with other Sun x64 and SPARC server platforms. This approach not only provides a consistent look and feel across the product line, but it simplifies administration through consistent component placement and shared components. Beyond mere consistency, this approach provides a datacenter design focus that places key technology where it can make a difference for the operations. Enhanced System and Component Serviceability Finding and identifying servers and components in a modern datacenter can be a challenge. Sun Fire X4140, X4240, and X4440 servers are optimized for lights-out datacenter configurations with easy-to-identify servers and modules. Color-coded operator panels provide straightforward diagnostics. Systems are designed for deployment in hot-isle / cold-isle multiracked deployments with both front and rear diagnostic LEDs and Fault Remind features to help identify faulty or failed components. Consistent connector layouts for power, networking, and management make moving between Sun s systems straightforward. All hot-plug components are toolless and easily available for serviceability. For instance, an integral hinged lid provides access to dual fan modules so that fans can be serviced without exposing sensitive components, or causing unnecessary downtime. Robust Chassis, Component, and Subassembly Design Sun s volume servers share chassis that are carefully designed to provide reliability and cool operation. Even features such as the hexagonal chassis ventilation holes are designed to provide the best compromise for high strength, maximum air flow, and maximum reductions in EMI emissions. Next-generation hard disk drive carriers share the hexagonal ventilation holes of the chassis and provide a seven-percent smaller front plate for greater storage density while increasing airflow to the system. A removable disk cage in each system plugs directly in front of the fan tray assemblies, allowing airflow to be directed both above and below disk drives, and then above and below memory DIMMs and mezzanine boards to efficiently cool the system. Dual fan modules are isolated from chassis to avoid transfer of rotational vibration to other system components. Also, integration of the fan power board into the Fan Tray assembly protects users from electrical shock during fan removal/insertion. In spite of their computational, I/O, and storage density, Sun s servers are able to maintain adequate cooling using conventional technologies. Efficient modular fan assemblies keep the chassis within an effective operating temperature range. Minimized DC-DC power conversions also contribute to overall system efficiency.

13 11 The Evolution of x64 Systems Sun Microsystems, Inc. By providing 12 volt power to the motherboard, power conversion stages are eliminated. This approach reduces generated heat, and introduces further efficiencies to the system. Minimized Cabling for Maximized Airflow To minimize cabling and increase reliability, a variety of smaller boards and riser cards are employed on both SPARC and x64 servers, appropriate to each chassis. These infrastructure boards serve various functions in the Sun Fire X4140, X4240, and X4440 servers. Power distribution boards (PDBs) distribute system power from the dual power supplies to the motherboard and to the disk backplane (via a connector board) Connector boards eliminate the need for many discrete cables, providing a direct card plug-in interconnect to distribute control and most data signals to the disk backplane, fan boards, and the PDB. Fan boards provide connections for power and control for both the primary and secondary fans in the front of the chassis. No cables are required since every dual fan module plugs directly into one of these PDBs which, in turn, plugs into the Connector Board. PCI Express riser cards plug directly into the motherboard, allowing PCI Express cards to be installed. The disk backplane mounts to the disk cages in the two chassis, delivering disk data through two 4-channel discrete mini-sas cables from the installed host bus adapter (HBA) card. An 8-disk backplane is offered for the Sun Fire X4140 and X4440 servers while the Sun Fire X4240 server supports a 16-disk backplane. Also provided via the disk backplane, are two USB connections to the front of the system.

14 12 AMD Opteron Processor Technology Sun Microsystems, Inc. Chapter 2 AMD Opteron Processor Technology Sun Fire X4140, X4240, and X4440 servers are powered by AMD Opteron processors, utilizing AMD s Direct Connect Architecture and NVIDIA chipsets for scalability and fast I/O throughput. These servers support both Dual-Core AMD Opteron processors as well as Third-Generation Quad-Core AMD Opteron processors. The sections that follow describe the architecture and feature set of AMD Opteron processors. Dual-Core AMD Opteron Processors Second-Generation AMD Opteron processors are native Dual-Core AMD Opteron processors that feature AMD s Direct Connect Architecture. These processors offer a common core architecture that is consistent across 1-socket, 2-socket, and 4-socket systems, and is also consistent with previous AMD Opteron processors. This strategy helps organizations minimize the cost of transitions while they maximize past investments in software and hardware optimization. AMD Opteron processors are offered in three series: 1000 Series Single socket 2000 Series Up to two sockets 8000 Series Four to Eight sockets Innovative Processor Technology The AMD Opteron processor extends the ubiquitous x86 architecture to accommodate 64-bit processing. Formerly known as x86-64, AMD s enhancements to the x86 architecture allow seamless migration to the superior performance of 64-bit technology. Dual-Core AMD Opteron processors offer considerable advantages, including: AMD64 technology AMD64 technology lets 64-bit operating systems provide full, transparent, and simultaneous 32-bit and 64-bit platform application multitasking. This approach lets systems run the existing installed-base of 32-bit applications and operating systems at peak performance, while providing a 64-bit migration path. Direct Connect Architecture AMD s Direct Connect Architecture helps to reduce the very real challenges and bottlenecks of system architecture. Memory is directly connected to the processor, optimizing memory performance I/O is directly connected to the processor, for more balanced throughput and I/O Processors are directly connected to other processors, allowing for more linear symmetrical multiprocessing Integrated DDR2 memory controller A 144-bit wide, on-chip DDR2 memory controller provides 128 bits for data and 16 bits for ECC and Enhanced ECC technologies, while providing low-latency memory bandwidth that scales as processors are added.

15 13 AMD Opteron Processor Technology Sun Microsystems, Inc. AMD HyperTransport technology AMD HyperTransport Technology provides a scalable bandwidth interconnect between processors, I/O subsystems, and other chipsets. Quad-Core upgradeability AMD Opteron processors with DDR2 memory are designed to offer a seamless upgrade path from dual-core to quad-core processors. Similar power and thermal envelops help protect investments, letting organizations upgrade to Quad-Core AMD Opteron processors while realizing similar power efficiencies. AMD Virtualization (AMD-V) AMD Virtualization reduces overhead by selectively intercepting instructions destined for guest environments while the Direct Connect Architecture helps guest operating systems run at near native speed. A virtualizationaware integrated memory controller provides efficient isolation of virtual machine memory. Enhanced performance per watt Energy-efficient DDR2 memory uses up to 30 percent less power than DDR1 memory, and up to 58 percent less power than FB- DIMM memory. In addition, AMD PowerNow! technology with Optimized Power Management can deliver performance on demand, while minimizing power consumption. Dual-Core Processor Architecture The AMD Opteron processor (Figure 2) was designed from the start for multicore functionality, with a crossbar switch and system request interface. This approach defines a new class of computing by combining full x86 compatibility, a highperformance 64-bit architecture, and the economics of an industry-standard processor. Second-Generation Dual-Core AMD Opteron Core 1 Core KB L1 Cache 128 KB L1 Cache 1MB L2 Cache 1MB L2 Cache System Request Interface Crossbar Switch DDR2 Memory Controller HyperTransport 0 HyperTransport 1 HyperTransport 2 Figure 2. High-level architectural perspective of a Dual-Core AMD Opteron processor Enhancements of the AMD Opteron processor over the legacy x86 architecture include: bit general-purpose integer registers that quadruple the general-purpose register space available to applications and device drivers as compared to x86 systems

16 14 AMD Opteron Processor Technology Sun Microsystems, Inc bit XMM registers provide enhanced multimedia performance to double the register space of any current SSE/SSE2 implementation A full 64-bit virtual address space offers 40 bits of physical memory addressing and 48 bits of virtual addressing that can support systems with up to 256 terabytes of physical memory Each processor core has a dedicated 1 MB Level-2 cache, and both cores use the System Request Interface and Crossbar Switch to share the Memory Controller and access the three HyperTransport links. This sharing represents an effective approach since performance characterizations of single-core based systems have revealed that the memory and HyperTransport bandwidths are typically under-utilized, even while running high-end server workloads. The Second-Generation AMD Opteron processor integrates three HyperTransport technology links, providing a scalable bandwidth interconnect among processors, I/O subsystems, and other chip-sets. HyperTransport technology interconnects help increase overall system performance by removing I/O bottlenecks and efficiently integrating with legacy buses, increasing bandwidth and speed, and reducing processor latency. At 16 x 16 bits and 1 GHz operation, HyperTransport technology provides support for up to 8 GB/s bandwidth per link. HyperTransport Technology HyperTransport technology is a high-speed, low-latency, point-to-point link designed to increase the communication speed between integrated circuits in computers, servers, embedded systems, and networking and telecommunications equipment. Second- Generation and Third-Generation AMD Opteron processors continue to use HyperTransport technology links to provide a scalable bandwidth interconnect among processors, I/O subsystems, and other chip sets. HyperTransport technology: Helps increase overall system performance by removing I/O bottlenecks typically found in Front Side Bus (FSB) architectures, efficiently integrating with legacy buses, increasing bandwidth and speed, and reducing latency of processors. Provides up to 8 GB/second bandwidth per link at 16 x 16 bits, 1 GHz operation, offering significantly more bandwidth than most current technologies, and sufficient bandwidth for supporting new interconnects such as PCI Express Uses low-latency responses and low pin counts for enhanced performance and reliability Maintains compatibility with legacy PC buses while being extensible to new Systems Network Architecture (SNA) buses Appears transparent to operating systems, so that peripheral drivers continue to operate

17 15 AMD Opteron Processor Technology Sun Microsystems, Inc. Quad-Core AMD Opteron Processors Unlike multichip package technology (MCP), native Quad-Core AMD Opteron processors incorporate four processor cores on a single silicon die. Despite this innovation, Quad- Core AMD Opteron processors are electrically, thermally, and socket-compatible with Second-Generation AMD Opteron Socket F (1207) processors. Figure 3 provides a blocklevel diagram of the third-generation Quad-Core AMD Opteron processor. Third-Generation Quad-Core AMD Opteron Core 1 Core 2 Core 3 Core KB L1 L2 Cache 512 KB L2 Cache 128 KB L1 Cache 512 KB L2 Cache 128 KB L1 Cache 512 KB L2 Cache 2 MB L3 Cache System Request Interface Crossbar Switch 128 KB L1 Cache 512 KB L2 Cache DDR2 Memory Controller HyperTransport 0 HyperTransport 1 HyperTransport 2 Figure 3. Third-Generation Quad-Core AMD Opteron processor block-level diagram Quad-Core AMD Opteron processors go beyond simply adding two additional cores. With a native multicore design, all four cores share the same silicon, and are directly connected via AMD s Direct Connect Architecture. Processors, I/O, and memory controller logic are all connected to each other, aiding performance and reducing bottlenecks. Quad-Core AMD Opteron processors provide a broad set of significant enhancements, described in the sections that follow. Enhanced AMD PowerNow! Technology Enhanced AMD PowerNow! technology provides significant power-management advancements. AMD CoolCore technology can reduce energy consumption and heat generation by turning off unused parts of the processor. Independent Dynamic Core technology allows each core to vary its clock frequency depending on the specific performance requirements of the applications it is supporting, helping to reduce power consumption. Dual Dynamic Power Management (formerly called splitplane ) provides an independent power supply to the cores and to the memory controller, allowing them to operate on different voltages, depending on usage.

18 16 AMD Opteron Processor Technology Sun Microsystems, Inc. Investment Protection Not only are Quad-Core AMD Opteron processors the first native x86 quad-core processors, but they are the first quad-core processors designed to operate within similar thermal and power envelops as AMD s current Second-Generation processors. This consistency allows simplified upgradeability and protects organizations investments in AMD Opteron based systems with a seamless upgrade path. Virtualization Enhancements Virtualization is memory intensive, and Quad-Core AMD Opteron processors provide exceptional memory throughput with an integrated memory controller. AMD Virtualization introduces Rapid Virtualization Indexing (formerly called nested page tables ) and a tagged translation look-a-side buffer (TLB). While TLBs exist in almost every processor architecture, AMD implemented tagged TLBs to improve virtual to physical memory lookups from one virtual machine to another. AMD s Rapid Virtualization Indexing feature is designed to reduce the overhead penalty associated with virtualization technologies by moving the process of managing virtual memory from software to hardware. This approach reduces the complexity of existing x86 virtualization solutions and facilitates increased performance and efficiency for many virtualized workloads. Support for High Performance Computing (HPC) A variety of features coalesce in Quad-Core AMD Opteron processors to make them ideal for HPC workloads. AMD Memory Optimizer Technology increases memory throughput by up to 50 percent compared to previous generations of the AMD Opteron processor. AMD Wide Floating Point Accelerator provides 128-bit Streaming SIMD Extensions (SSE) floating point capabilities, letting each core simultaneously execute up to four floating point operations (FLOPS) per clock four times the floating-point computations of previous-generation AMD opteron processors. AMD Balanced Smart Cache provides significant cache enhancements with 128 KB of Level-1 cache, and 512 KB of Level-2 cache per core, combined with 2 MB of shared Level-3 cache shared across all cores. Processor Design for Energy Efficiency Power consumption continues to be one of the top concerns for managing today's datacenters. Quad-Core AMD Opteron processors address this concern by providing industry-leading overall power-efficiency that can deliver significant performance gains over Dual-Core AMD Opteron processors while operating in the same thermal envelope. All AMD Opteron processor series current Single-Core, Dual-Core, Quad-Core, and future AMD Opteron processors have been designed to a consistent power and thermal specification.

19 17 AMD Opteron Processor Technology Sun Microsystems, Inc. Average CPU Power (ACP) Average CPU Power (ACP) is a metric that offers a relevant estimation of the power consumption for AMD Opteron processors. ACP is determined by breaking down multiple components of the power consumed within the processor, including the power dedicated to the cores, the integrated memory controller, and to HyperTransport technology links. In contrast, thermal design power (TDP) refers to the power that processors are capable of consuming, and is the specification that system designers typically follow. ACP and TDP are both valid indicators of processor power. AMD has referenced processor power consumption based on TDP values to date. However, ACP represents a relevant measure that reflects power consumption while running server-class enterprise workloads. In particular, ACP is a useful metric for datacenter operators to use when estimating power budgets to size their datacenters, while TDP is more useful and relevant to system designers. Table 3 illustrates both TDP and ACP for Quad-Core AMD Opteron processors. Table 3. TDP and ADP for Quad-Core AMD Opteron processors AMD Opteron Processor Low Power CPU Modules (HE) Standard Power Modules Performance Optimized Power (SE) Quad-core processors TDP 75 W 115 W 137 W ACP 55 W 75 W 105 W Enhanced AMD PowerNow! and Independent Dynamic Core Technology Enhanced AMD PowerNow! Technology is designed to reduce power consumption of the entire quad-core processor. The native quad-core design of Third-Generation AMD Opteron processors lets enhanced power management address each of the four cores independently. Independent Dynamic Core technology allows each core to vary its frequency, based on the specific needs of the application. This ability allows for more precise power management to reduce datacenter energy consumption and thereby reduce total cost of ownership (TCO). Power consumption is related to the voltage level of the voltage supply to the processor as well as the frequency of operation. General purpose systems are designed to operate at a voltage level and frequency level that meets their peak computational performance. Unfortunately, this level of operation can consume significant amounts of power, especially when peak processor performance is not required. Power is typically saved by reducing the supply voltage of the processor when peak performance is not needed. With this approach, the sections of the processor which are unused have the clock frequency reduced which reduces power consumption.

20 18 AMD Opteron Processor Technology Sun Microsystems, Inc. As shown in Figure 5, the core frequency with the Dual-Core AMD Opteron processor is locked based on the load characteristics of Core 0. Core 1 will operate at the same core frequency even though it's load characteristics are low. With Independent Dynamic Core Technology, the native quad-core processor can operate each of the cores at different frequencies based on the load characteristics of that particular core. Dual-Core Native Quad-Core Idle 75% 1% MHz Idle MHz Idle 75% 35% Core 0 MHz Idle Core 1 MHz Core 0 Core 1 Idle 10% MHz Idle 1% Core 2 Core 3 MHz MHz and Voltage are locked to the highest utilized core s p-state. MHz is independently adjusted separately per core. Figure 4. Independent Dynamic Core technology adjusts frequency on a per-core basis Dual Dynamic Power Management Dual Dynamic Power Management (formally splitplane ) allows each processor to maximize the power-saving benefits of AMD PowerNow! technology without compromising performance. Dual Dynamic Power Management can reduce idle power consumption and allow for per-processor power management in multisocket systems to decrease power consumption. Figure 5 illustrates a Quad-Core AMD Opteron processor powered by both a conventional unified power supply, as well as powered from independent voltage supplies. Core 1 Core 2 Core 3 Core 4 2 MB L3 Cache CPU Power Core 1 Core 2 Core 3 Core 4 2 MB L3 Cache Unified Power Supply System Request Interface Crossbar Switch System Request Interface Crossbar Switch DDR2 Memory Controller HyperTransport 0 HyperTransport 1 HyperTransport 2 Memory Controller Powerr DDR2 Memory Controller HyperTransport 0 HyperTransport 1 HyperTransport 2 Figure 5. With Dual Dynamic Power Management system motherboards can deliver separate power for the processor cores and memory controller

21 19 AMD Opteron Processor Technology Sun Microsystems, Inc. The illustration shows that power for the CPU and Memory Controller on Third- Generation AMD Opteron processors can be powered from independent voltage supplies if supported by the system motherboard offering greater performance and better power management. These same processors can also run in legacy systems with a unified power supply. Second-Generation AMD Opteron processors use a unified voltage plane for the memory and processor cores. These processors are still compatible with motherboards designed to support Dual Dynamic Power Management but they will deliver the same voltage to the CPU and Memory Controller power. High-Bandwidth Chip-Level I/O for High Performance Computing Implementing a multicore processor is only one part of providing fast and reliable performance. In addition to computational performance, high performance computing and other demanding applications require the ability to move data between processors, memory, and I/O with a minimum of bottlenecks. Third-Generation Quad-Core AMD Opteron processors are designed to provide high-bandwidth chip-level interconnects to other processors, memory, and system I/O. Multiple HyperTransport links connect between multiple processors and to system I/O bridges. Integrated memory controllers provide fast low-latency access to memory. HyperTransport Dual Link Hypertransport Dual Link refers to two-socket or four-socket system configurations in which the processors are connected by a pair of HyperTransport Technology links. Each link represents a HyperTransport path that runs at speeds up to 1 GHz for up to 8 GB/s of theoretical bandwidth between each processor and each processor's attached controllers. The Quad-Core AMD Opteron processor has one (2000 Series) or three (8000 Series) coherent Hypertransport links that allow each processor to access another processor s memory. HyperTransport Dual Link works by ganging a coherent and non-coherent HyperTransport link together. With HyperTransport Dual Link the peak available bandwidth between the two processors doubles to 16 GB/s which can provide up to three to five percent better system performance. Integrated Memory Technology AMD Opteron processors integrate a DDR memory controller directly into the processor. The memory controller runs close to the processor s core frequency and greatly increases bandwidth to the processor at significantly reduced latencies. The performance-enhancing effect is even more dramatic within multisocket AMD Opteron systems, because each additional processor has its own memory controller, allowing memory bandwidth to scale within the server.

22 20 AMD Opteron Processor Technology Sun Microsystems, Inc. AMD Opteron processors are designed to work with Double Data Rate (DDR) SDRAM. Similar to first-generation DDR memory, DDR2 memory cells transfer data both on the rising and falling edge of the clock (a technique called dual pumping ). The key difference between DDR and DDR2 is that in DDR2 the bus is clocked at twice the speed of the memory cells, so four words of data can be transferred per memory cell cycle. As a result, DDR2 can effectively operate at twice the bus speed of DDR, without speeding up the memory cells themselves. Figure 3 illustrates the Quad-Core AMD Opteron processor architecture featuring the cache controller and three stages of caches. The dedicated per-core 128 KB L1 cache provides a 64 KB instruction cache and a 64KB for data, and is capable of delivering two data loads per cycle instead of one load per cycle of competing x86 processors. The latency for the L1 cache is three clock cycles with very fast access time. The quad-core architecture also features a dedicated per-core 512 KB L2 cache to eliminate conflicts common in shared caches. These caches are 16-way set associative, and the latency for each core to retrieve data from its L2 cache is 12 clock cycles. A large, 2 MB L3 cache is shared between all processor cores in Quad-Core AMD Opteron processors. The L3 cache is 32-way set associative and is based on a non-inclusive victim cache architecture. The latency for any core to retrieve data from the L3 cache is less than 38 clock cycles. The L2 cache was designed for those applications that are running on a single core and consume most or all of the 2 MB L3 cache. This situation can cause a problem on other processor architectures that do not have three levels of cache, since the shared cache can be busy serving one core while the others are starved. In AMD Opteron processors, even if one thread is consuming the L3 cache, other threads run effectively from the core s L2 cache, which is sized to accommodate the majority of modern working sets. AMD Virtualization Technology Virtualization technology lets organizations achieve higher levels of efficiency, utilization, and flexibility by dividing a given system into several virtual machines allowing the consolidation of many legacy systems onto one physical machine. AMD's Virtualization (AMD-V) technology provides an enhanced AMD Opteron instruction set that subsumes some tasks that virtual machine managers (VMMs) typically perform through software emulation. Quad-Core AMD Opteron processors with Direct Connect Architecture help enable industry leading virtualization platform efficiency. Featuring AMD-V technology with Rapid Virtualization Indexing, Quad-Core AMD Opteron processors can accelerate the performance of virtualized applications and improve the efficiency of switching among virtual machines. This feature allows organizations to host more virtual machines and users per system to maximize the consolidation and power-saving benefits of virtualization.

23 21 AMD Opteron Processor Technology Sun Microsystems, Inc. Third-Generation Quad-Core AMD Opteron processors offer enhancements to AMD-V that provide a balanced approach to improve virtualization performance and help enable near-native performance for virtualized applications. AMD Opteron processors also provide silicon feature-set enhancements that are designed to improve performance, reliability, and security of existing and future virtualization environments to support more users. Some of the AMD-V enhancements that are built into the Third- Generation AMD Opteron architecture include: Direct Connect Architecture to Host More Virtual Machines (VMs) Per Server AMD s Direct Connect Architecture helps improve application performance within a virtual machine. This architecture provides direct CPU-to-memory, CPU-to-I/O, and CPU-to-CPU connections to streamline server virtualization. The Integrated Memory Controller is designed to improve performance on memory-intensive virtualization environments through high bandwidth, low latency, and scalable access to memory. HyperTransport technology optimizes the movement of data and the sharing of resources among VMs and I/O for greater system scalability. Tagged Translation Look-aside Buffer for Increased Responsiveness Unique to AMD Opteron processors, the Tagged Translation Look-aside Buffer (TLB) allows for faster switching times between virtual machines by maintaining a mapping to the VM s individual memory spaces. Competing solutions cannot distinguish one VM's memory space from another s, resulting in additional memory management overhead and reduced responsiveness when switching between virtual machines. Device Exclusion Vector (DEV) for More Efficient Security Device Exclusion Vector (DEV) performs security checks in hardware, protecting memory access to un-authorized requests from external devices. The DEV controls access to virtual machine memory based on permission, isolating virtual machines for secure operation. The DEV performs these security checks in hardware, rather than software resulting in efficiency. The DEV creates Protection Domains that deny memory access for unauthorized requests from external devices, such as hard disks, network controllers, and other devices. Rapid Virtualization Indexing for Better Performance in a Virtualization Environment Rapid Virtualization Indexing is an enhancement to AMD-V technology in Quad- Core AMD Opteron processors. This feature is designed to dramatically increase the performance of virtualized applications while providing faster switching between virtual machines. Rapid Virtualization Indexing allows users to host more VMs per server and maximize the benefits of virtualization. This feature must be supported in the virtualization software.

24 22 Server Architecture Sun Microsystems, Inc. Chapter 3 Server Architecture Sun Fire X4140, X4240, and X4440 servers are designed to provide high performance with high reliability and low power consumption. The sections that follow detail physical and architectural aspects of the systems, highlighting similarities and differences between the three server platforms. System-Level Architecture Sun Fire X4140, X4240, and X4440 servers all employ the same motherboard, but offer different functionality depending on the system, as determined by both chassis and supported system modules. Though they occupy different chassis, the Sun Fire X4140 and Sun Fire X4440 both share a similar system architecture, illustrated in Figure 6. Shaded boxes in the diagram indicate functionality only provided on Sun Fire X4440 servers. DDR2 DDR2 Sun Fire X4440 Server only (8) Disk Backplane (4) SAS Lanes (4) SAS HDDs (4) SAS Lanes (4) SAS HDDs 10.7 GB/s 10.7 GB/s USB 2.0 USB to IDE DVD/CD/RW CPU-3 CPU-1 IO55 4x 1Gb Ethernet CPU GB/s Hypertransport 8GB/s CPU GB/s MCP55 USB 2.0 Electrical Lanes (4) SAS Lanes Cable USB Hub x4 x8 x8 x16 x8 x8 PCI Express Slot 5 x8 Slot 4 x8 Slot 3 x8 Slot 2 x16 Slot 1 x8 Slot 0 x8 RAID card USB 2.0 Rear Rear Front Front Internal DDR2 Mezzanine Card DDR2 USB 2.0 PCI AST2000 Q GP 0608 TAN A2 Serial RJ-45 Management 10/100 Ethernet VGA Video Figure 6. Sun Fire X4140 and X4440 motherboard block-level diagram Sun Fire X4140 servers support one or two AMD Opteron processors and up to 16 DDR2 DIMM slots. Sun Fire X4440 servers support up to four AMD Opteron processors with the addition of a mezzanine card that connects via dual Hypertransport links. As shown in

25 23 Server Architecture Sun Microsystems, Inc. the illustration, the Sun Fire X4440 server provides HyperTransport Dual Link technology on the optional mezzanine card. Both the Sun Fire X4140 and X4440 servers provide an eight-disk backplane driven by a RAID host bus adapter (HBA) expansion card that occupies one of the PCI Express slots. Two SAS cables with four lanes each connect form the RAID HBA card to the eight-disk backplane. Figure 8 illustrates the Sun Fire X4240 server block-level diagram. Like the Sun Fire X4140 server, the Sun Fire X4240 provides support for up to two AMD Opteron processors. The two rack-unit chassis allows deployment of a 16-disk backplane that attaches to a RAID HBA card residing in one of the PCI Express slots. DDR2 + (16) Disk Backplane (4) SAS Lanes 28 Port + (4) SAS Lanes SAS Expander (4) SAS Lanes (4) SAS HDDs (4) SAS HDDs (4) SAS HDDs + + (4) SAS Lanes (4) SAS HDDs 10.7 GB/s CPU-1 USB 2.0 USB to IDE DVD/CD/RW IO55 4x 1Gb Ethernet CPU GB/s Hypertransport 8GB/s MCP55 USB 2.0 Electrical Lanes (4) SAS Lanes Cable USB Hub x4 x8 x8 x16 x8 x8 PCI Express Slot 5 x8 Slot 4 x8 Slot 3 x8 Slot 2 x16 Slot 1 x8 Slot 0 x8 RAID Card USB 2.0 Rear Rear Front Front Internal DDR2 USB 2.0 PCI AST2000 Q GP 0608 TAN A2 Serial RJ-45 Management 10/100 Ethernet VGA Video Figure 7. Sun Fire X4240 motherboard block-level diagram Memory Subsystem The AMD Opteron processor s integrated DDR2 memory controller improves the way that typical x64 processors access main memory, resulting in increased bandwidth, reduced memory latencies, and increased processor performance. The dual-channel DDR2 memory controller on each processor is capable of yielding a memory bandwidth of 10.7 GB per second (two 5.35 GB per second channels). Each AMD Opteron processor

26 24 Server Architecture Sun Microsystems, Inc. supports up to eight registered DDR2-667 DIMMs. Sun Fire X4140 and X4240 servers support up to 64 GB of memory, and Sun Fire X4440 servers support up to 128 GB of memory. 1 GB, 2 GB, and 4 GB DIMMs are available and supported. I/O Subsystem The I/O subsystem in Sun Fire X4140, X4240, and X4440 servers is designed to provide maximum throughput and flexibility to suite a wide variety of server needs. NVIDIA nforce Pro 3600 (MCP55) Chipset In Sun Fire X4140, X4120, and X4440 servers, one AMD Opteron processor connects to an NVIDIA nforce Pro 3600 (MCP55) chipset via a 1 GHz HyperTransport connection. The servers utilize the NVIDIA nforce Pro 3600 chipset to provide a wealth of system I/O. The MCP55 architecture facilitates full-featured motherboards that provide maximum performance and low latency, along with lower levels of power consumption and heat dissipation. The first CPU (CPU 0) connects with the chipset across a 1 GHz HyperTransport link. The NVIDIA nforce Pro 3600 chipset provides: Three PCI Express interfaces that extend to three of the PCI Express slots (one 16-lane and two 8-lane interfaces 1 Dual IEEE MACs for two 10/100/1000Base-T gigabit auto-negotiating Ethernet interfaces Five USB 2.0 ports LPC bus interface connecting to the AST2000 and BIOS boot flash One 32-bit 33 MHz PCI bus connecting to the AST2000 Integrated SATA 3.0 Gigabit per second controllers System power sequencing and monitoring of many of the systems power status indicators NVIDIA nforce Pro 3050 (IO-55) Chipset The NVIDIA nforce Pro 3050 (IO-55) chipset augments the I/O capabilities of Sun Fire X4140, X4240, and X4440 servers, connecting to CPU 1 via another dedicated 16-bit HyperTransport interconnect operating at a 1 GHz clock rate. 1.Sun Fire X4140 servers provide three PCI Express slots total, comprised of one x16 port from the NVIDIA chip and two x8 ports from the nvidia MCP55 chip. One physical PCI Express slot on all systems is dedicated to house a low-profile RAID HBA card.

27 25 Server Architecture Sun Microsystems, Inc. The NVIDIA nforce Pro 3050 chipset provides: Three PCI Express interfaces to low profile PCI Express slots (one 16-lane, one 8-lane, and one 4-lane) Dual IEEE MACs for two 10/100/1000Base-T gigabit auto-negotiating Ethernet interfaces Integrated SATA 3.0 Gigabit per second controllers I/O and RAID Options Up to eight or inch disk drives are supported per system, depending on the server selected. The same disk drives can be used on Sun Fire X4140, X4240, and X4440 servers as on Sun s latest x64 and SPARC servers. All of these drives share a next-generation hard drive carrier design that is optimized for both airflow and density. Drives insert into a modular disk tray and cable-free disk backplane that increases reliability and serviceability. In all three servers, SAS disks are controlled by a host bus adapter (HBA) that occupies one of the systems x8 PCI Express slots. A choice of host bus adapters is provided, both running easy-to-use Sun StorageTek RAID Manager software. Options include: The Sun StorageTek SAS RAID Host Bus Adapter, Internal an 8-channel HBA card that supports 3 Gb/second SAS and RAID levels 0, 1, 10, 1E, 5, 6, 5EE, and 50 The Sun StorageTek PCI Express SAS Host Bus Adapter, Internal (LSI 3081E) a HBA card that supports RAID levels 0, 1, and 1EE Two SAS cables with four lanes each connect to the disk backplane to control the disk drives. In the case of the Sun Fire X4240, the four SAS links connect to a SAS expander in the 16-disk backplane (the LSISASX28). The SAS expander then provides the 16 SAS links to the individual disks in the drive tray. Sun Fire X4140 Server The compact Sun Fire X4140 server provides significant computational power in a spaceefficient low-power 1U rackmount package. With high levels of price/performance and a low acquisition cost, this server is ideally suited to the delivery of horizontally-scaled transaction and Web services, and can function as a very capable HPC compute node. The server is designed to address the challenges of today's datacenter with greatly reduced power consumption and a small physical footprint. Enclosure The 1U Sun Fire X4140 server enclosure is designed for use in a standard 19-inch rack (Table 4).

28 26 Server Architecture Sun Microsystems, Inc. Table 4. Dimensions and weight of the Sun Fire X4140 server Dimension U.S. International Height inches (1 RU) 44 millimeters Width (not including ears) inches 426 millimeters Depth (not including PSU handle) Weight (approximate, without PCI Express cards or rackmount slide rail kit) inches 714 millimeters pounds kilograms The Sun Fire X4140 server includes the following major components: One or two Dual-Core or Quad-Core AMD Opteron processors From 2 GB up to 64 GB of memory in up to 16 DDR2 DIMM slots (1 GB, 2 GB, and 4 GB DDR2 DIMMs supported) Four on-board 10/100/1000 Mbps Ethernet ports Three internal MD2 Low Profile PCI Express slots (one x16 and two x8 slots) Five USB 2.0 ports (two forward facing, two rear facing, one internal) Up to eight small form factor (SFF) SAS hot-swappable 2.5-inch internal SAS disk drives with add-on SAS Host Bus Adapter (diskless configurations also available) Integrated Lights out Management (ILOM) system controller Dual redundant (N+N) hot-swappable high-efficiency 650 watt power supply units running at 8.2 Amps RMS at 100 VAC Seven fan assemblies (each with two fans), under environmental monitoring and control, N+1 redundancy, accessed through a dedicated top panel door

29 27 Server Architecture Sun Microsystems, Inc. Front and Rear Perspectives Figure 8 illustrates the front and rear panels of the Sun Fire X4140 server. System status indicators Disk drive numbering map DVD+/-RW drive USB ports Hard disk drives Component status indicators Redundant (N+N) power supply units PCI Express slots System status indicators Serial and network Management ports 10/100/1000 Ethernet ports USB ports Video port (HD-15) Figure 8. Sun Fire X4140 server, front and rear panels External features of the Sun Fire X4140 server include: Front and rear system and component status indicator lights provide locator (white), service required (amber), and activity status (green) for the system A disk drive numbering map for easy access Eight hotplug SAS disk drives insert through the front panel of the system One slimline, slot-accessible DVD+/-RW is accessed through the front panel Five USB 2.0 ports are provided, two on the front panel, two on the rear, and one internally Two hotplug/hotswap (N+N) power supplies with integral fans insert from the rear Rear power-supply indicator lights convey the status of each power supply A single AC plug is provided on each hotplug/hotswap power supply Four 10/100/1000Base-T autosensing Ethernet ports are provided An HD-15 video port is provided A total of three PCI Express card slots are provided Two management ports are provided for use with the ILOM system controller (The RJ-45 serial management port provides the default connection to the ILOM controller while the network management port supports an optional RJ-45/10/100Base-T connection to the ILOM system controller.)

30 28 Server Architecture Sun Microsystems, Inc. Sun Fire X4240 Server The expandable Sun Fire X4240 server is optimized to deliver a dual-socket Web and virtualization platform with maximized expansion capabilities. In addition to the capabilities of the Sun Fire X4140 server, the Sun Fire X4240 adds support for additional PCI Express expansion and support for up to 16 disk drives. Enclosure The Sun Fire X4240 server features a compact yet expandable 2U rackmount chassis (Table 5), giving organizations the flexibility to scale their I/O and storage needs without wasting precious space. Table 5. Dimensions and weight of the Sun Fire X4240 server Dimension U.S. International Height 3.49 inches (1 RU) 88 millimeters Width (not including ears) inches 426 millimeters Depth (not including PSU handle) Weight (approximate, without PCI Express cards or rackmount slide rail kit) 28 inches millimeters pounds kilograms The Sun Fire X4240 server includes the following major components: Two Dual-Core or Quad-Core AMD Opteron processors From 2 GB up to 64 GB of memory in up to 16 DDR2 DIMM slots (1 GB, 2 GB, and 4 GB DDR2 DIMMs supported) Four on-board 10/100/1000 Mbps Ethernet ports Six internal MD2 Low Profile PCI Express slots including one x16, four x8, and one x4 (x8 physically) Five USB 2.0 ports (two forward facing, two rear facing, one internal) Up to 16 SFF SAS hot-swappable 2.5-inch internal SAS disk drives with add-on SAS Host Bus Adapter (diskless configurations also available) Integrated Lights out Management (ILOM) system controller Dual redundant (N+N) hot-swappable high-efficiency 1050 watt power supply units running at 8.2 Amps RMS at 100 VAC Six fan assemblies (each with two fans), under environmental monitoring and control, N+1 redundancy, accessed through a dedicated top panel door

31 29 Server Architecture Sun Microsystems, Inc. Front and Rear Perspectives Figure 9 illustrates the front and back panels of the Sun Fire X4240 server. Component status indicators System status indicators DVD+/-RW Drive USB ports Disk drives Disk drive numbering map Redundant (N+N) Power supply units PCI Express slots System status indicators Serial and network Management ports 10/100/1000 Ethernet ports USB ports Video port (HD-15) Figure 9. Sun Fire X4240 server, front and rear panels External features of the Sun Fire X4240 server include: Front and rear system and component status indicator lights provide locator (white), service required (amber), and activity status (green) for the system A disk drive numbering map for easy access Up to 16 hotplug SAS disk drives insert through the front panel of the system One slimline, slot-accessible DVD+/-RW is accessed through the front panel Five USB 2.0 ports (two forward facing, two rear facing, one internal) Two hotplug/hotswap (N+N) power supplies with integral fans insert from the rear Rear power-supply indicator lights convey the status of each power supply A single AC plug is provided on each hotplug/hotswap power supply Four 10/100/1000Base-T autosensing Ethernet ports are provided An HD-15 video port is provided A total of six PCI Express card slots are provided on the rear panel Two management ports are provided for use with the ILOM system controller (The RJ-45 serial management port provides the default connection to the ILOM controller while the network management port supports an optional RJ-45/10/100Base-T connection to the ILOM system controller.)

32 30 Server Architecture Sun Microsystems, Inc. Sun Fire X4440 Server The expandable Sun Fire X4440 server offers four sockets for AMD Opteron processors and up to 128 GB of memory, making it ideal as a scalable datacenter compute engine. Well suited as a database server and for consolidation of multiple less capable servers, Sun Fire X4440 servers combine processor scalability with ample memory, I/O expansion, and disk storage capacity. The Sun Fire X4440 server is also ideal for HPC environments where delivering high-density floating point performance is critical. Enclosure The Sun Fire X4440 server is delivered in a compact 2U rackmount chassis (Table 6). Table 6. Dimensions and weight of the Sun Fire X4440 server Dimension U.S. International Height 3.46 inches (2 RU) millimeters Width (not including ears) inches millimeters Depth (not including PSU handle) 28 inches millimeters Depth (with PSU handle) inches millimeters Weight (approximate, with PCI Express cards but without rackmount slide rail kit) 63 pounds 28.6 kilograms The Sun Fire X4440 server includes the following major components: Two or four Dual-Core AMD Opteron processors or Quad-Core AMD Opteron processors Up to 128 GB of memory in up to 16 DDR2 DIMM slots (1 GB, 2 GB, and 4 GB DDR2 DIMMs are supported) Four on-board 10/100/1000 Mbps Ethernet ports Six internal MD2 Low Profile PCI Express slots including one x16, four x8, and one x4 slot (x8 physically) Five USB 2.0 ports (two forward facing, two rear facing, one internal) Up to eight SFF SAS hot-swappable 2.5-inch internal SAS disk drives with add-on SAS Host Bus Adapter (diskless configurations also available) Integrated Lights out Management (ILOM) system controller Dual redundant (N+N) high-efficiency 1,050 watt power supply units (hot swappable in redundant configuration) Six fan assemblies (each with two fans), under environmental monitoring and control, N+1 redundancy, accessed through a dedicated top-panel door

33 31 Server Architecture Sun Microsystems, Inc. Front and Rear Perspectives Figure 9 illustrates the front and back panels of the Sun Fire X4440 server. System status indicators Component status indicators DVD+/-RW Drive USB ports Disk drives Disk drive numbering map Redundant (N+N) Power supply units PCI Express slots System status indicators Serial and network Management ports Figure 10. Sun Fire X4440 server, front and rear panels 10/100/1000 Ethernet ports USB ports Video port (HD-15) External features of the Sun Fire X4440 server include: Front and rear system and component status indicator lights provide locator (white), service required (amber), and activity status (green) for the system A disk drive numbering map for easy access Up to eight hotplug SAS disk drives insert through the front panel of the system One slimline, slot-accessible DVD+/-RW accessed through the front panel Five USB 2.0 ports, two on the front panel, two on the rear, and one internal Two hotplug/hotswap (N+N) power supplies with integral fans insert from the rear Rear power-supply indicator lights that convey the status of each power supply A single AC plug provided on each hotplug/hotswap power supply Four 10/100/1000Base-T autosensing Ethernet ports An HD-15 video port A total of six PCI Express card slots are on the rear panel Two management ports for use with the ILOM system controller (The RJ-45 serial management port provides the default connection to the ILOM controller while the network management port supports an optional RJ-45/10/100Base-T connection to the ILOM system controller.)

34 32 Enterprise-Class Operating System and Management Software Sun Microsystems, Inc. Chapter 4 Enterprise-Class Operating System and Management Software Unlike many x64 systems, Sun Fire X4140, X4240, and X4440 servers are ideally suited for enterprise environments. Not only are the systems designed with the datacenter in mind, but the operating system and management software provided with these systems allow them to serve the most important and mission-critical applications. The sections that follow describe the Solaris 10 OS and key enterprise-grade management technology. Solaris 10 OS Support Among the available operating systems, the Solaris OS is ideal for large-scale enterprise deployments. Supported on all of Sun s x64 and SPARC platforms, the Solaris OS has specific features that can enhance flexibility and performance with different features affecting different processors as noted in the sections that follow. Solaris Containers for Consolidation, Secure Partitioning, and Virtualization Solaris Containers comprise a group of technologies that work together to efficiently manage system resources, virtualize the system, and provide a complete, isolated, and secure runtime environment for applications. Solaris Containers can be used to partition and allocate the considerable computational resources of Sun Fire X4140, X4240, and X4440 servers. Solaris Zones and Solaris Resource Management work together with the Solaris fair-share scheduler. Solaris Zones Solaris Zones can be used to create an isolated and secure environment for running applications. A zone is a virtualized operating system environment created within a single instance of the Solaris OS. Zones can be used to isolate applications and processes from the rest of the system. This isolation helps enhance security and reliability since processes in one zone are prevented from interfering with processes running in another zone. Resource Management Resource management tools provided with the Solaris OS lets administrators dedicate resources such as CPU cycles to specific applications. CPUs in multicore multiprocessor systems such as Sun Fire X4140, X4240, and X4440 servers can be logically partitioned into processor sets and bound to a resource pool, and can ultimately be assigned to a Solaris zone. Resource pools provide the capability to separate workloads so that consumption of CPU resources does not overlap. Resource pools also provide a persistent configuration mechanism for processor sets and scheduling class assignment. In addition, the dynamic features of resource pools let administrators adjust system resources in response to changing workload demands.

35 33 Enterprise-Class Operating System and Management Software Sun Microsystems, Inc. Solaris Dynamic Tracing (DTrace) to Instrument and Tune Live Software Environments When production systems exhibit nonfatal errors or sub-par performance, the sheer complexity of modern distributed software environments can make accurate root-cause diagnosis extremely difficult. Unfortunately, most traditional approaches to solving this problem have proved time-consuming and inadequate, leaving many applications languishing far from their potential performance levels. The Solaris DTrace facility provides dynamic instrumentation and tracing for both application and kernel activities even allowing tracing of application components running in a Java Virtual Machine (JVM ) 1. DTrace lets developers and administrators explore the entire system to understand how it works, track down performance problems across many layers of software, or locate the cause of aberrant behavior. Tracing is accomplished by dynamically modifying the operating system kernel to record additional data at locations of interest. Best of all, although DTrace is always available and ready to use, it has no impact on system performance when not in use, making it particularly effective for monitoring and analyzing production systems. NUMA Optimization in the Solaris OS With memory managed by each processor on Sun Fire X4140, X4240, and X4440 servers, the implementation represents a non-uniform memory access (NUMA) architecture. In NUMA architectures, the speed needed for a processor to access its own memory is slightly different than that required to access memory managed by another processor. The Solaris OS provides technology that can specifically help applications improve performance on NUMA architectures. Memory Placement Optimization (MPO) The Solaris 10 OS uses MPO to improve the placement of memory across the physical memory of a server, resulting in increased performance. Through MPO, the Solaris 10 OS works to help ensure that memory is as close as possible to the processors that access it, while still maintaining enough balance within the system. As a result, many database and HPC applications are able to run considerably faster with MPO. Hierarchical lgroup support (HLS) HLS improves the MPO feature in the Solaris OS. HLS helps the Solaris OS optimize performance for systems with more complex memory latency hierarchies. HLS lets the Solaris OS distinguish between the degrees of memory remoteness, allocating resources with the lowest possible latency for applications. If local resources are not available by default for a given application, HLS helps the Solaris OS allocate the nearest remote resources. Solaris ZFS Solaris ZFS offers a dramatic advance in data management, automating and consolidating complicated storage administration concepts and providing unlimited scalability with the world s first 128-bit file system. ZFS is based on a 1.The terms "Java Virtual Machine" and "JVM" mean a Virtual Machine for the Java platform.

36 34 Enterprise-Class Operating System and Management Software Sun Microsystems, Inc. transactional object model that removes most of the traditional constraints on I/O issue order, resulting in dramatic performance gains. ZFS also provides data integrity, protecting all data with 64-bit checksums that detect and correct silent data corruption. A Secure and Robust Enterprise-Class Environment Best of all, the Solaris OS doesn t require arbitrary sacrifices. The Solaris Binary Compatibility Guarantee helps ensure that existing Solaris applications continue to run unchanged, protecting investments. Certified multilevel security protects Solaris environments from intrusion. Sun s comprehensive Fault Management Architecture means that elements such as Solaris Predictive Self Healing can communicate directly with the hardware to help reduce both planned and unplanned downtime. Integrated Lights Out Management (ILOM) Each Sun Fire X4140, X4240, and X4440 server offers an Integrated Lights Out Manager (ILOM) service processor, allowing remote management for all activities that do not require physically touching the system. Industry standards are embraced throughout, letting these systems easily integrate into existing environments. In addition, since the ILOM service processor is a core component of Sun Fire X4140, X4240, and X4440 systems, and there is no additional charge for this functionality. On-Board ILOM Firmware and Connections The ILOM service processor connects to all major components via on-board interfaces such as I 2 C with a separate management network provided for remote access. Equipped with field-upgradeable firmware, the ILOM service processor supplies management functions for fan speed control and diagnostic LEDs, and provides a wealth of connections to individual server components. Sensors can use the ILOM service processor to generate entries in the system event log when the sensor crosses a certain value. Examples in Sun Fire X4140, X4240, and X4440 servers include: Chassis sensors for intrusion, power supply failure, temperature failure, or fan failure Front and back panel sensors sensing the state of various LEDs and system locate button Motherboard temperature sensors to monitor the ambient temperature chip on the motherboard Power supply sensors to determine whether power supplies are present, connected to AC power, and/or powering the system Hard disk drive backplane sensors to determine the presence and health of the disk backplane Fan sensors to determine the presence of the fan tray, the speed of individual fans, and to detect the failure of individual fans.

37 35 Enterprise-Class Operating System and Management Software Sun Microsystems, Inc. ILOM Communication Channels, User Management, and Security Access to ILOM functionality on Sun Fire X4140, X4240, and X4440 servers can be made through a variety of both out-of-band and in-band communication channels. Out-ofband communication helps ensure that effective management can take place even in the event of hardware or networking failures, and includes: A management serial port that provides direct console access via a command line interface (CLI) A dedicated Ethernet port that provides a web-based GUI (over HTTPS), a CLI via SSH, IPMI 2.0, and SNMP v1, v2c, and v3 In-band communication to the ILOM service processor is provided via the host OS running on the server. A variety of management tools can be used to access management information on individual servers: Intelligent Platform Management Interface (IPMI) with IPMItool IPMItool is a simple command-line interface to systems that support the Intelligent Platform Management Interface (IPMI) v2.0 specification. IPMItool provides the ability to read the sensor data repository and print sensor values, display the contents of the system event log, print field-replaceable unit information, read and set LAN configuration parameters, and perform remote chassis power control. IPMItool was originally written to take advantage of IPMIover-LAN interfaces but it is also capable of using the system interface as provided by a Linux kernel device driver such as OpenIPMI or a Solaris OS driver called BMC that is provided with the Solaris 10 OS. IPMItool is available under a BSDcompatible license. IPMItool is not designed to replace the OpenIPMI library but instead provides a completely command-line oriented tool that can be used by administrators in conjunction with other tools. Where possible, IPMItool supports commaseparated values for output to facilitate parsing by other scripts or programs. IPMItool is designed to run quick command-response functions that can be as simple as turning the system on or off, or as complex as reading in the sensor data records while extracting and printing detailed sensor information for each record. Simple Network Management Protocol (SNMP) Management SNMP management provides remote access by SNMP-compliant entities to monitor and control network devices and manage configurations including statistics collection, performance, and security on a network. SNMP is a network management protocol used almost exclusively in TCP/IP networks. Sun Fire X4140, X4240, and X4440 servers provide SNMP MIBs to manage and monitor the servers using any SNMP-capable network management system, such as HP OpenView Network Node Manager (NNM), Tivoli, CA Unicenter, or IBM Director. The MIB data describes the information being managed, reflects current and recent server status and provides server statistics.

38 36 Enterprise-Class Operating System and Management Software Sun Microsystems, Inc. SNMP v1, v2c, and v3 are supported with v3 selected by default (v1 and v2c are disabled by default). SNMP sets can be selected or disabled (default). An IPMIspecific trap called a Platform Event Trap, or PET, may also be generated. The following SNMP MIBs are supported: SNMP-FRAMEWORK-MIB SNMP-USER-BASED-SM-MIB SNMP-MPD-MIB ENTITY-MIB SUN-PLATFORM-MIB Role-Based Administration Different management users can be defined with corresponding roles and responsibilities. Up to 10 user IDs can be created locally on the service processor with each user ID consisting of a user name and the roles that are allowed. By default, Administrator and Operator roles are defined. Authentication is carried out against a local service processor database. Alternately, an LDAP client is implemented in the ILOM service processor as well to allow authentication against an LDAP server (LDAP groups must be mapped to service processor roles). Up to 10 concurrent active sessions are supported on the service processor, including serial, secure shell and web clients. User accounts can be authenticated through LDAP, Radius, and Active Directory. Remote Keyboard, Video, Mouse, and Storage (RKVMS) To facilitate effective and full-featured remote management, the ILOM service processor provides remote keyboard, video, mouse, and storage (RKVMS) support that is tightly integrated with Sun Fire X4140, X4240, and X4440 servers. Together these capabilities allow the servers to be administered remotely, while accessing keyboard, mouse, video and storage devices local to the administrator (Figure 11). ILOM Remote Console support is provided on the ILOM service processor and can be downloaded and

39 37 Enterprise-Class Operating System and Management Software Sun Microsystems, Inc. executed on the management console. Input/output of virtual devices is handled between ILOM on the server and ILOM Remote Console on the web-based client management console. ILOM Remote Console Displays Remote Video in Application Window Video (Up to Graphics Redirect Over Ethernet Local Mouse and Keyboard Management Console ILOM Remote Console Connected to ILOM Over Management Ethernet Sun Fire X4140/X4240/X4440 Server Floppy Disk or Floppy Image Keyboard, Mouse, CDROM, and Floppy are Seen as USB Devices by BIOS and OS CDROM, DVDROM or.iso Image Remote Keyboard, Mouse and Storage Emulated as USB Devices by ILOM Figure 11. Remote keyboard, video, mouse, and storage (RKVMS) support in the ILOM service processor allows full-featured remote management for Sun Fire X4140, X4240, and X4440 servers Remote Keyboard and Mouse Support Through the ILOM service processor, the Sun Fire X4140, X4240, and X4440 servers detect a USB keyboard and mouse. ILOM Remote Console captures mouse and keyboard input on the management console and sends it to the ILOM service processor. The service processor then transmits mouse and keyboard inputs on the respective USB buses on the server. The server receives keyboard entries and mouse movements as if they were generated by local USB devices. Remote Video Support Each server incorporates a VGA graphics controller, provided by the AST2000 chip. Graphics from the ATI graphics controller is sent to the ILOM service processor. The service processor then redirects the video signal to ILOM Remote Console running on the management system over a network connection, where the video is displayed on the management console. ILOM Remote Console supports 16-bit video to accommodate higher quality.