SUN FIRE X4150 AND X4450 SERVER ARCHITECTURE

Transcription

1 SUN FIRE X4150 AND X4450 SERVER ARCHITECTURE Breakthrough Density Using Sun and Intel Technology White Paper October 2007 SunWIN #

2 Table of Contents Executive Summary Managing Datacenter Capacity and Complexity Introducing the Sun Fire X4150 and X4450 Servers Comparing the Sun Fire X4150 and Sun Fire X4450 Servers Intel Advantage Multi-Chip Packaging Intel Core Microarchitecture Sun Fire X4150 and Sun Fire X4450 Server Architectures Sun Fire X4150 System-Level Architecture Sun Fire X4150 Server Overview Sun Fire X4450 System-Level Architecture Sun Fire X4450 Server Overview System Processors and Chipsets Memory Subsystem I/O Subsystem Enclosure Features Service Processor and System Management RAS Features Enterprise-Class Software The Solaris Operating System Linux Environments Microsoft Windows Environments VMware Support Conclusion

3 Executive Summary Sun Microsystems, Inc. Executive Summary In recent years, emerging applications and expanding Web services have fueled an escalating demand for mission-critical IT resources, with increasing numbers of collaborative business initiatives focused on information sharing and continuous data availability. As a result, IT data centers experienced rapid growth as the number of applications, the population of users, and the quantity of transactions multiplied. Today IT managers face the task of administering a complex storage and computing infrastructure one that typically contains many small servers that were added over time to meet ongoing resource demands. Since administrative, energy, and real estate costs continue to skyrocket, sprawl in the data center can severely strain IT budgets. To curb operational expenses, many companies are turning to consolidation and virtualization to improve resource utilization and enhance business agility. Sun s expertise in delivering mission-critical computing solutions and its focus on ecoresponsibility are evident in the designs of two new platforms, the Sun Fire X4150 and the Sun Fire X4450 servers. Using high performance Intel Xeon processor technology, Sun has engineered powerful 1U and 2U systems that are ideal for HPC, grid computing, database, Web infrastructure, and consolidation and virtualization initiatives. These systems offer breakthrough compute, memory, storage, and I/O density. At the same time, they feature tremendous energy efficiency as well as flexible support for multiple operating systems the Solaris Operating System, Linux, Windows, and VMware. The Sun Fire X4150 and Sun Fire X4450 servers showcase Intel s talent for creating highperformance commodity platforms along with Sun s innovative engineering and quality system design. In addition to remarkable expandability and density (such as 1TB of internal storage and up to four quad-core Intel Xeon processors in a single 2U chassis), these servers feature several redundant and hot-swappable components, efficient frontto-back air flow, highly efficient power supplies, and built-in system management tools. Engineered for mission-critical application availability, the Sun Fire X4150 and Sun Fire X4450 servers strive to conserve valuable energy resources and lower operational expenses while delivering high performance in an extremely small footprint.

4 2 Managing Datacenter Capacity and Complexity Chapter 1 Managing Datacenter Capacity and Complexity In today s competitive global marketplace, companies place ever-increasing demands on the IT infrastructure for compute and storage resources. As a result, most datacenters have grown exponentially in recent years, with large numbers of servers and storage racks continuously added to satisfy business requirements and meet escalating resource demands. Such rapid growth has produced its own set of challenges. The cost of maintaining large numbers of servers can prove to be formidable, especially at a time when real estate costs are rising, energy costs are skyrocketing, and department budgets are flat or declining. Managing the complexity that results from sprawling server resources puts tremendous strain on IT staff and budgets. To successfully meet business needs, IT management must control both cost and complexity while simultaneously scaling processing and storage capacity. Today the responsibility of managing datacenter capacity and rapid growth commonly fuels IT initiatives like consolidation and virtualization. Typically, these initiatives strive to improve resource utilization, reduce administrative complexity, and drive down IT costs. Consolidating many small servers into fewer powerful systems helps to minimize administrative workloads while increasing capacity and conserving valuable datacenter floor space. It can also help to reduce energy costs, vastly improving available performance relative to the amount of energy consumed. In addition, when IT organizations consolidate under-utilized resources, virtualization technologies can balance IT resource demands, applying key resources as needed to business-critical applications to enhance agility. Introducing the Sun Fire X4150 and Sun Fire X4450 Servers To help IT managers address the challenge of increasing capacity while managing datacenter growth, Sun offers new Intel Xeon processor-based systems the Sun Fire X4150 and Sun Fire X4450 servers. These systems feature high performance and unprecedented density in energy-efficient and compact 1U and 2U form-factors. With capabilities that complement the rest of the Sun server product line, the Sun Fire X4150 and Sun Fire X4450 servers raise the bar for 32- and 64-bit enterprise-class computing. These systems offer: Best-in-class performance. The Sun Fire X4150 and Sun Fire X4450 servers feature dual- and quad-core Intel Xeon 5000 and 7000 Sequence processors. With large on-die caches and high clock speeds, these processors offer high system performance and throughput compared with systems based on earlier generation Intel chipsets. In high-end configurations, Sun Fire X4150 and Sun Fire X4450 servers can house multiple quad-core Intel Xeon processors for example, the Sun Fire X4450 server

5 3 Managing Datacenter Capacity and Complexity supports up to four quad-core Intel Xeon X7350 processors clocked at 2.93GHz, for a maximum of 16 cores and impressive performance in a compact 2U chassis. Remarkable density. Density is the cornerstone of the Sun Fire X4150 and X4450 server designs. When populated in a 40-rack unit (RU) enclosure, these 1U and 2U servers facilitate a single rack with up to 320 cores, 1280 DIMM slots, and 120 PCI Express (PCIe) slots. In addition, a single 1U or 2U system can house up to 1TB internally using eight 2.5-inch SAS drives (via a PCIe Host Bus Adapter) or up to six internal 2.5-inch SATA drives. The Sun Fire X4150 and Sun Fire X4450 servers provide the type of densities needed to achieve consolidation and virtualization efficiencies. Such density facilitates the consolidation of many smaller servers, helping to conserve real estate, lower energy expense, and reduce costly administrative talent. In addition, these servers support multiple operating systems, which helps to simplify consolidation efforts and diminish server sprawl. Extensive system expandability. The ability to expand a server over time reduces the need for additional capital acquisitions and lowers application lifecycle costs. The Sun Fire X4150 and Sun Fire X4450 servers feature either two or four CPU sockets per system and either 64GB or 128GB of memory (respectively, using 4GB FBDIMMs), along with a maximum of 1TB of internal storage. Four Gigabit Ethernet ports are standard. Because of these breakthrough system densities, these servers are designed to scale to support new users, more transactions, or new 32-bit or 64-bit applications, enhancing system longevity and increasing overall return on investment (ROI). Improved energy efficiency. Sun offers a portfolio of eco-responsible products and computing solutions to address customer IT needs. In the Sun Fire X4150 and Sun Fire X4450 servers, Intel Xeon processors incorporate new technologies that minimize power use and enhance energy efficiency. The Intel Core Microarchitecture optimizes processor performance relative to the power consumed. Power management capabilities limit power to unused execution units in each core, which helps to reduce power and cooling requirements. High-efficiency power supplies in the server chassis lessen overall power consumption. Variable speed fans, disk carrier design, and front-to-back air flow in the chassis help to effectively cool the system and maintain appropriate processor and system ambient temperatures. Enterprise-class high availability. The Sun Fire X4150 and Sun Fire X4450 servers are designed with enterprise-class RAS (Reliability, Availability, and Serviceability) features. To maximize uptime, systems include redundant hot-swappable fans and can be configured with redundant hot-swappable power supplies. Using a Sun StorageTek Host Bus Adapter (HBA), internal SAS disk drives can be configured for RAID 0, 1, 1E, 10, 5, 5EE, 50, 6, and 60 when mirroring is implemented, drives are also hot-swappable. Four integrated Gigabit Ethernet ports enhance network availability and can be installed in failover configurations. On-board system management tools encourage remote, proactive monitoring and intervention.

6 4 Managing Datacenter Capacity and Complexity Simplified system management. To support out-of-band management, the Sun Fire X4150 and Sun Fire X4450 servers incorporate a Service Processor that features robust Embedded Lights-Out Manager (Embedded LOM) applications. This built-in hardwarebased management functionality allows administrators to monitor and manage systems remotely, allowing them to take corrective action as necessary and minimize unplanned downtime. The Sun Fire X4150 and Sun Fire X4450 servers combine best-in-class performance with noteworthy compute, memory, and I/O capacities. As a result, these systems are designed to scale up, scale out, and scale within, enabling deployment in a wide range of application architectures: Scale-up architectures: With up to 16 cores available, these servers are well-suited to scale for growing workloads that deliver Web, database, and other key infrastructure services. Scale-out architectures: With large memory capacities, internal storage, four Gigabit Ethernet ports, and high-bandwidth PCI expansion for high-speed system interconnects (such as fiber channel and InfiniBand), these servers can scale to solve complex computing problems demanding intensive computing power and data bandwidth. Scale-within: With the ability to support Solaris 10 Virtualization and VMware, Sun Fire X4150 and Sun Fire X4450 servers are ideal systems to host virtualization technologies and consolidate multiple applications within a single extensible platform. Figure 1 shows the 1U Sun Fire X4150 and the 2U Sun Fire X4450 server enclosures. The Sun Fire X4150 Server The Sun Fire X4450 Server Figure 1. Sun Fire X4150 and Sun FIre X4450 Servers

7 5 Managing Datacenter Capacity and Complexity Comparing the Sun Fire X4150 and Sun Fire X4450 Servers Table 1 summarizes features of the Sun Fire X4150 and Sun Fire X4450 server platforms. Both the Sun Fire X4150 and X4450 server systems are good solutions for applications when high density is needed to satisfy fast-growing workloads such as OLTP, database, and Web service delivery. Table 1. Feature Comparison for Sun Fire X4150 and Sun Fire X4450 Servers Feature Sun Fire X4150 Server Sun Fire X4450 Server Chassis 1U 2U Number of CPU sockets 2 4 Supported processor types Number of cores per 40 RU rack enclosure Dual-core Intel Xeon Processor 5100 Series or quad-core Intel Xeon Processor 5300 Series Up to 320 cores Dual-core Intel Xeon Processor 7200 Series or quad-core Intel Xeon Processor 7300 Series Up to 320 cores Processor system bus Dual Front-Side Bus (FSB) Quad Front-Side Bus (FSB) Number of memory slots Memory capacity Up to 64GB (using 4GB FBDIMMs) Up to 128GB (using 4GB FBDIMMs) Memory type DDR2 FBDIMM DDR2 FBDIMM Internal storage Up to inch SAS (with HBA) or up to inch SATA Up to inch SAS (with HBA) or up to inch SATA Removable media 1 EIDE DVD/RW 1 EIDE DVD/RW Number of PCI-Express slots Three total (x8) Six total (two x8 and four x4) Number of GigE ports 4 on-board 4 on-board Number of USB ports 4 external plus 1 internal 4 external plus 1 internal System management tools RAS components Embedded Lights-Out Manager Hot swappable and redundant power supplies, fans, disk drives Embedded Lights-Out Manager Hot swappable and redundant power supplies, fans, disk drives As Table 1 shows, the two systems share a number of features, including: Intel Xeon-based architecture with support for multiple dual- and quad-core processors Large memory capacities using FBDIMM memory modules Multiple high-bandwidth Front Side Bus (FSB) system interconnects Large-capacity internal storage, with either a maximum of eight internal SAS disks (using an HBA) or six internal SATA drives Expandability using PCI Express cards Built-in quad Gigabit Ethernet support Service processor featuring Embedded Lights-Out Manager functionality Support for multiple operating systems

8 6 Managing Datacenter Capacity and Complexity Notable differences between the Sun Fire X4150 and Sun Fire X4450 servers include: Chassis enclosure (1U versus 2U) Number of processor sockets (2 versus 4) Supported processor types (Intel Xeon Processor 5000 Sequence, versus Intel Xeon Processor 7000 Sequence) Memory capacity (up to 64GB versus 128GB using 4GB FBDIMMs) PCI expansion capabilities (3 PCIe slots versus 6 PCIe slots) Multiple off-the-shelf configurations of each platform are available, along with a wide spectrum of options to tailor each system for specific workload requirements. The Sun Fire X4150 and Sun Fire X4450 servers offer the density and configurability necessary to realize operational, administrative, and energy cost-savings the goals of many IT strategic plans.

9 The Intel Advantage Sun Microsystems, Inc. Chapter 2 The Intel Advantage Sun Microsystems, Inc. and Intel Corporation are collaborating in an alliance targeted at building a comprehensive family of servers based on Intel Xeon processors and at optimizing performance for the Solaris Operating System (OS). With the Intel processorbased Sun Fire X4150 and Sun Fire X4450 servers, Sun extends its existing X64 system portfolio and complements its powerhouse SPARC processor-based platforms. Moving forward, Sun and Intel continue to participate in joint engineering efforts aimed at enhancing the Solaris OS, Java technologies, and systems built using Intel Xeon processor designs. In this new family of servers, Sun s well-known engineering expertise combines with an emphasis on performance, quality, reliability, and eco-responsibility. The Sun Fire X4150 and Sun Fire X4450 servers leverage dual- or quad-core Intel Xeon processors that feature key Intel technologies for virtualization, I/O acceleration, and energy efficiency. These 64-bit processors are compatible with the legacy of IA-32 software, instantly making available a large volume of existing 32-bit applications as well as emerging 64- bit applications. Figure 2. 50W Quad-core Intel Xeon L7345 Processor. This chapter introduces the Intel Xeon processors and chipsets used in these new Sun models. The Sun Fire X4150 server has two processor sockets that can be populated with either dual- or quad-core Intel Xeon Processor 5000 Series, for a maximum of 8 cores in a 1U system enclosure. In contrast, the Sun Fire X4450 server contains four processor sockets supporting up to four dual- or quad-core Intel Xeon Processor 7000 Series, for a maximum of 16 cores in a single 2U chassis. The related system chipsets include a Northbridge Memory Controller Hub (MCH) and a Southbridge Input/Output Hub (IOH). For an overview of Intel chipsets used in these Sun servers, see Chapter 3. For detailed information on the Intel chipsets, see the Web sites:

10 8 The Intel Advantage Multi-Chip Packaging The Intel Xeon Processor 5300 Series and 7000 Sequence processors use a Multi-Chip Package (MCP) to deliver quad-core configurations. This packaging approach increases die yields and lowers manufacturing costs, which helps Intel and Sun to deliver higher performance at lower price points. Intel Xeon Processor 5100 and 5300 Series Utilizing the Intel Core Microarchitecture, the Intel Xeon Processor 5100 and 5300 Series provide performance for multiple application types and user environments in a substantially reduced power envelope. The dual-core 5100 Series processors provide significant performance headroom for multithreaded applications and can boost system utilization through virtualization and application responsiveness. The quad-core 5300 Series processors maximize performance-per-watt, providing increased density for datacenter deployments where energy efficiency is a key goal. Figure 3 shows logical diagrams of the 5100 and 5300 Series processors. The 5100 Series processor includes two execution cores, each with a 64K level-1 cache (32K instruction/ 32K data). Both cores share a 4 MB level-2 cache to increase cache-to-processor data transfers, maximize memory-to-processor bandwidth, and reduce latency. The 5300 Series processor uses Multi-Chip Packaging and provides four execution cores, with two cores sharing a 4 MB level-2 cache for a total of 8 MB. With two sockets in the Sun Fire X4150 server, this enables a maximum of 8 processor cores in a single 1U form factor. Figure 3. Intel Series 5100 Processors use a single die in one package while the quad-core Series 5300 processors use two dies

11 9 The Intel Advantage Intel Xeon Processor 7200 and 7300 Series In the Sun Fire X4450 servers, the dual-core Intel Xeon Processor 7200 Series and quadcore Intel Xeon Processor 7300 Series incorporate two die per processor package, with each die capable of containing two processor cores (Figure 4). In the dual-core Intel Xeon 7200 Series, each die includes one processor core, but in the quad-core Intel Xeon Processor 7300 Series, each die contains two cores. In a Sun Fire X4450 server with four processors, this enables up to 16 execution cores in a compact 2U enclosure. Figure 4. Dual-core Intel 7200 and quad-core 7300 Series processors both use two dies in one package The 5000 and 7000 Sequence processor families share these additional features: An on-die Level 1 (L1) instruction data cache (64KB per die) An on-die Level 2 (L2) cache (4MB per die for a total of 8MB in packages with two die) Multiple, independent Front Side Buses (FSBs) that act as high-bandwidth system interconnects. The Intel Xeon 5000 Sequence processors support both 1066MT/s and 1333MT/s Front Side Buses enabling 8.5GB/s data transfer rates (at 1066MT/s ) or 10.5GB/s (at 1333MT/s). The 7000 Sequence supports Front Side Bus interconnects at 1066MT/s. Intel Core Microarchitecture WIth a goal of reducing the processor energy footprint, the Intel Core Microarchitecture delivers high performance with low power consumption. The microarchitecture includes several features designed specifically to enhance performance while maintaining energy efficiency: Out-of-order execution conserves processor-to-memory bandwidth, improving memory access and increasing overall processing performance Speculative prefetches move data to the L2 cache before an L1 cache request occurs Large L2 caches (4MB on 5100 Series dual-core, 8MB on other quad-core processors) increase system memory efficiency and enhance performance Power management capabilities modulate power delivery to execution cores, limiting power to unused components

12 10 The Intel Advantage These features help to conserve processor power, which can translate into energy savings and lower power and cooling costs. Table 2 and Table 3 list typical power envelopes for Intel Xeon processors configurable in the Sun Fire X4150 and Sun Fire X4450 servers. Table 2. Power Envelope for Intel Xeon Processor 5000 Sequence used in Sun Fire X4150 servers Processor Speed L2 Cache FSB Power Dual-core GHz 4MB 1333MT/s 80W Quad-core L GHz 8MB 1066MT/s 50W Quad-core E GHz 8MB 1066MT/s 80W Quad-core E GHz 8MB 1333MT/s 80W Quad-core X GHz 8MB 1333MT/s 120W Table 3. Power Envelope for Intel Xeon Processor 7000 Sequence used in Sun Fire X4450 servers Processor Speed L2 Cache FSB Power Dual-core E GHz 8MB 1066MT/s 80W Quad-core L GHz 8MB 1066MT/s 50W Quad-core E GHz 4MB 1066MT/s 80W Quad-core E GHz 8MB 1066MT/s 80W Quad-core X GHz 8MB 1066MT/s 130W

13 11 Sun Fire X4150 and Sun Fire X4450 Server Architectures Chapter 3 Sun Fire X4150 and Sun Fire X4450 Server Architectures The Sun Fire X4150 and Sun Fire X4450 servers are designed to provide best-in-class performance with high reliability and low power consumption. This chapter details physical and architectural aspects of the systems, highlighting similarities and differences between the two server designs. Sun Fire X4150 System-Level Architecture Figure 5 and Figure 6 contain system-level block diagrams for Sun Fire X4150 servers using SAS internal drives and SATA internal drives, respectively. Other than the difference in storage subsystems, these block diagrams reflect identical system-level architectures. The architecture of the Sun Fire X4150 server is similar to that of the Sun Fire X4450 server multiple host processors connect to a Northbridge Memory Controller Hub (MCH) which in turn connects to a Southbridge I/O Hub (IOH). Intel Xeon 5100/5300 TM XEON TM XEON Intel Xeon 5100/5300 B3 B2 B1 B0 FSB 1333 MT/s FSB 1333 MT/s C0 C1 C2 C GB/s 10.5 GB/s DIMMs Dual FSB to MCH Channel C Channel B 5.3 GB/s 5.3 GB/s DIMMs MCH Blackford 5000P DIMMs 5.3 GB/s 5.3 GB/s Channel D Channel A A3 A2 A1 A0 PCI-E x8 PCI-E ESI (PCI-E) PCI-E x8 PCI-E SAS/RAID Controller PCI-E x8 USB IOH ESB-2 USB to IDE PCI-E x4 PCI 32-bit 33 MHz 2x USB IDE USB Hub AST2000 Q GP 0608 TAN A2 CD/DVD 1x Internal USB 2.0 2x Rear USB 2.0 2x Front USB 2.0 2x 1GB Ethernet 2 & 3 2x 1GB Ethernet 0 & 1 Serial RJ-45 Management 10/100 Ethernet VGA Video D0 D1 D2 D3 DIMMs PCI-E x16-2 PCI-E x16-1 PCI-E x16-0 8x SAS HDDs Figure 5. Block Diagram of Sun Fire X4150 Server with SAS drives

14 12 Sun Fire X4150 and Sun Fire X4450 Server Architectures Intel Xeon 5100/5300 TM XEON TM XEON Intel Xeon 5100/5300 B3 B2 B1 B0 FSB 1333 MT/s FSB 1333 MT/s C0 C1 C2 C GB/s 10.5 GB/s DIMMs Dual FSB to MCH Channel C Channel B 5.3 GB/s 5.3 GB/s DIMMs MCH Blackford 5000P DIMMs 5.3 GB/s 5.3 GB/s Channel D Channel A A3 A2 A1 A0 PCI-E x8 PCI-E ESI (PCI-E) PCI-E x8 PCI-E x8 USB SATA x6 IOH ESB-2 USB to IDE PCI-E x4 PCI 32-bit 33 MHz 2x USB IDE USB Hub AST2000 Q GP 0608 TAN A2 CD/DVD 1x Internal USB 2.0 2x Rear USB 2.0 2x Front USB 2.0 2x 1GB Ethernet 2 & 3 2x 1GB Ethernet 0 & 1 Serial RJ-45 Management 10/100 Ethernet VGA Video D0 D1 D2 D3 DIMMs PCI-E x16-2 PCI-E x16-1 PCI-E x16-0 6x SATA HDDs Figure 6. Block Diagram of Sun Fire X4150 Server with SATA Drives Details about key subsystems system chipsets, memory subsystems, I/O subsystems, etc. are given later in this chapter. Sun Fire X4150 Server Overview The Sun Fire X4150 server includes these major components: One or two dual- or quad-core Intel Xeon Processor 5000 Series Up to 64 GB of memory populated in 16 Fully-Buffered Dual Inline Memory Module (FBDIMM) slots (1GB, 2GB, or 4GB FBDIMMs are supported) Four on-board 10/100/1000 Mbps Ethernet ports Three low-profile PCI Express slots, each 8-lane Up to eight internal 2.5-inch SAS drives (via a PCI Express Host Bus Adapter) or up to six internal 2.5-inch SATA drives (via an integrated SATA controller) Five USB 2.0 ports An integrated Service Processor featuring Embedded Lights-Out Manager (Embedded LOM) functionality Up to two hot-swappable, high-efficiency power supply units (PSUs) for N+1 redundancy Seven hot-swappable, variable speed fan modules (for N+1 redundancy), each containing two fans operating under environmental monitoring

15 13 Sun Fire X4150 and Sun Fire X4450 Server Architectures Sun Fire X4150 System Enclosure The Sun Fire X4150 server enclosure is designed to occupy one rack unit in a standard 19-inch rack (Table 4). Table 4. Dimensions and weight of the Sun Fire X4150 server Dimension U.S. International Height 1.73 inches (1 RU) 44 millimeters Width inches inches (including ears ) millimeters millimeters (including ears )) Depth 28 inches millimeters inches (including PSU handles) millimeters (including PSU handles) Weight 30.6 pounds minimum 40.6 pounds maximum 13.9 kilograms minimum 18.4 kilograms maximum Sun Fire X4150 System Front and Rear Perspectives Figure 7 illustrates the front and rear panels of the Sun Fire X4150 server. Component status indicators System status indicators DVD drive USB ports Hard disk drives Redundant (N+1) power supply units PCI Express slots System status indicators Serial and network Management ports 10/100/1000 Ethernet ports USB ports VGA port Figure 7. Sun Fire X4150 server, front and rear panels External features and connections include: Front and rear status indicator lights, reporting locator (white), service required (amber), and activity status (green) for the system and components Up to eight hot-plug SAS or up to six SATA disk drives, which insert through the front panel One slimline, slot-accessible DVD-RW, accessible through the front panel Four USB ports, two on the front panel and two on the rear. (A fifth internal USB port is used to attach internal boot devices.)

16 14 Sun Fire X4150 and Sun Fire X4450 Server Architectures Up to two power supply units (for N+1 redundancy) with integrated fans, with each power supply having a single, independent AC plug on the rear panel Rear power-supply indicator lights, showing the status of each hot-swappable power supply Four 10/100/1000Base-T autosensing Ethernet ports, accessible on the rear panel Three PCI Express slots, in which low-profile cards can be installed from the rear panel Two management ports (one 10/100 Mbit Ethernet port and one RJ-45 serial management port on the rear panel) for default connections to the Service Processor VGA video port with an analog HD-15 VGA connector on the rear panel Sun Fire X4450 System-Level Architecture Figure 8 and Figure 9 contain system-level block diagrams for Sun Fire X4450 servers with SAS internal drives and SATA internal drives, respectively. Except for the difference in storage, these block diagrams are identical. The architecture is similar to the Sun Fire 4150 server s system-level architecture multiple processors interface to the Northbridge MCH and Southbridge IOH. Intel Xeon 7200/7300 TM XEON TM XEON TM XEON TM XEON Intel Xeon 7200/7300 B7 B6 B5 B4 B3 B2 B1 B0 FSB 1066 MT/s FSB 1066 MT/s FSB 1066 MT/s FSB 1066 MT/s C0 C1 C2 C3 C4 C5 C6 C7 DIMMs Channel B 8.5 GB/s 8.5 GB/s Channel C DIMMs D0 D1 D2 D3 D4 D5 D6 D7 MCH Clarksboro 7000P DIMMs Channel A Channel D PCI-E x4 DIMMs PCI-E x8-3 PCI-E x8 PCI-E x16-0 A7 A6 A5 A4 A3 A2 A1 A0 PCI-E ESI (PCI-E) PCI-E SAS/RAID Controller PCI-E x4 PCI-E x8-4 PCI-E x8 PCI-E x16-1 PCI-E x4 PCI-E x8-5 PCI-E x4 PCI-E x16-2 USB IOH ESB-2 USB to IDE PCI-E x4 2x USB Figure 8. Block Diagram of Sun Fire X4450 Server with SAS Drives PCI 32-bit 33 MHz IDE USB Hub AST2000 Q GP 0608 TAN A2 8x SAS HDDs CD/DVD 1x Internal USB 2.0 2x Rear USB 2.0 2x Front USB 2.0 2x 1GB Ethernet 2 & 3 2x 1GB Ethernet 0 & 1 Serial RJ-45 Management 10/100 Ethernet VGA Video

17 15 Sun Fire X4150 and Sun Fire X4450 Server Architectures Intel Xeon 7200/7300 TM XEON TM XEON TM XEON TM XEON Intel Xeon 7200/7300 B7 B6 B5 B4 B3 B2 B1 B0 FSB 1066 MT/s FSB 1066 MT/s FSB 1066 MT/s FSB 1066 MT/s C0 C1 C2 C3 C4 C5 C6 C7 DIMMs Channel B 8.5 GB/s 8.5 GB/s Channel C DIMMs D0 D1 D2 D3 D4 D5 D6 D7 MCH Clarksboro 7000P DIMMs Channel A Channel D PCI-E x4 DIMMs PCI-E x8-3 PCI-E x8 PCI-E x16-0 A7 A6 A5 A4 A3 A2 A1 A0 PCI-E ESI (PCI-E) PCI-E x4 PCI-E x8-4 PCI-E x8 PCI-E x16-1 PCI-E x4 PCI-E x8-5 PCI-E x4 PCI-E x16-2 USB IOH ESB-2 SATA x6 USB to IDE PCI-E x4 2x USB Figure 9. Block Diagram of Sun Fire X4450 Server with SATA Drives PCI 32-bit 33 MHz IDE USB Hub AST2000 Q GP 0608 TAN A2 6x SATA HDDs CD/DVD 1x Internal USB 2.0 2x Rear USB 2.0 2x Front USB 2.0 2x 1GB Ethernet 2 & 3 2x 1GB Ethernet 0 & 1 Serial RJ-45 Management 10/100 Ethernet VGA Video Key subsystems the chipset, the memory subsystem, the I/O subsystem, etc. are discussed later in this chapter. Sun Fire X4450 Server Overview The Sun Fire X4450 server includes the following major components: Up to four dual- or quad-core Intel Xeon Processor 7000 Series Up to 128 GB of memory in 32 Fully-Buffered Dual Inline Memory Module (FBDIMM) slots (1GB, 2GB, or 4GB FBDIMMs supported) Four on-board 10/100/1000 Mbps Ethernet ports Six low-profile PCI Express slots, two 8-lane and four 4-lane Up to eight internal 2.5-inch SAS drives (via a PCI Express Host Bus Adapter) or up to six internal 2.5-inch SATA drives (via an embedded SATA controller) Five USB 2.0 ports An integrated Service Processor that features Embedded Lights-Out Manager (Embedded LOM) functionality

18 16 Sun Fire X4150 and Sun Fire X4450 Server Architectures Up to two hot-swappable, high-efficiency power supply units (PSUs) for N+1 redundancy Six hot-swappable, variable speed fan modules (for N+1 redundancy), each with two fans operating under environmental monitoring and control Sun Fire X4450 System Enclosure The Sun Fire X4450 server enclosure occupies two rack units in a standard 19-inch rack (Table 4). Table 5. Dimensions and weight of the Sun Fire X4450 server Dimension U.S. International Height 3.46 inches (2 RU) millimeters Width inches inches (including ears ) millimeters millimeters (including ears )) Depth 28 inches millimeters inches (including PSU handles) millimeters (including PSU handles) Weight 41.7 pounds minimum 56.3 pounds maximum 18.9 kilograms minimum 25.6 kilograms maximum Sun Fire X4450 System Front and Rear Perspectives Figure 7 illustrates the front and rear panels of the Sun Fire X4450 server Component status indicators System status indicators DVD drive USB ports Hard disk drives Redundant (N+1) power supply units PCI Express slots System status indicators Serial and network Management ports 10/100/1000 Ethernet ports USB ports VGA port Figure 10. Sun Fire X4450 server, front and rear panels

19 17 Sun Fire X4150 and Sun Fire X4450 Server Architectures External features include: Front and rear status indicator lights, reporting locator (white), service required (amber), and activity status (green) for the system and components Up to eight hot-plug SAS or up to six SATA disk drives, which insert through the front panel One slimline, slot-accessible DVD-RW, accessible through the front panel Four USB ports, two on the front panel and two on the rear. (A fifth internal USB port is used to attach internal boot devices.) Up to two power supply units (for N+1 redundancy) with integrated fans, with each power supply having a single, independent AC plug on the rear panel Rear power-supply indicator lights, showing the status of each hot-swappable power supply Four 10/100/1000Base-T autosensing Ethernet ports, accessible on the rear panel Six PCI Express slots, in which low-profile cards can be installed from the rear panel Two management ports (one 10/100 Mbit Ethernet port and one RJ-45 serial management port on the rear panel) for default connections to the Service Processor VGA video port with an analog HD-15 VGA connector on the rear panel System Processors and Chipsets Although the Sun Fire X4150 and Sun Fire X4450 servers are based on different Intel Xeon processors and associated chipsets, they share a similar system-level architecture. In both systems, multiple host processors interface to the Northbridge MCH over multiple Front-Side Buses (FSBs) two FSBs in the Sun Fire X4150 server and four in the Sun Fire X4450 server. The Southbridge IOH interfaces to the I/O devices, enabling expandability along with high I/O throughput. Each chipset is designed to match processor performance with memory capacity, I/O expandability, and interconnect bandwidth. Intel Xeon 5000 Sequence Chipset Refer back to the Sun Fire X4150 server block diagrams in Figure 5 and Figure 6 (page 11 and page 12). The Sun Fire X4150 server incorporates the Intel Xeon Processor 5000 Sequence chipset consisting of: One or two dual-core Intel Xeon 5100 Series processors or quad-core Intel Xeon 5300 Series processors The Northbridge Intel 5000P Memory Controller Hub (MCH) The Southbridge Intel 6321ESB Input/Output Handler IOH (ESB-2). The ESB-2 is interconnected to the MCH using one ESI link and one PCIe link. The ESB-2 also provides support for up to six SATA disks connected to the disk backplane. Dual independent Front Side Buses (FSBs) act as a system interconnect between each processor socket and the MCH. Operating at either 1066MT/s or 1333MT/s, the 64-bit wide FSBs are capable of peak bandwidths up to 8.5GB/s or 10.5GB/s respectively. For

20 18 Sun Fire X4150 and Sun Fire X4450 Server Architectures more information on the Intel Xeon Processor 5000 Sequence chipset, see Intel Xeon 7000 Sequence Chipset Refer back to the Sun Fire X4450 server block diagrams in Figure 8 and Figure 9 (page 14 and page 15) The Sun Fire X4450 server uses an Intel Xeon Processor 7000 Sequence chipset, which consists of: Up to four dual-core Intel Xeon 7200 Series processors or quad-core Intel Xeon 7300 Series processors The Northbridge Intel 7000P Memory Controller Hub (MCH) The Southbridge Intel 6321ESB Input/Output Handler IOH (ESB-2) With the 7000 Sequence chipset, four FSB buses connect between the four processor sockets and the MCH. Running at 1066MT/s, the 64-bit wide FSBs are capable of peak bandwidths of 8.5GB/s each. For more information on the Intel Xeon Processor 7000 Sequence chipset, see Memory Subsystem To address memory-intensive applications, multiple FBDIMM memory channels help to support large memory densities and high bandwidth in the Sun Fire X4150 and Sun Fire X4450 servers. The memory subsystems are based on similar Intel MCH controller designs that feature four independent memory channels. In the Sun Fire X4150 server, each memory channel supports up to four FBDIMMs, enabling up to 16 FBDIMMs per system for a maximum capacity of 64GB using 4GB modules. In the Sun Fire X4450 server, each channel supports up to eight memory modules for up to 32 FBDIMMs per system, allowing a memory capacity of up to 128GB in the 2U chassis using 4GB FBDIMMs. In both systems, the four memory channels are organized into two branches. Modules of identical size must be populated in pairs, starting with memory slot 0 on each channel (slot 0 on each channel is marked with white handles). The same 667MHz, PC DDR2 FBDIMM modules in either 1MB, 2MB, or 4MB capacities are supported in both servers. Peak read bandwidth to the FBDIMMs is 5.3GB/s per channel (21GB/s total with the 1333MT/s system bus), and peak write bandwidth is 2.7GB/s per channel (10.7GB/s total with the 1333MT/s system bus). Memory modules feature Error Checking and Correcting (ECC) with Chipkill technology for high reliability.

21 19 Sun Fire X4150 and Sun Fire X4450 Server Architectures I/O Subsystem Designed for the headroom needed to expand systems and scale applications, the Sun Fire X4150 and X4450 servers feature a PCI Express expansion bus, integrated storage, four on-board Intel Gigabit Network Interface Controllers (NICs), and five USB ports. The same Southbridge ESB-2 chip is used on both server models. It is interconnected to each system s Northbridge MCH using one ESI link and one PCIe link. The ESB-2 provides two built-in Gigabit Ethernet NICs going to external NIC ports 0 and 1. On both systems, two additional Gigabit Ethernet NICs (port 2 and 3) are connected to a Dual Gigabit Intel Ophir chip, which interfaces to the ESB-2 using a 4-lane PCIe link. The ESB-2 also supports all USB functionality two USB ports go from the ESB-2 to the rear of the system, one is routed to an internal USB hub for the two front USB connections, and one USB port is inside the chassis for internal boot devices. The ESB-2 also supports a USB-to-IDE interface that enables the connection of an optional internal EIDE DVD/RW drive, and connects to the Service Processor via a 32- bit, 33MHz PCI channel. In addition, the ESB-2 provides on-board support for up to six SATA disks connected to the disk backplane. System Network Interfaces Multiple on-board Gigabit Ethernet connections promote flexible connectivity, as well as configurations that support network interface failover. Each server features four 10/ 100/1000Mb/s Ethernet ports on the rear panel, numbered in sequence from left to right. Each port auto-negotiates its link connection LEDs above the port indicate the speed of the established link (green signifies that the established link is 1000Mb/s). All four Ethernet interfaces support PXE boot for network interface booting. PCI Express Expansion Bus The Sun Fire X4150 and Sun Fire X4450 servers include a PCI Express expansion bus that accommodates 64-bit low-profile cards. On both systems, three right-angle risers plug directly into the motherboard to enable PCI expansion (Figure 11). On the Sun Fire X4150 server, the risers provide one slot per riser, for a total of three internal 8-lane PCI- Express slots (all with x16 mechanical connectors). In contrast, on the Sun Fire X4450 servers, each of three tall risers supports two card slots with one x8 and one x16 mechanical connector for a total of six slots in the 2U chassis.

22 20 Sun Fire X4150 and Sun Fire X4450 Server Architectures x8 electrical x16 mechanical PCI Riser for Sun Fire X4150 Server (1 slot per riser) x8 mechanical x16 mechanical PCI Riser for Sun Fire X4450 Server (2 slots per riser) Figure 11. Right-angle PCI risers Table 6 and Table 7 summarize PCI electrical and mechanical characteristics for each slot. Cards should be compliant with Revision 1.0a of the PCI Express Card Electromechanical Specification. Table 6. Characteristics of Sun Fire X4150 PCI Slots Riser/Slot Numbers Electrical Mechanical Connects to Riser 0/Slot 0 x8 x16 ESB-2 Riser 1/Slot 1 x8 x16 MCH Riser 2/Slot 2 x8 x16 MCH Table 7. Characteristics of Sun Fire X4450 PCI Slots Riser/Slot Numbers Electrical Mechanical Connects to Riser 0 Slot 0 Slot 3 x8 x4 x16 x8 MCH MCH Riser 1 Slot 1 Slot 4 x8 x4 x16 x8 MCH MCH Riser 2 Slot 2 Slot 5 x4 x4 x16 x8 ESB-2 ESB-2 On the Sun Fire X4150 server, Slot 0 interfaces to the ESB-2 IOH while Slots 1 and 2 connect directly to the MCH. On the Sun Fire X4450 server, Slots 2 and 5 are routed to the ESB-2 while Slots 0, 1, 3, and 4 connect to the MCH.

23 21 Sun Fire X4150 and Sun Fire X4450 Server Architectures PCI cards are installed with a horizontal orientation. Slots are protected by a honeycombed grill that helps to increase air flow through the chassis and decrease EMI emissions. Integrated Storage In addition to breakthrough compute, memory, and I/O densities, the Sun Fire X4150 and Sun Fire X4450 servers offer internal storage of up to 1TB in each chassis and up to 1.8TB in each chassis when 300GB SATA drives become available. Both systems support up to eight internal 2.5-inch SAS drives (via a PCI Express Host Bus Adapter) or up to six internal 2.5-inch SATA drives (via an on-board SATA controller). The ample storage density is due partly to an innovative new drive carrier design that incorporates honeycombed grillwork to facilitate effective air flow above and below each drive. The carrier includes an ejection handle that simplifies drive removal drives are hotswappable when disk mirroring is configured. Drive status lights indicate Ready to remove, Fault, and Status. The disks plug into a 1U (Sun Fire X4150 server) or 2U (Sun Fire X4450 server) backplane board. Initially these servers are available with either 73GB or 146GB 10,000RPM SAS drives. Shortly after initial release, 200GB or 300GB SATA disks (at 5400RPM) will also be available. Note that mixing SAS and SATA devices in a system is not supported. Disk Controller and RAID Options The Sun Fire X4150 and Sun Fire X4450 servers offer three basic options for disk controllers: The embedded SATA controller, which supports up to six internal disks in RAID 0, 1, and 10 configurations. (Again, SATA disks become available after the initial server release.) The Sun StorageTek SAS adapter, which supports hardware RAID 0, 1, or 1E. Based on LSI 3081E technology, this controller is a low-profile card with two external 4-port SAS connectors. (This adapter also becomes available subsequent to the initial server release.) The Sun StorageTek SAS RAID adapter, which offers hardware RAID 0, 1, 1E, 5, 5EE, 6, 10, 50, and 60. Based on Adaptec technology, this adapter is a low-profile card with two internal 4-port SAS connectors. Cables from the adapter are wired within the system directly to the disk backplane. This adapter includes 256MB of DDR2 memory on-board and a battery-backed write cache for 72-hour backup, which helps to deliver protected, high availability storage. DVD/USB Assembly A slim form-factor IDE optical drive assembly is also available as an option to the Sun Fire X4150 and Sun Fire X4450 servers. This assembly provides an internal DVD-RW device as well as two USB ports accessible from the front panel. Alternatively, systems

24 22 Sun Fire X4150 and Sun Fire X4450 Server Architectures can be ordered with an option that features two USB ports only (and no DVD device), or with no DVD/USB option at all. Enclosure Features The Sun Fire X4150 and Sun Fire X4450 servers feature innovative chassis designs engineered to conserve system power and lessen the need for cooling. The power and cooling efficiency of these systems exceeds that of many competitive systems configured with similar processing, memory, and storage capacities. The effective frontto-back airflow design helps to lower component temperatures, reducing the number of fans needed to cool the system. Key chassis design features include: Hex-shaped, honeycombed air inlet holes that enhance airflow and provide EMI shielding Front-mounted fans (located directly behind the disk drive cage) that pull air through the chassis, over system components, and exhaust it out the rear panel Cable and component placement that helps to optimize cooling efficiency by channeling airflow effectively Fewer DC-DC conversions, which enhances power efficiency and generates less heat Power Distribution Engineered for high availability as well as low energy consumption, the Sun Fire X4150 and Sun Fire X4450 servers can be configured with two highly-efficient, redundant, hotswappable AC/DC power supply units (PSUs), each with separate power cords. The PSUs differ between systems the Sun Fire X4150 server uses 650W PSUs, while the Sun Fire X4450 server requires 1050W PSUs. Configuring a system with a second power supply enables N+1 redundancy, supplying continuous power to the system if a single power supply fails. In both systems the power supplies are high-efficiency units, having a typical efficiency rating that exceeds at least 80%. Each PSU features a non-removable internal fan that supplies independent PSU cooling. Three light indicators display power supply status information ( AC, Fault, and OK ). Both the Sun Fire X4150 and Sun Fire X4450 servers use a Power Distribution Board (PDB) that provides connections between the power supplies and major system components. The PDB contains a single 10Amp 12V-to-5V DC-DC supply used to power the disk subsystem and the optional IDE DVD-RW device. Fan Assemblies The Sun Fire X4150 and Sun Fire X4450 server enclosures are designed for efficient frontto-back air flow. Variable speed fans run under the control of the on-board Service Processor, which monitors processor temperatures and system ambient air

25 23 Sun Fire X4150 and Sun Fire X4450 Server Architectures temperature. Based on these readings, the fans operate at the lowest speeds possible to provide sufficient cooling conserving power usage, prolonging fan life, and reducing acoustical noise. Fan assemblies differ between the Sun Fire X4150 and Sun Fire X4450 servers. The Sun Fire X4150 server houses seven fan modules, with each module containing two 40mm, 10,000RPM fans. In the Sun Fire X4450 server, there are six fan modules, with each module accommodating two 60mm fans. A green status light on a fan module indicates proper operation while an amber light indicates a fan fault. Fan modules on both systems are designed for redundancy a backup fan enables system continuity in the event of a fan failure. The fans are also hotswappable such that a module with a failed fan can be removed and a new fan module inserted without shutting down the system. Rack Mounting To shorten time-to-deployment, the Sun Customer Ready program can pre-install Sun Fire X4150 or Sun Fire X4450 servers in a rack configuration. Alternatively, the 1U and 2U servers can be field-installed into Sun or other third-party racks. The following options are available from Sun to facilitate rack mounting: Rack-Mounting Slide Rail Kit. This is a 4-point mounted slide rail kit (i.e., mounting points are located at the rack front and rear). Tool-less Rack Kit. As the name implies, this rack-mounting kit snaps into most thirdparty racks without requiring any tools. Cable Management Arm. The Cable Management Arm supports and protects cables as the server slides into and out of the rack. The Slide Rail Kit and optional Cable Management Arm enable Sun Fire X4150 or Sun Fire X4450 servers to be racked in: Sun Rack 900 (-38 and -36N) Sun Rack 1000 (-38 and -42) Third Party ANSI/EIA-310-D-1992 or IEC compliant racks in 19-inch/482.6mm panel-width series The Slide Rail Kit includes hardware to mount to rack rails with either 6mm threaded holes, #10-32 threaded holes, #10 clearance holes, or square unthreaded holes per ANSI/EIA 310-D-1992 or IEC standards. Note that not all third-party racks are compatible with the slide rail kit. Rack density will vary widely based on systems installed, power distribution (in-cabinet or external), power source (single-phase or three-phase), and whether redundant power is required.

26 24 Sun Fire X4150 and Sun Fire X4450 Server Architectures Service Processor and System Management Similar to other Sun servers, the Sun Fire X4150 and Sun Fire X4450 servers feature a built-in, hardware-based service processor. Integrating the Service Processor on the motherboard enables remote power control and remote system monitoring capabilities, which greatly simplifies system management. The Sun Fire X4150 and Sun Fire X4450 servers incorporate an Aspeed AST2000 chip as the on-board Service Processor. The Aspeed AST2000 combines a graphics controller and a Service Processor into a single chip, saving space and power. It uses two USB ports for virtual devices and one 32-bit, 33Mhz PCI bus for data to connect to the Southbridge ESB-2 (refer back to system block diagrams on page 11 through page 15). For system management operations, the AST2000 supports three external interfaces on the rear panel: one RJ45 RS-232 serial interface one 10/100 MB/s Ethernet network interface (also an RJ-45 connector) one HD-15 SVGA Video port The serial interface is intended for console use only (it does not support modem connection). Default serial port configuration parameters are 9600 baud, 8 data bits, no parity, 1 stop bit, and no flow control. By default, the BIOS directs system output to the Service Processor, or it can be redirected to the serial port by changing a BIOS option. The BIOS can also be used to check or modify the Service Processor s IP address. Embedded Lights-Out Manager (Embedded LOM) The need for high availability in business-critical systems calls for robust and proactive system management. The AST2000 Service Processor features Embedded LOM, a library of firmware-resident system management capabilities. The AST2000 Service Processor is IPMI 2.0 compliant and enables management functionality that includes: system configuration information retrieval; key hardware component monitoring; remote power control; full local and remote keyboard/video/ mouse (KVM) access; remote media attachment; SNMP V1, V2c, and V3 support and event notification and logging. Digital thermal sensors on the processor die are used to monitor temperatures, which helps to increase reliability. The Service Processor continuously monitors and detects errors on these and other sensors in the system: Memory Voltage CPU Core Voltage System 3.3V System 5V System 12V CPU Fan

27 25 Sun Fire X4150 and Sun Fire X4450 Server Architectures System Fans CPU Temperature Chassis Temperature Through Embedded LOM, administrators can track status information from these sensors and take action before a minor system issue escalates into a serious problem. Secure access to the Service Processor and Embedded LOM is available in several ways: Via a Secure Shell (SSH) command line interface. To access Embedded LOM through a remote Secure Shell (SSH), a Secure Shell communications application must be available on the remote administrative system. Using an IPMI v2.0 command interface Via Simple Network Management Protocol (SNMP) v3 interfaces Using the Web-based graphical interface (Figure 12) Redirecting the managed server s console to the Remote Console application (Figure 13) Figure 12. The Web interface to Embedded LOM on the Service Processor facilitates GUI-based out-of-band management

28 26 Sun Fire X4150 and Sun Fire X4450 Server Architectures The Web-based graphical interface to Embedded LOM offers a wide spectrum of management capabilities, from retrieving current system configuration data to monitoring the status of key hardware components. (Figure 12 shows one of the many management screens accessible through the GUI interface.) The Remote Console application (Figure 13) is a piece of layered software that allows remote clients to view the graphical console as though clients were directly attached to the managed server s video connector. Remote Console can be configured through the the Embedded LOM GUI interface. The remote keyboard, mouse, CD-ROM drive, or floppy drive then appear as standard USB devices. Figure 13. The Remote Console mirrors the video display from the VGA device to the remote management system The Desktop Management Task Force s (DMTF) Systems Management Architecture for Server Hardware (SMASH) command line protocol is supported over both the serial interface and the secure shell network interface. A Web server and the Java Webstart Remote Console application are embedded in the service processor, minimizing the need for software installation on remote administrative workstations. For enhanced security, the Service Processor includes multi-level role-based access to features. The Service Processor supports native and Active Directory Service lookup of authentication data. All functions can be provided out-of-band through a designated serial or network interface, virtually eliminating any performance impact to workload processing.