An Oracle White Paper December A Technical Overview of Oracle s SPARC SuperCluster T4-4

Transcription

1 An Oracle White Paper December 2012 A Technical Overview of Oracle s SPARC SuperCluster T4-4

2 Introduction... 2 SPARC SuperCluster T4-4 Components... 3 SPARC T4-4 Servers... 5 Oracle Exadata Storage Server X Sun ZFS Storage 7320 Appliance... 8 Networking Sun Datacenter InfiniBand Switch Oracle Solaris Configurations Managing SPARC SuperCluster Infrastructure Services Within SPARC SuperCluster Oracle Database 11g Release Oracle Exalogic Elastic Cloud Software Oracle Solaris Cluster Conclusion... 22

3 Introduction The Oracle SPARC SuperCluster T4-4 is a multi-purpose engineered system that has been designed, tested and integrated to run mission critical enterprise applications and rapidly deploy cloud services while delivering extreme efficiency, cost savings, and performance. It is well suited for multi-tier enterprise applications with Web, database and application components. This versatility along with powerful, bundled virtualization capabilities makes it an ideal platform on which to consolidate large numbers of applications, databases, and middleware workloads, or to deploy complex, multi-user development, test, and deployment environments. It combines highly available and scalable technologies, such as optional Oracle Database 11g Real Application Clusters (Oracle RAC) and optional Oracle Solaris Cluster software with industry-standard hardware. Its architecture enables a high degree of isolation between concurrently deployed applications, which may have varied security, reliability, and performance requirements. SPARC SuperCluster T4-4 provides an optimal solution for all database workloads, ranging from scanintensive data warehouse applications to highly concurrent online transaction processing (OLTP) applications. With its combination of the Oracle Exadata Storage Server, Oracle Database software, and the latest hardware components, SPARC SuperCluster T4-4 delivers extreme performance in a highly available, highly secure environment. One example of the performance gains with SPARC SuperCluster comes with each Oracle Exadata Storage Server used in the rack. Each Oracle Exadata Storage Server uses Smart Flash Logging, which both improves user transaction response time, and increases overall database throughput for I/O intensive workloads by accelerating performance critical database algorithms. Customers can integrate SPARC SuperCluster T4-4 systems with Oracle Exadata or Oracle Exalogic machines by using the available InfiniBand expansion ports and optional data center switches. The InfiniBand technology used by SPARC SuperCluster T4-4 offers high bandwidth, low latency, hardware-level reliability, and security. For application environments that follow Oracle's best practices for highly scalable, fault-tolerant systems, no application architecture or design changes are required to Page 2

4 benefit from SPARC SuperCluster T4-4. Deployments can also connect many SPARC SuperCluster T4-4 systems or a combination of SPARC SuperCluster T4-4 systems and Oracle Exadata Database Machines to develop a single, large-scale environment. Customers can integrate SPARC SuperCluster T4-4 systems with their current data center infrastructure using the available 10 GbE ports in each SPARC T4-4 server within SPARC SuperCluster. SPARC SuperCluster T4-4 Components The Oracle SPARC SuperCluster incorporates Oracle Exadata storage technology for enhancing the performance of the Oracle Database. SPARC SuperCluster incorporates optional Exalogic Elastic Cloud Software technology to accelerate performance of Java middleware and applications, generalpurpose applications, and the Oracle Database 11g Release 2. It is an engineered system designed to host the entire Oracle software solution stack. Third party ISV software, as well as customer developed software can also be installed in the Application Domains. In addition to the Oracle Exadata Storage Servers built in to SPARC SuperCluster, Exadata Storage Expansion Racks can be used to add capacity and bandwidth to the system. SPARC SuperCluster T4-4 is designed to fully leverage an internal InfiniBand fabric that connects all of the processing and storage components within SPARC SuperCluster T4-4 to form a single, large computing device. Each SPARC SuperCluster T4-4 is connected to the customer s data center networks via 10 GbE (traffic) and 1 GbE (management) interfaces. SPARC SuperCluster is a complete, pre-configured clustered solution utilizing the following components: SPARC T4-4 Servers Oracle s SPARC T4-4 server offers a large memory capacity and a highly integrated design that supports virtualization and consolidation of mission-critical applications. There are two SPARC T4-4 servers in the half rack SPARC SuperCluster configurations and four in the full rack. Each SPARC T4-4 server has the following: o Four SPARC T4 Processors Each processor comes with eight cores and eight threads per core. The SPARC T4 utilizes the latest advanced S3 core design. o 1 TB of Memory Sixty-four of the latest 16 GB DDR3 memory DIMMs. o Eight Disk drives There are six 600GB SAS2 disk drives and two 300GB SSD disk drives. o Sun PCIe Dual Port QDR InfiniBand Host Channel Adapters Low latency 40Gb/s InfiniBand HCA in a modular hot-pluggable PCI Express (PCIe) ExpressModule form factor. There are four InfiniBand cards in each SPARC T4-4 server. o Sun Dual 10 GbE SFP+ PCIe 2.0 Low Profile network interface cards These NICs provide client access to each SPARC T4-4 server. They are in a modular hot-pluggable PCI Express (PCIe) ExpressModule form factors. There are four 10 GbE cards in each SPARC T4-4 server. Sun ZFS Storage 7320 Appliance Providing 60 TB of capacity, this appliance uses Flash-enabled Hybrid Storage Pools to improve application response times. Its performance scalability for file- Page 3

5 based I/O and ease-of-management make it a good fit for managing shared application data files within SPARC SuperCluster. Oracle Exadata Storage Servers Provides three critical technologies to improve database performance: Smart Scale Out Storage, Smart PCI Flash Cache, and Hybrid Columnar Compression. There are three Oracle Exadata Storage Servers in SPARC SuperCluster half rack configurations and six in the full rack. Oracle Exalogic Provides extreme performance for Java applications, Oracle Applications, and all other enterprise applications, and reduces application implementation and ongoing costs versus traditional enterprise application platforms and private clouds assembled from separately sourced components. Sun Datacenter InfiniBand Switches Provide a high throughput, low latency, and scalable fabric suitable for fabric consolidation of inter-process communication, network and storage. Infiniband delivers up to 63% higher Transactions Per Second for Oracle RAC over GbE networks. There are three InfiniBand switches in SPARC SuperCluster offering private connectivity within the system. Ethernet Management Switch Provides network management connectivity to the management ports on all servers and switches used in SPARC SuperCluster. (A Cisco switch is provided and customers can use their own switch if desired.) Integrated Virtualization - Enhance security, increase utilization and improve reliability through Oracle Solaris Zones and through Oracle VM Server for SPARC (previously also known as Logical Domains, or LDoms). Oracle Enterprise Manager Ops Center - Oracle Enterprise Manager Ops Center delivers a converged hardware management solution for SPARC SuperCluster T4-4 that integrates management across the infrastructure stack to help IT managers deploy and manage SPARC SuperCluster T4-4 more efficiently. Figure 1 shows the two SPARC SuperCluster T4-4 configurations currently offered. Page 4

6 Figure 1. SPARC SuperCluster T4-4 in half and full rack configurations SPARC SuperCluster configurations are fixed and contain a specific combination of servers, storage, network, and software elements. This is to ensure the highest quality for integration and testing during production. The platform can expand the amount of storage. SPARC T4-4 Servers The SPARC T4-4 server provides high throughput and computing density along with built-in virtualization and extreme scalability and is a highly efficient platform for deploying large scale, mission critical applications. Figure 2. SPARC T4-4 server Page 5

7 Architected to reduce planned and unplanned downtime, the SPARC T4-4 server includes advanced reliability, availability, and serviceability capabilities to avoid outages and reduce recovery time. These design elements are vital for a mission critical system such as SPARC SuperCluster. Design features that boost the reliability of SPARC T4-4 server include: Advanced CPU integration The SPARC T4 is an eight-core processor, with each core featuring eight threads. Each processor provides 4MB of L3 cache, critical in reducing the time required to read and process data. Each SPARC T4-4 server in SPARC SuperCluster comes with the maximum of four SPARC T4 processors. Extended-ECC Memory The memory system has redundant components built on to each memory DIMM that allows a DIMM to continue to operate even with partial failure. Utilizing 16GB DIMMs, each SPARC T4-4 server in SPARC SuperCluster comes with 1TB of memory. Fault-resilient power options and hot-swappable components Systems feature redundant, hot-swap power supply and fan units. Redundant storage can be created using hot-swap disk drives with disk-mirroring software. Hardware redundancy Redundant power, redundant fans, redundant data paths The 10GbE cards will be used by clients for access to SPARC SuperCluster compute nodes. In addition to the 10 GbE cards, also included are four Quad Data Rate (QDR) InfiniBand cards. These will be used for internal communication and access to the Oracle Exadata Storage Servers and the Sun ZFS 7320 Storage Appliance. The applications that run on the SPARC T4-4 server run in one of two types of logical domains: 1. Database Domain: A domain dedicated to running Oracle Database 11g Release 2, using Oracle Exadata Storage Servers for database storage. This domain must run Oracle Solaris Application Domain: A domain dedicated to running applications on either Oracle Solaris 11 or Oracle Solaris 10. Application Domains running Oracle Solaris 10 also support the use of Oracle Solaris Legacy Zones, for applications that require either a Solaris 8 or 9 environment. Application Domains running Oracle Solaris 11 also support the use of Oracle Solaris 10 zones. Exalogic Elastic Cloud Software runs only in an Application Domain on Oracle Solaris 11. Platform management is done via the service processor, or Integrated Lights Out Manager (ILOM 3.0). Integrated Lights Out Manager provides a command-line interface (CLI), a Web-based graphical user interface (GUI), and Intelligent Platform Management Interface (IPMI) functionality to aid outof-band monitoring and administration. The management software, Oracle Enterprise Manager Ops Center, communicates with the ILOM to manage and monitor the SPARC T4-4 servers. Oracle Exadata Storage Server X3-2 The Oracle Exadata Storage Server runs the Exadata Storage Server Software, which provides the unique and powerful software technology of the Oracle Exadata Database Machine, including Smart Scan, Smart PCI Flash Cache, Smart Flash Logging, I/O Resource Manager, Storage Indexes, and Page 6

8 Hybrid Columnar Compression. The hardware components of the Oracle Exadata Storage Server (also referred to as an Oracle Exadata cell) were carefully chosen to match the needs of high performance database processing. The Oracle Exadata software is optimized to take the best possible advantage of the hardware components and Oracle Database. Each Oracle Exadata cell delivers outstanding I/O performance and bandwidth to the database. When used in SPARC SuperCluster, the Oracle Exadata Storage Servers can only be accessed by Oracle Database 11g Release 2 running in the Database Domain. They cannot be used for any other purpose beyond Oracle Database 11g Release 2. Figure 3. Oracle Exadata Storage Server (Oracle Exadata Cell) The Oracle Exadata Storage Server delivers exceptional performance by way of the following critical technologies: Smart Scan: Processes queries at the storage layer, returning only relevant rows and columns to the database server. As a result, much less data travels over fast 40Gb InfiniBand interconnects dramatically improving the performance and concurrency of simple and complex queries. Smart PCI Flash Cache: Addresses the disk random I/O bottleneck problem by transparently caching 'hot' frequently accessed data to fast solid-state storage. It provides up to 30X improvement in response time for reads over regular disk and up to 20X more write performance; a hundred-fold improvement in IOPS for reads over regular disk; and is a less expensive higher capacity alternative to memory. Smart Flash Log: Takes advantage of the flash memory in Oracle Exadata storage to speed up log writes. Hybrid Columnar Compression: Can reduce the size of data warehousing tables by an average of 10 times, and archive tables by 50 times. This offers significant savings on disk space for primary, standby and backup databases, and will dramatically improve the performance of data warehousing queries. Each Oracle Exadata Storage Server comes with 1.6TB of flash storage and either twelve 600 GB 15,000 RPM High Performance SAS disks or twelve 3 TB 7,200 RPM High Capacity SAS disks. The High Performance SAS disk-based Oracle Exadata Storage Servers provide up to 3.25 TB of uncompressed useable capacity, and up to 1.8 GB/second of raw data bandwidth. The High Capacity SAS disk-based Oracle Exadata Storage Servers provide up to 16 TB of uncompressed useable capacity, and up to 1.3 GB/second of raw data bandwidth. When stored in compressed format, the amount of user data and the amount of data bandwidth delivered by each cell significantly increases. Page 7

9 While Exadata Storage Server can only be accessed by a productional Oracle Database 11g Release 2 database running in the Database Domain, Oracle Database 11g Release 2 can be run in an Application Domain on Oracle Solaris 11 for testing and development purposes. The DB could then use the Sun ZFS Storage 7320 Appliance to store the database. Oracle's Hybrid Columnar Compression (HCC) software technology could also be used on the Sun ZFS Storage 7320 Appliance to test HCC functionality before moving the database to the Database Domain and using the Exadata Storage Servers. If additional Exadata Storage Servers are required, the Exadata Storage Expansion Rack can be connected to the InfiniBand fabric for additional storage. Up to seven of these racks can be connected to a single SPARC SuperCluster rack. Sun ZFS Storage 7320 Appliance For shared file storage, SPARC SuperCluster includes a ZFS Storage 7320 Appliance, which features a common, easy-to-use management interface, and the industry s most comprehensive analytics environment. To deliver high performance using cost-effective components, the Sun ZFS Storage Appliance file system, Oracle Solaris ZFS, seamlessly optimizes access to the different types of media in the Hybrid Storage Pools. Oracle Solaris ZFS was designed to automatically recognize different I/O patterns and place data in the best storage media for optimal performance. The Sun ZFS Storage 7320 in SPARC SuperCluster features a two-node cluster configuration that enables high performance and high availability to maximize business productivity. This is the configuration used in a SPARC SuperCluster solution. Figure 4. Sun ZFS Storage 7320 Appliance Page 8

10 Figure 5. Real time analytics helps administrators increase storage optimization and reduce down time The ZFS 7320 can be made accessible from the Database Domain, and used for RMAN backups or flatfile staging. Networking Sun Datacenter InfiniBand Switch 36 InfiniBand technology has emerged as an attractive fabric for building large supercomputing grids and clusters. As an open standard, InfiniBand presents a compelling choice over proprietary interconnect technologies that depend on the success and innovation of a single vendor. Figure 6. Sun Datacenter InfiniBand Switch 36 Each SPARC SuperCluster T4-4 contains three Sun Datacenter InfiniBand Switch 36 switches, in either the full rack or half rack version, two of which are leaf switches (the third is used as a spine switch to connect two racks together). The two leaf switches are connected to each other to provide redundancy should one of the two leaf switches fail. In addition, each SPARC T4-4 server, Oracle Exadata Storage Server, and Sun ZFS Storage 7320 storage controller has connections to both leaf switches to provide redundancy in the InfiniBand connections should one of the two leaf switches fail. The following figure shows how redundancy is achieved with the InfiniBand connections between the SPARC T4-4 servers and the leaf switches in a half rack configuration. Page 9

11 Figure 7. Infiniband Network Connectivity Oracle Solaris SPARC SuperCluster supports both Oracle Solaris 10 and Oracle Solaris 11. Oracle Solaris 11 is used exclusively for the Database Domains and optionally in Application Domains,. Applications and third party applications that are not yet ready for Oracle Solaris 11 can run in an Application Domain on Oracle Solaris 10. Oracle Solaris includes the following features that make it the right operating system for SPARC SuperCluster: Advanced reliability Uptime is enhanced through comprehensive testing across an integrated solution stack and features such as predictive self-healing for hardware and software faults, data integrity with Oracle Solaris ZFS, and live observability with Oracle Solaris DTrace. Superior performance Oracle Solaris is optimized for throughput and scalability for the latest SPARC processor technologies and has achieved record-setting benchmarks for TPC-H, TPC-C, PeopleSoft, Oracle Business Intelligence Enterprise Edition, and many others. Built-in virtualization Oracle Solaris Zones and Oracle VM Server for SPARC (previously known as LDoms) along with other OS and network virtualization capabilities enable efficient consolidation for flexibility and performance without significant overhead. Pervasive security infrastructure Oracle Solaris provides the compartmentalization and control needed for multitenancy environments and enables governments and financial institutions to meet their strict requirements. Page 10

12 Committed support Oracle offers sustaining support for Oracle Solaris releases for as long as customers operate their systems, making it possible to keep software infrastructures in place for as long as it makes business sense. Fault Management and Predictive Self-Healing Oracle Solaris provides an architecture for building and deploying systems and services capable of fault management and predictive self-healing. The predictive self-healing feature in Oracle Solaris automatically diagnoses, isolates, and recovers from many hardware and application faults. As a result, business-critical applications and essential system services can continue uninterrupted in the event of software failures, major hardware component failures, and even software misconfiguration problems. Oracle Solaris Fault Manager. The Oracle Solaris Fault Manager Architecture (FMA) in Oracle Solaris collects data relating to hardware and software errors. This facility automatically and silently detects and diagnoses the underlying problem, with an extensible set of agents that automatically respond by taking the faulty component offline. Oracle Solaris Service Manager. The Oracle Solaris Service Manager Facility (SMF) feature in Oracle Solaris creates a standardized control mechanism for application services by turning them into first-class objects that administrators can observe and manage in a uniform way. These services can then be automatically restarted if an administrator accidentally terminates them, if they are aborted as the result of a software programming error, or if they are interrupted by an underlying hardware problem. Predictive self-healing and fault management can offline processor threads or cores in faults, retire suspect pages of memory, log errors or faults from I/O, or any other issue detected by the system. Oracle Solaris Zones Oracle Solaris provides a unique partitioning technology called Oracle Solaris Zones that enable creation of a virtualized operating system environment within a single instance of Oracle Solaris. A Zone is a virtualized operating system environment created within a single instance of Oracle Solaris. Oracle Solaris Zones can be used to isolate applications and processes from the rest of the system. This isolation helps enhance security and reliability since processes in one zone are prevented from interfering with processes running in another zone. Oracle Solaris Zones can also be used to run older versions of the Oracle Solaris operating system, databases, and applications on the latest generation SPARC servers. Hardware migrations and software updates can be deferred until times when both are more convenient. Legacy applications can benefit from faster hardware by migrating to new systems while running on an older OS revision. Resource management tools provided with Oracle Solaris enable administrators to allocate resources such as CPUs to specific applications or zones. CPUs in a multiprocessor system (or threads in a multi-core processor) can be logically partitioned into processor sets and bound to a resource pool, which in turn can be assigned to a zone. Resource pools provide the capability to separate workloads so that consumption of CPU resources does not overlap. They also provide a persistent configuration mechanism for processor sets and scheduling class assignment. In addition, the dynamic features of Page 11

13 resource pools enable administrators to adjust system resources in response to changing workload demands. Oracle VM Server for SPARC The SPARC T4-4 server also supports Oracle VM Server for SPARC (i.e. LDoms) virtualization technology. Oracle VM Server for SPARC provides full virtual machines that run an independent operating system instance and contain virtualized CPU, memory, storage, console, and cryptographic devices. Within the Oracle VM Server for SPARC architecture, operating systems such as Oracle Solaris 11 are written to the hypervisor, which provides a stable, idealized, and virtualizable representation of the underlying server hardware to the operating system in each domain. Each domain is completely isolated. The configurations for domains on the SPARC T4-4 servers in SPARC SuperCluster are slightly different from those often deployed in SPARC T4-4 servers used outside SPARC SuperCluster. Most notable is how I/O is presented to the domains. While a SPARC T4-4 server used outside a SPARC SuperCluster can have the I/O virtualized so it is shared across domains, the I/O is not virtualized for domains on a SPARC SuperCluster. All I/O on the SPARC T4-4 server in a SPARC SuperCluster is done via the Root Complex. In a PCI Express system, a Root Complex device connects the processor and memory subsystem to the PCI Express switch fabric comprised of one or more switch devices. In the case of the SPARC T4 server, this Root Complex is integrated in to the SPARC T4 processor itself. For SPARC SuperCluster, the SPARC T4-4 server supports a maximum of four logical domains, one per Root Complex. The Root Complex is the boundary so that the maximum performance can be achieved in every logical domain. When using an Database Domain, it will have either two or four Root Complexes. When using an Application Domain, it can have one or more Root Complexes. The following figure shows the mapping between the Root Complexes and the specific PCIe slots. Figure 8. Slot assignment per Root Complex The following figure shows a graphical representation of a half rack system with the DB and Page 12

14 Application Domain configuration. Figure 9. Domains mapped to a half rack system. Notice how the CPU and memory is configured separately from the Root Complex. When SPARC SuperCluster is installed, allocation granularity is 25% (8 CPU cores and 256GB of memory) per domain. This means the number and types of domains, number of Root Complexes, cores and memory must be decided upon at installation time. The customer can reassign CPU and memory after the installation using a standalone tool, setcoremem. The granularity is 4 CPU cores and 32GB of memory. Configurations SPARC SuperCluster is a complete solution that has been designed, tested and integrated to run the Oracle software stack, third party ISV software, as well as any customer developed software. Oracle has tested, tuned, and optimized the best possible hardware and software combinations so customers can take advantage of this integration resulting in a shorter deployment time and better utilization. Figure 10 shows the components of SPARC SuperCluster T4-4. The numbered items in the figure are identified in the table that immediately follows the figure. Page 13

15 Figure 10. Full SPARC SuperCluster T4-4 Rack TABLE 1. MAPPING OF COMPONENTS OF SPARC SUPERCLUSTER 1 Oracle Exadata Storage Servers (6) 5 Sun Disk Shelf 2 SPARC T4-4 servers (4) 6 Ethernet management switch 3 Sun ZFS Storage 7320 storage controllers (2) 7 Sun Datacenter InfiniBand Switch 36 spine switch 4 Sun Datacenter InfiniBand Switch 36 leaf switches (2) Page 14

16 TABLE 2. COMPONENT CONFIGURATIONS COMPONENT SPARC T4-4 SERVER CONFIGURATION Four SPARC T4 processors, each with eight cores (eight threads per core), operating at 3.0 GHz with 4MB of L3$ 1 TB of memory in 64 DDR3 DIMM slots 6 x 600GB internal SAS2 disk drives 2 x 300GB internal SSD disk drives o 4 x Sun Dual 10 GbE SFP+ PCIe 2.0 PCIe Express Module Adapter Supports Pluggable Transceivers (2 port) o 4 x Sun Quad Data Rate (QDR) InfiniBand PCIe Express Module Host Channel Adapter (2 port) ILOM Service Processor SUN ZFS STORAGE 7320 APPLIANCE Two controllers, each has: 2 x E5620 (4-core, 2.40G, 80W CPU) 6 x 16GB 1333 DIMMS (96G total) 1 x 2-port InfiniBand HCA (CX2) 2 x 500GB SATA disks 4 x 512GB read-optimized SSDs 1 x Thebe-Ext HBA 1 x Cluster card One disk shelf containing: 20 x 3TB SAS 7200rpm 4 x 73GB write-optimized SSDs THREE (HALF RACK) OR SIX (FULL RACK) ORACLE EXADATA STORAGE SERVERS Each Oracle Exadata Storage Server has a single Sun X3-2L server with: 2 sockets Xeon E5-2630L 6 2.9GHz 12MB L3 64GB Memory (16 x 4GB LV 1333Mhz DDR3 DIMMs) 1 * SAS-2 RAID HBA12 disks, either 600GB@15Krpm (High Performance) or 3TB@7200rpm (High Capacity) 4 x Sun Flash Accelerator F40 PCIe total 1.6TB 1 x QDR IB HCA (2 ports) THREE SUN DATACENTER INFINIBAND 36 PORT SWITCHES ETHERNET MANAGEMENT SWITCH Each 1U switch provides thirty six (36) QSFP InfiniBand Ports This 1U switch provides 48 Ethernet Ports, where each port has a wire-speed of 10/100/1000 BASE-T. Currently using a Cisco 4948 switch. The card slots have been assigned so as to minimize slot, bus, and controller bottlenecks. The SPARC T4-4 server configuration was designed to address a wide range of database and application services. The only optional component for the SPARC T4-4 servers is Fiber Channel (FC) cards. A maximum of eight per SPARC T4-4 server is allowed. Page 15

17 The T4-4 nodes are configured independently of each other. For HA, they are usually configured in pairs. This allows for balanced performance, and simplifies installation. The half rack has two nodes. The full rack has four nodes, where all the nodes can be configured the same, or differently, from the other nodes. Again, for balanced HA, configuring in pairs should be considered. A full rack also supports all four nodes having the same configuration. For domain types, multiple domains of each type are supported in a single node. Oracle Solaris Zones are supported in the Database Domain. This allows for multiple Oracle RAC and/or single-instance databases supported per Database Domain. There must be at least one Database Domain on each of the T4-4 nodes in a SPARC SuperCluster T4-4 half-rack, or on two nodes in a full-rack. The Oracle DB 11gR2 instances can be run in the Database Domain or in a zone in the Database Domain. An Oracle DB 11gR2 instance for test and dev can be run in a Solaris zone separate from an instance for productional use in a different zone. For customers that are already using an Oracle Exadata Database Machine, and are placing the Oracle SPARC SuperCluster in the same datacenter as the Exadata Database Machines, Exadata Storage servers in SPARC SuperCluster can be managed from the existing Exadata Database Machine. The Exadata Database Machine connects to the Exadata Storage Servers over a shared InfiniBand fabric. InfiniBand Connections There are specific cabling guidelines for SPARC SuperCluster. Accuracy is assured since all the components are assembled and cabled up at the factory before shipping the system to final destination. All InfiniBand connectivity is internal to SPARC SuperCluster. If connecting up to eight SPARC SuperCluster racks or Exadata Storage Expansion racks, then the InfiniBand fabric is extended from SPARC SuperCluster to those other racks. 10GbE Connections Just as with the InfiniBand cables, there are specific 10GbE cabling guidelines for SPARC SuperCluster. Cabling is between specific ports on the switches and the servers. The 10GbE connectivity is the only supported interface for clients to access SPARC SuperCluster. Managing SPARC SuperCluster Oracle Enterprise Manager Ops Center delivers a converged hardware management solution for SPARC SuperCluster that integrates management across the infrastructure stack. With advanced virtualization management and reporting capabilities, application-to-disk management, intelligent configuration management and more, Oracle Enterprise Manager Ops Center helps IT managers reduce complexity and streamline and simplify infrastructure management. The inclusion of Ops Center with every SPARC SuperCluster enables datacenter administrators to monitor and manage the storage, network, servers, Oracle Solaris, and virtualized environments from a single interface. This improves operational efficiency and lowers operational costs. Page 16

18 Figure 11. Management of the Oracle Red Stack Using Oracle Enterprise Manager Ops Center 12c, SPARC SuperCluster is managed as a specific engineered solution, not just a rack of servers and storage. Oracle Enterprise Manager Ops Center is the most comprehensive management solution for Oracle SPARC SuperCluster hardware infrastructure. Offering a single console to manage multiple server architectures and myriad operating systems, Oracle Enterprise Manager Ops Center can manage the components in SPARC SuperCluster using asset discovery, provisioning of firmware and operating systems, automated patch management, patch and configuration management, virtualization management, and comprehensive compliance reporting. Oracle Enterprise Manager Ops Center automates workflow and enforces compliance via policy-based management all through a single intuitive interface. With Oracle Enterprise Manager Ops Center, IT staff can implement and enforce datacenter standardization and best practices, regulatory compliance, and security policies while efficiently deploying infrastructure to meet business requirements. Figure 12 shows the intuitive browser-based user interface for Oracle Enterprise Manager Ops Center. Page 17

19 Figure 12. The Oracle Enterprise Manager Ops Center interface There are other tools that can be used to manage a SPARC SuperCluster. Some of these tools are native to the hardware (i.e., ILOM, Sun ZFS Storage Appliance), while others are part of the management software (i.e. Oracle Enterprise Manager Ops Center, Oracle Solaris Cluster Manager). Some of the tools require install and setup after SPARC SuperCluster has been installed. The ILOM exchanges data with Ops Center for highly functional and easy to use management. If customers wish they can access the ILOM directly. Other tools to manage SPARC SuperCluster: ILOM (i.e. the Service Processor) on SPARC T4-4 server o Command Line Interface (CLI), over serial or Ethernet o Browser User Interface (BUI), when using a browser for datacenter management o Once the server is set up, there should be little need to connect directly to the ILOM, except to connect to the Oracle Solaris domain console. o SNMP The MIB should be given to the existing datacenter management tool in the event Oracle Enterprise Manager Ops Center is not used to monitor SPARC SuperCluster. ILOM on Sun ZFS Storage 7320 Controller o Command Line Interface (CLI), over serial or Ethernet o Browser User Interface (BUI), when using a browser for datacenter management Page 18

20 o There should be very little reason to connect to the ILOM for platform management. Almost all management should be done with a browser connecting to the Sun ZFS Storage Appliance software. InfiniBand Switches and Ethernet Switch o Once Oracle Enterprise Manager Ops Center is properly setup during post-installation, it can be used to monitor the InfiniBand switches and the Cisco switch using SNMP. If Oracle Enterprise Manager Ops Center is not used at the customer site, then the MIBs for the switches should be added to the current datacenter management tool currently being used. Oracle Solaris Domains and Applications o It is highly recommended that Oracle Enterprise Manager Ops Center be used to manage all Oracle Solaris instances. Oracle Solaris does provide a built in SNMP for basic monitoring of events. The major element that will need to be installed and set up separately from the installation of SPARC SuperCluster is the specific application software that is unique to a given deployment. Infrastructure Services Within SPARC SuperCluster SPARC SuperCluster can provide a wide range of services within a single rack enclosure. Some of those services can include custom applications developed by customers, or software offered by Oracle. This includes Oracle Exadata software, or Oracle Fusion Middleware. Oracle Database 11g Release 2 The Database Domain on SPARC SuperCluster provides an optimal solution for all database workloads, ranging from scan-intensive data warehouse applications to highly concurrent online transaction processing (OLTP) applications. With its combination of smart Exadata Storage Server Software, complete and intelligent Oracle Database software, and the latest industry-standard hardware components, the Database Domain on the SPARC SuperCluster system is designed to deliver extreme performance in a highly available, highly secure environment. Oracle provides unique clustering and workload management capabilities so that the Database Domain is well suited for consolidating multiple databases into a single grid. Delivered as a complete pre-optimized, and pre-configured package of software, servers, and storage, the Database Domain is fast to implement, and can support large-scale business applications without time consuming configuration and tuning. The Database Domain on the SPARC SuperCluster system does not include any Oracle software licenses. Appropriate licensing of the following software is required when the SPARC SuperCluster system is used as a database server: Oracle Database Exadata Storage Server Software Page 19

21 In addition, Oracle recommends that the following software be licensed: Oracle Real Application Clusters Oracle Partitioning Oracle Exalogic Elastic Cloud Software Oracle Exalogic Elastic Cloud Software on SPARC SuperCluster utilizes the SPARC T4-4 servers, flash, Infiniband I/O fabric and storage technology with Oracle Solaris 11 operating systems to provide a tested, tuned, optimized and factory assembled platform for Oracle Fusion Middleware and Oracle's business applications portfolio. Exalogic Elastic Cloud Software is the unique set of software components and tools that have been optimized for Oracle's Fusion Middleware and business applications. As stated earlier, Oracle Exalogic Elastic Cloud software can only be run in a Solaris 11- based Application Domain. As Oracle currently offers the Exalogic Elastic Cloud X3-2, it is important to note the differences when deploying Oracle Exalogic Elastic Cloud Software on SPARC SuperCluster. There are really only a few key points of difference: SPARC SuperCluster utilizes SPARC server nodes and supports applications on Oracle Solaris 8, 9, 10 and 11. Exalogic Elastic Cloud X3-2 utilizes x86 server nodes and supports Oracle Linux (UEK bit only) and Oracle Solaris 11 Express (or Oracle Solaris 10 applications in Oracle Solaris 10 Zones). SPARC SuperCluster incorporates Oracle Exadata Storage Servers and is a recommended platform for Oracle RAC, supporting Data Warehouse and OLTP use cases. Exalogic Elastic Cloud X3-2 provides no optimization for Oracle Database and is designed to be deployed in conjunction with Oracle Exadata. It is important to note the functional differences between the Exalogic Elastic Cloud X3-2 solution and running the Oracle Exalogic Elastic Cloud Software on SPARC SuperCluster. Here are the features of Oracle Exalogic Elastic Cloud Software that are immediately available on SPARC SuperCluster: Oracle WebLogic Server JDBC and data source optimizations Oracle WebLogic Server cluster state replication optimizations Oracle WebLogic Server socket direct protocol (on the SPARC SuperCluster InfiniBand stack only) Coherence messagebus API support (on the SPARC SuperCluster InfiniBand stack only) Oracle Exalogic Elastic Cloud X3-2 is designed to provide a platform for application business logic and compute-intensive workloads, whereas Oracle Exadata Database X3-2 is designed for application databases and storage-intensive workloads. The ability to combine the application tier with the database on a single, integrated engineered solution demonstrates one of the greatest values of SPARC SuperCluster. By adding Oracle Enterprise Manager and Oracle Enterprise Manager Ops Center to manage the entire software and hardware stack, SPARC SuperCluster T4-4 provides a complete Page 20

22 solution. In many customer environments, it is highly desirable that all data services be configured in a high-availability configuration. The Oracle Solaris Cluster software addresses this requirement for a variety of customer deployment scenarios. Having deployed a cluster, the next step is for platform management and monitoring. Oracle Enterprise Management Ops Center is the recommended package to manage the servers, network, and storage. This can be used along side Oracle Enterprise Manager, which manages the Oracle application software offerings. Oracle Solaris Cluster To limit outages due to those single points of failure, mission-critical services need to be run in clustered physical servers, or Oracle Solaris Zones, that efficiently and smoothly take over the services from failing nodes or zones, with minimal interruption to the customer experience. While SPARC SuperCluster is designed with full redundancy at the hardware level, Oracle Solaris Cluster provides the best HA solution for Oracle SPARC servers running Oracle Solaris and applications. Tightly coupled with Oracle Solaris, Oracle Solaris Cluster detects failures without delay (zero-second delay), provides much faster failure notification, application failover, and reconfiguration time. Significantly reducing services recovery time achieves much faster resumption of IT services. Features in Oracle Solaris Cluster that speed this process on SPARC SuperCluster include: Integrates tightly with the Predictive Self Healing framework and supports applications controlled by SMF- in Oracle Solaris Zones and logical domains. Makes extensive use of Oracle s storage management and volume management capabilities. Supports Oracle Solaris ZFS as a failover file system and as a boot file system, allowing the use of ZFS storage as the single file system type used. Leverages ZFS features such as pooled storage, built-in redundancy, and data integrity. Uses I/O multipathing (MPxIO) in Oracle Solaris to represent and manage devices that are accessible through multiple I/O controller interfaces within a single instance of Oracle Solaris. Supports network IP multipathing to enhance resiliency and throughput in a clustered environment. Integrates with Oracle Enterprise Manager Ops Center. Offers secure administrative capabilities through RBAC capabilities in Oracle Solaris to enhance security. Virtualization with Oracle Solaris Cluster Oracle Solaris Cluster works seamlessly with Oracle s virtualization technologies to consolidate multiple applications within the same cluster of physical servers, optimizing resource use, ensuring availability of mission-critical services, and improving data integrity. A cluster can be a mix of whole T4-4 nodes, logical domain guests, or Oracle Solaris Zones. Page 21

23 Multiple options are available when using Oracle Solaris Zones. The failover approach treats zones as black boxes, which can be easily restarted or moved among cluster nodes. This solution supports Oracle Solaris 8, 9 and 10 Zones. Conclusion SPARC SuperCluster T4-4 is designed to help IT organizations consolidate multiple workloads in an environment that has been optimized for performance and availability. By incorporating SPARC SuperCluster T4-4 into their IT infrastructure, IT organizations can expect to: Consistently meet SLAs due to the performance and availability characteristics of SPARC SuperCluster Reduce IT management costs by managing a consolidated infrastructure that comes with an integrated management environment Rapidly deploy applications and databases without spending days or weeks optimizing and testing the hardware and software infrastructure Page 22

24 A Technical Overview of the Oracle SPARC SuperCluster T4-4 December 2012 Version 2.0 Author: Gary Combs Copyright 2012, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA U.S.A. Worldwide Inquiries: Phone: Fax: Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open Company, Ltd oracle.com