<Insert Picture Here> Exadata Hardware Configurations and Environmental Information

Transcription

1 <Insert Picture Here> Exadata Hardware Configurations and Environmental Information Revised July 1, 2011

2 Agenda Exadata Hardware Overview Environmental Information Power InfiniBand Network Ethernet Network 2

3 <Insert Picture Here> Exadata Hardware Overview 3

4 Exadata Hardware Complete, Pre-configured, Tested for Extreme Performance Database Servers (X2-2 and X2-8) Exadata Storage Servers InfiniBand Switches Ethernet Switch Pre-cabled Keyboard, Video, Mouse (KVM) hardware* Power Distribution Units (PDUs) Ready to Deploy * KVM not available in X2-8 models 4

5 Complete Family Of Exadata Machines For OLTP, Data Warehousing & Consolidated Workloads Storage Expansion Exadata X2-8 Exadata X2-2 Copyright 2011, Oracle Corporation and/or its affiliates 5

6 Exadata Database Machine X2-8 Full Rack Extreme Performance for Consolidation, Large OLTP and DW 2 Xeon-based Eight-processor Database servers (Sun Fire X4800) High Core, High Memory Database Servers 128 CPU cores (64 per server) 2 TB (1 TB per server) 10 GigE connectivity to Data Center 16 x 10GbE ports (8 per server) 14 Exadata Storage Servers X2-2 All with High Performance 600GB SAS disks OR All with High Capacity 2 TB SAS disks High Speed Flash 5.3 TB 3 Sun Datacenter InfiniBand Switch port Managed QDR (40Gb/s) switch 1 Admin Cisco Ethernet switch Redundant Power Distributions Units (PDUs) Avialable in Full Rack only 6

7 Exadata Database Machine X2-2 Full Rack Pre-Configured for Extreme Performance 8 Xeon-based Dual-procesor Database Servers (Sun Fire X4170 M2) 96 cores (12 per server) 768 GB memory expandable to 1152 GB (96GB per server expandable to 144 GB) 10 GigE connectivity to Data Center 16 x 10GbE ports (2 per server) 14 Exadata Storage Servers X2-2 All with High Performance 600GB SAS disks OR All with High Capacity 2 TB SAS disks High Speed Flash 5.3 TB 3 Sun Datacenter InfiniBand Switch port Managed QDR (40Gb/s) switch 1 Admin Cisco Ethernet switch Keyboard, Video, Mouse (KVM) hardware Redundant Power Distributions Units (PDUs) Available in Half and Quarter Rack configurations 7

8 Exadata X2-2 Memory Expansion Kit Optional kit for X2-2 customers who consolidate large numbers of databases and require more memory Changes memory configuration of the X4170 M2 database server from 96Gb (12 x 8 GB) to 144 GB (18 x 8 GB Minor reduction in performance due to lower memory bandwidth Adding memory drops the memory frequency from 1333 MHz to 800 MHz Available for X2-2 Full, X2-2 Half, and X2-2 Quarter Racks 8

9 Exadata Storage Expansion Full Rack For additional storage backups, historical data, unstructured data 18 Exadata Storage Servers X2-2 All with High Capacity 2 TB SAS disks 3 Sun Datacenter InfiniBand Switch port Managed QDR (40Gb/s) switch 1 Admin Cisco Ethernet switch Keyboard, Video, Mouse (KVM) hardware Redundant Power Distributions Units (PDUs) High Speed Flash 6.75 TB 432 TB of raw storage 216 CPU cores Available in Half and Quarter Rack configurations 9

10 Exadata Database Machine Hardware Summary X2-8 Full X2-2 Full X2-2 Half X2-2 Quarter Database Servers Database Grid Cores Database Grid Memory (GB) (max 1152) 384 (max 576) 192 (max 288) InfiniBand switches Ethernet switch KVM No Yes Yes Yes Exadata Storage Servers Storage Grid CPU Cores Raw Flash Capacity 5.3 TB 5.3 TB 2.6 TB 1.1 TB Raw Storage Capacity Usable mirrored capacity Usable Triple mirrored capacity High Perf 100 TB 100 TB 50 TB 21.6 TB High Cap 336 TB 336 TB 168 TB 72 TB High Perf 45 TB 45 TB 22.5 TB 9.5 TB High Cap 150 TB 150TB 75 TB 31.5 TB High Perf 30 TB 30 TB 15 TB 6.5 TB High Cap 100 TB 100TB 50 TB 21.5 TB Copyright 2011, Oracle Corporation and/or its affiliates 10

11 Exadata Storage Expansion Rack Summary Full Half Quarter InfiniBand switches Ethernet switch KVM Yes Yes Yes Exadata Storage Servers Storage Grid CPU Cores Raw Flash Capacity 6.75 TB 3.4 TB 1.5 TB Raw Storage Capacity Usable mirrored capacity Usable Triple mirrored capacity High Cap 432 TB 216 TB 96 TB High Cap 194 TB 97 TB 42.5 TB High Cap 130 TB 65 TB 29 TB Copyright 2011, Oracle Corporation and/or its affiliates 11

12 Exadata Storage Server X2-2 (Sun Fire X4270 M2) Processors Memory Disks Flash Disk Controller Network Remote Management Power Supplies 2 Six-Core Intel Xeon L5640 Processors (2.26 GHz) 24 GB (6 x 4GB) 12 x 600 GB 15K RPM High Performance SAS (hot-swap) OR 12 x 2 TB 7.2K RPM High Capacity SAS (hot-swap) 4 x 96 GB Sun Flash Accelerator F20 PCIe Cards Disk Controller HBA with 512MB Battery Backed Cache 2 InfiniBand 4X QDR (40Gb/s) Ports (1 Dual-port PCIe 2.0 HCA) 4 Embedded Gigabit Ethernet Ports 1 Ethernet port (ILOM) Redundant Hot-Swappable power supplies and fans 12

13 X2-2 Database Server (Sun Fire X4170 M2) Processors Memory Local Disks Disk Controller 2 Six-Core Intel Xeon X5675 Processors (3.06 GHz) 96 GB (12 x 8GB) expandable to 144 GB (18 x 8GB) 4 x 300GB 10K RPM SAS Disks (Hot-Swappable) Disk Controller HBA with 512MB Battery Backed Cache Network 2 x InfiniBand 4X QDR (40Gb/s) Ports (1 Dual-port PCIe 2.0 HCA) ) 4 x 1GbE Ethernet Ports 2 x 10GbE Ethernet SFP+ Ports (1 Dual-port 10GbE PCIe 2.0 network card based on the Intel GbE Controller technology) Remote Management Redundancy 1 Ethernet port (ILOM) Redundant Hot-Swappable power supplies and fans 13

14 X2-8 Database Server (Sun Fire X4800) Processors Memory Local Disks Disk Controller Network 8 x Eight-Core Intel Xeon X7560 Processors (2.26 GHz) 1 TB (128 x 8 GB) 8 x 300GB 10K RPM SAS Disks (Hot-Swappable) Disk Controller HBA with 512MB Battery Backed Cache 8 x InfiniBand 4X QDR (40Gb/s) Ports (4 Dual-port PCE 2.0 Express Modules) Two Network Express Modules (NEM), providing a total of 8 x 1GbE Ethernet Ports 8 x 10 GbE Ethernet SFP+ Ports (via 4 Fabric Express Modules (FEM) based Intel GbE Controller technology) Remote Management Redundancy 1 Ethernet port (ILOM) Redundant Hot-Swappable power supplies and fans 14

15 <Insert Picture Here> Exadata Hardware Environmental Information 15

16 Exadata Database Machine Environmental Power Cooling Airflow: frontto-back X2-8 Full X2-2 Full X2-2 Half X2-2 Quarter Max 17.0 kw (17.4 kva) 14.0 kw (14.3 kva) 7.2 kw (7.3 kva) 3.6 kw (3.7 kva) Typical 11.9 kw (12.2 kva) 9.8 kw (10.0 kva) 5.1 kw (5.2 kva) 2.7 kw (2.75 kva) Max Typical 58,050 BTU/hr (61,200 kj/hr) 40,630 BTU/hr (42,840 kj/hr) 47,800 BTU/hr (50,400 kj/hr) 33,400 BTU/hr (35,300 kj/hr) 24,600 BTU/hr (25,950 kj/hr) 17,400 BTU/hr (18,400 kj/hr) 12,300 BTU/hr (13,000 kj/hr) 9,200 BTU/hr (9,700 kj/hr) Max ~2,690 CFM ~2,200 CFM ~1,130 CFM ~550 CFM Typical ~1,880 CFM ~1,560 CFM ~840 CFM ~410 CFM Weight 2080 lbs (943.5 kg) 2131 lbs (966.6 kg) 1329 lbs (602.8 kg) 902 lbs (409.1 kg) Dimensions Operating Temperature IP addresses (H) x (W) x (D) (1998mm (H) x 600mm (W) x 1200mm (D)) 5 ºC to 32 ºC (41 ºF to 89.6 ºF), 10% to 90% relative humidity, non-condensing. Altitude Operating: Up to 3,048 m, max. ambient temperature is de-rated by 1 C per 300 m above 900 m InfiniBand Ethernet Network Drops (Minimum) External Connectivity 1GbE GbE (SFP+) InfiniBand

17 Exadata Storage Expansion Rack Environmental Power Cooling Airflow: frontto-back Full Half Quarter Max 12.6 kw (12.3 kva) 6.9 kw (7.1 kva) 3.4 kw (3.5 kva) Typical 8.8 kw (9.0 kva) 4.8 kw (5.0 kva) 2.4 kw (2.5 kva) Max Typical 43,000 BTU/hr (45,400 kj/hr) 30,100 BTU/hr (31,800 kj/hr) 23,600 BTU/hr (24,900 kj/hr) 16,500 BTU/hr (17,400 kj/hr) 11,600 BTU/hr (12,250 kj/hr) 8,100 BTU/hr (8,600 kj/hr) Max ~1,980 CFM ~1,090 CFM ~530 CFM Typical ~1,390 CFM ~760 CFM ~375 CFM Weight 2023 lbs (917.6 kg) 1275 lbs (578.3 kg) 875 lbs (396.8 kg) Dimensions Operating Temperature IP addresses (H) x (W) x (D) (1998mm (H) x 600mm (W) x 1200mm (D)) 5 ºC to 32 ºC (41 ºF to 89.6 ºF), 10% to 90% relative humidity, noncondensing. Altitude Operating: Up to 3,048 m, max. ambient temperature is de-rated by 1 C per 300 m above 900 m InfiniBand Ethernet Network Drops (Minimum) External Connectivity InfiniBand

18 Typical Data Center Deployment (Airflows) Rack Front Cold Air Inlet Rack Rear Hot Air Outlet Assuming perforated tiles (~ 400 CFM/tile) in front of the rack for cold air intake 4 floor tiles for Full Rack 3 floor tiles for Half Rack 1 floor tile for Quarter Rack Tiles can be arranged in any order in front of the rack as long as cold air from the tiles can flow into the rack Inadequate cold air flow could result in higher air inlet temperatures in the servers due to exhaust air recirculation Each data center has a unique environment, therefore a universal optimal deployment is not possible 18

19 <Insert Picture Here> Exadata Database Machine PDUs 19

20 Exadata Database Machine PDUS Each Rack comes with 2 (two) redundant Power Distribution Units (PDUs) Same PDUs for the Full, Half, and Quarter Rack Four choices for the Exadata Database Machine X2-2 and Exadata Storage Expansion Racks Low Voltage 15kVA Single Phase Americas / Japan / Taiwan Low Voltage 15 kva Three Phase Americas / Japan / Taiwan High Voltage 15 kva Single Phase EMEA / APAC (excluding Japan / Taiwan) High Voltage 15 kva Three Phase EMEA / APAC (excluding Japan / Taiwan) Four choices for the Exadata Database Machine X2-8 (different from the X2-2 and Storage Expansion Rack) Low Voltage 22 kva Single Phase Americas / Japan / Taiwan Low Voltage 24 kva Three Phase Americas / Japan / Taiwan High Voltage 22 kva Single Phase EMEA / APAC (excluding Japan / Taiwan) High Voltage 24 kva Three Phase EMEA / APAC (excluding Japan / Taiwan) 20

21 X2-2 and Storage Expansion Racks Low Voltage Single Phase PDU (Americas / Japan/ Taiwan) 2 (Two) 72Amp 15kVA PDUs Total of 6 (Six) Power Plugs (3 x 30Amp Plugs per PDU) VAC Data Center Receptacle Type - NEMA L6-30R 21

22 X2-2 and Storage Expansion Racks Low Voltage Three Phase PDU (Americas / Japan/ Taiwan) 2 (Two) 69Amp 15kVA PDUs Total of 2 (Two) Power Plugs (1 x 60Amp Plug per PDU) VAC 3ph Data Center Receptacle Type - IEC309-3P4W-IP67 60A 250VAC 3ph (Hubbell equivalent is HBL460R9W) 22

23 X2-2 and Storage Expansion Racks High Voltage Single Phase PDU EMEA / APAC (excluding Japan/ Taiwan) 2 (Two) 72 Amp 15kVA PDUs Total of 6 (Six) Power Plugs (3 x 25 Amp Plugs per PDU) VAC Data Center Receptacle Type - IEC309-2P3W-IP44 32A 250VAC (Hubbell equivalent is HBL332R6W) 23

24 X2-2 and Storage Expansion Racks High Voltage Three Phase PDU EMEA / APAC (excluding Japan/Taiwan) 2 (Two) 62.7Amp 15kVA PDUs Total of 2 (Two) Power Plugs (1 x 25Amp Plug per PDU) 220/ /415VAC Data Center Receptacle Type - IEC309-4P5W-IP44 32A 400VAC 3ph (Hubbell equivalent is HBL532R6W) 24

25 X2-8 Low Voltage Single Phase PDU (Americas / Japan/ Taiwan) 2 (Two) Amp 22kVA PDUs Total of 6 Power Plugs (3 x 36.8 Amp Plugs per PDU) VAC Data Center Receptacle Type - Hubbell CS8265C 25

26 X2-8 Low Voltage Three Phase PDU (Americas / Japan/ Taiwan) 2 (Two) 120Amp 24kVA PDUs Total of 4 (Four) Power Plugs (2 x 60Amp Plug per PDU) VAC 3ph Data Center Receptacle Type - IEC309-3P4W-IP67 60A 250VAC 3ph (Hubbell equivalent is HBL460R9W) 26

27 X2-8 High Voltage Single Phase PDU EMEA / APAC (excluding Japan/ Taiwan) 2 (Two) 96 Amp 22kVA PDUs Total of 6 (Six) Power Plugs (3 x 32 Amp Plugs per PDU) VAC Data Center Receptacle Type - IEC309-2P3W-IP44 32A 250VAC (Hubbell equivalent is HBL332R6W) 27

28 X2-8 High Voltage Three Phase PDU EMEA / APAC (excluding Japan/Taiwan) 2 (Two) 109Amp 24kVA PDUs Total of 4 (Four) Power Plugs (2 x 25Amp Plug per PDU) 220/ /415VAC Data Center Receptacle Type - IEC309-4P5W-IP44 32A 400VAC 3ph (Hubbell equivalent is HBL532R6W) 28

29 <Insert Picture Here> Exadata InfiniBand Network 29

30 InfiniBand Network Unified InfiniBand Network Storage Network RAC Interconnect External Connectivity (optional) High Performance, Low Latency Network 80 Gb/s bandwidth per link (40 Gb/s each direction) SAN-like Efficiency (Zero copy, buffer reservation) Simple manageability like IP network Protocols Zero-copy Zero-loss Datagram Protocol (ZDP RDSv3) Low CPU overhead (Transfer 3 GB/s with 2% CPU usage) Internet Protocol over InfiniBand (IPoIB) Looks like normal Ethernet to host software (tcp/ip, udp, http, ssh, ) 30

31 InfiniBand Network Uses Sun Datacenter 36-port Managed QDR (40Gb/s) InfiniBand switches Runs subnet manager and automatically discovers network topology Only one subnet manager active at a time 2 leaf switches to connect individual server IB ports 1 internal spine switch in Full Rack and Half Rack for scaling out to additional Racks Database Server and Exadata Servers Each server has Dual-port QDR (40Gb/s) IB HCA (X2-8 database server has 4 dual-port QDR Express modules ) Active-Passive Bonding Assign Single IP address per dual-port HCA Performance is limited by PCIe bus, so active-active not needed Connect one port from the HCA to one leaf switch and the other port to the second leaf switch for redundancy Connections pre-wired in the Factory 31

32 X2-2 Full Rack Server to Leaf Switch Leaf switch Port 1 Port 1 DB Servers (1,2,3,4) Exadata Servers (1,2,..,7) DB Servers (5,6,7,8) Exadata Servers (8,9,..,14) L1 L DB Servers (5,6,7,8) Exadata Servers (8,9,..14) DB Servers (1,2,3,4) Exadata Servers (1,2,..,7) Port 2 Port 2 Leaf switch InfiniBand network with Redundancy Servers connect to the two leaf switches Active & Passive ports balanced across switches Full Bandwidth even if switch fails Connections pre-wired at factory 32

33 X2-2 Half Rack Server to Leaf Switch DB Servers (1,2,3,4) Exadata Servers (1,2,3,4,5,6,7) 11 7 Port 1 L1 L2 11 DB Servers (1,2,3,4) Exadata Servers (1,2,3,4,5,6,7) Port 2 Leaf switch Leaf switch Connect Port1 from DB & Exadata Servers to Switch L1 Connect Port2 from DB & Exadata Servers to Switch L2 Interconnect two switches 7 links Pre-wired at Factory 33

34 X2-2 Quarter Rack Server to Leaf Switch Port 1 DB Servers (1,2) Exadata Servers (1,2,3) 5 7 L1 L2 Leaf switch Leaf switch 5 DB Servers (1,2) Exadata Servers (1,2,3) Port 2 Connect Port1 from DB & Exadata Servers to Switch L1 Connect Port2 from DB & Exadata Servers to Switch L2 Interconnect two switches 7 links Pre-wired at Factory 34

35 X2-8 Full Rack Server to Leaf Switch Leaf switch Port 1 Port 1 DB Server1 (EM 1,2,3,4) Exadata Servers (1,2,..,7) DB Server2 (EM 1,2,3,4) Exadata Servers (8,9,..,14) L1 L Leaf switch DB Server2 (EM 1,2,3,4) DB Server1 (EM 1,2,3,4) Exadata Servers (8,9,..14) Exadata Servers (1,2,..,7) Port 2 Port 2 InfiniBand network with Redundancy Servers connect to the two leaf switches Active & Passive ports balanced across switches Full Bandwidth even if switch fails Connections pre-wired at factory 35

36 Storage Expansion Full Rack Server to Leaf Switch Port 1 Port 1 Exadata Servers (1,2,..,9) Exadata Servers (10,11,..,18) Leaf switch 9 L1 9 L2 Leaf switch Exadata Servers (10,11,..18) Exadata Servers (1,2,..,9) Port 2 Port 2 InfiniBand network with Redundancy Servers connect to the two leaf switches Active & Passive ports balanced across switches Connections pre-wired at factory 36

37 Storage Expansion Half Rack Server to Leaf Switch Port 1 Exadata Servers (1,2,,9) 9 7 L1 L2 Leaf switch Leaf switch 9 Exadata Servers (1,.9) Port 2 Connect Port1 from DB & Exadata Servers to Switch L1 Connect Port2 from DB & Exadata Servers to Switch L2 Interconnect two switches 7 links Pre-wired at Factory 37

38 Storage Expansion Quarter Rack Server to Leaf Switch Exadata Servers (1,2,3,4) 7 Port 1 4 L1 L2 4 Exadata Servers (1,2,3,4) Port 2 Leaf switch Leaf switch Connect Port1 from DB & Exadata Servers to Switch L1 Connect Port2 from DB & Exadata Servers to Switch L2 Interconnect two switches 7 links Pre-wired at Factory 38

39 Spine and Leaf InfiniBand Switch DB and Exadata Servers Ports S 1 1 Spine switch Leaf switch L1 7 Leaf switch L2 Use 3 rd switch (S) as spine switch for expansion to multiple racks Available in Half and Full Racks Connect each leaf switch to spine switch (1 links wide) Interconnect leaf switches with each other (7 links wide) Enough bandwidth even if a switch fails Pre-wired at factory 39

40 Scaling Out to Multiple Full Racks Single InfiniBand Network Switch to a Fat Tree Topology Valid up to 8 Racks with no external switches Can go to 36 racks with 2 external switches Every leaf switch inter-connected with every spine switch Leaf switches not connected with other leaf switches Spine switches not connected with other spine switches Database and Exadata Server cabling unchanged. Inter-rack cabling done at installation time Up to 3 Racks adjacent to each other Extra 3m and 5m cables already included in Spares Kit Greater than 3 Racks Longer Optical cables need to be purchased 40

41 Two Rack Case Fat Tree Topology S S L1 L2 L1 L2 Rack 1 Rack 2 Leaf switches L1, L2 (2 per rack) Remove inter-switch links between leaf switches Spine switches S (1 per rack) Connect each leaf switch to every spine switch (4 links wide) 41

42 Multiple Rack Case Up to 8 Racks S S S L1 L2 L1 L2 L1 L2 Rack 1 Rack 2 Distribute 8 links from every leaf switch to every spine switch 2 rack case 4 links from every leaf to spine switch 3 rack case - 3,3,2 links from each leaf to the 3 spine switches 4 rack case 2 links from each leaf to spine switch Rack N 5,6,7,8 rack case 1 or 2 links from each leaf to spine (8 links from each leaf) Greater than 8 rack requires external switches 42

43 Scaling Out to 9 to 36 Racks Single InfiniBand Network Require 2 external spine 36-port InfiniBand switches Add another level to the Fat Tree topology Divide Racks into groups each having 2-8 racks Group related racks together for example, racks that run a common database or are part of a common cluster Inter-connect racks within each group using the two level Fat Tree topology Inter-connect the groups via the two external spine switches Connect each internal spine switch in each rack to the two external spine switches Can be done online. 43

44 Multiple Rack Case 9 to 36 Racks External Spine switches Group 1 (2 to 8 racks) Group N (2 to 8 racks) Interconnect internal spine switch with both external spine switches using 1 link each Bandwidth from/to each group is 80 Gb/s * number of racks in the group 44

45 Inter-connecting Quarter Racks Allow limited situations where a Quarter Rack can be inter-connected with other rack Case 1 - Connect two Quarter Racks The pre-requisite is that at least 4 ports reserved for external connectivity are open on each leaf switch Case 2 Connect one Quarter Rack with one other Half or Full Rack The pre-requisite is that at least 4 ports reserved for external connectivity are open on each leaf switch Case 3 Connect one Quarter Rack to two or more racks (all of which are either Exadata Half or Full Racks) 45

46 Case 1 Two Quarter Racks L1 2 L L2 2 L2 Quarter Rack 1 Quarter Rack 2 Leaf switches L1, L2 (2 per rack) Maintain the existing 7 inter-switch links between the leaf switches within each rack Inter-connect the leaf switches between racks with 2 links each Use the ports reserved for external connectivity 46

47 Case 2 Quarter Rack with One Half or Full Rack L1 2 L S1 L2 L2 1 2 Quarter Rack Half or Full Rack Leaf switches L1, L2 (2 per rack). The Half or Full Rack has a spine switch S1 Maintain the existing 7 inter-switch links between the leaf switches within each rack Inter-connect the leaf switches between racks with 2 links each Use the ports reserved for external connectivity 47

48 Case 3 Quarter Rack with Two or more Racks S S L1 L2 L1 L2 L1 L2 Quarter Rack Full/ Half Rack 1 Full / Half Rack N The 2 or more racks that the Quarter Rack will connect to must be all Full or Half Racks inter-connected via a Fat-tree topology Remove the 7 inter-switch links between the leaf switches within the Quarter Rack Connect each leaf switch of the Quarter Rack to each spine switch of the other Full / Half Racks with 2 links each (if there are less than 4 racks) or 1 link each (if there are more than 4 racks) 48

49 InfiniBand Network External Connectivity External connectivity ports for Connecting to media servers for Tape backup Data Loading Client / Application Access Validated InfiniBand cable lengths Up to 5m Passive Copper 4X QDR QSFP cables Up to 100m Fiber Optic 4X QDR QSFP cables (more expensive) Use available ports on the two Leaf switches 12 per Rack For HA connections, connect 1 port to one Leaf switch and another to the second Leaf switch 49

50 InfiniBand Network Addresses bondib0 for Database Servers X2-2 X2-8 Storage Expansion Full Half Quarter Full Full Half Quarter bondib0 for Exadata Storage Servers Total Can be private subnet assuming IB network not exposed outside the Database Machine 50

51 <Insert Picture Here> Exadata Ethernet Network 51

52 Admin Ethernet Network Database Server (ILOM) Database Server ( mgmt port) Exadata Storage Server (ILOM) Exadata Storage Server ( mgmt port) IB Switch (Mgmt port) Cisco switch 1 To Data Center Network 1 KVM switch (only in X2-2 models) Connect to Data center admin network Admin Ethernet switch pre-cabled connections Connected to hardware mgmt ports ILOM port on the Exadata Storage Server ILOM port on the Database Server Connected to software mgmt ports 1 Ethernet port on Exadata Storage Server 1 Ethernet port on Exadata Storage Server For software admin, root access,.. Connected to InfiniBand Switch management ports User must connect KVM console in the X2-2 models directly to data center network User can connect network ports in the PDUs directly to the data center network for monitoring 52

53 External Connectivity Ethernet Per Database Machine Admin Access 1 port from Admin Ethernet switch 1 port from KVM Switch (for X2-2 models) 2 from the PDUs Database / Client / Application Access Minimum 1 GbE or 1 10GbE port per Databaser Server X4170 M2 Database Server (x2-2 models with M2 servers) Additional 2 x 1GbE Ethernet ports available 2 x 10GbE Ethernet SFP+ ports available Use them for bonded client / application access or for additional connectivity X4800 Database Server (x2-8 models) Additional 7 x 1GbE Ethernet ports available 8 x 10GbE Ethernet SFP+ ports available Use them for bonded client / application access or for additional connectivity 53

54 Cable Choices - 10 GbE Network Ports Available on the X4170 M2 database servers in the X2-2 models and in the X4800 database servers in X2-8 model Each X4170 M2 database server has 2 x 10GbE SFP+ network ports Each X4800 database server has 8 x 10 GbE SFP+ network ports Can use them for bonded client / application access or for additional connectivity like backup, loading data. Customers must purchase the 10GbE cables and/or SFP+ transceivers separately (not included with the machine) Two choices of cables Copper cables (with built-in SFP+ tranceivers) upto 10m in lengths TwinAx copper cable 1m (Oracle part# X2130-1M) TwinAx copper cable 1m (Oracle part# X2130-3M) TwinAx copper cable 1m (Oracle part# X2130-5M) TwinAx copper cable 1m (Oracle part# X M) Optical cables (needs separate SR or LR SFP+ transceiver for each port) 10 GbE Transceiver SR (SFP+) (Oracle part# X2129A-Z) 10 GbE Transceiver LR (SFP+) (Oracle part# X5562A-Z) Purchase LC-LC terminated optical cables from a third party vendor. 54

55 Additional Ethernet Network Requirements Additional Ethernet IP addresses needed if multiple clusters are deployed on the Database Machine Full Rack, Half Rack, or Quarter Rack 3 IP addresses per additional cluster for SCAN addresses Static IP addressing only (no DHCP) All Ethernet IP addresses must be registered in DNS 55

56 Management Client Access Ethernet Network Addresses X2-2 X2-8 Storage Expansion Full Half Qtr Full Full Half Qtr ILOM for Database Servers ILOM for Exadata Cells Eth0 (mgmt) for DB Servers Eth0 (mgmt) for Exadata Cells Mgmt port for IB switches IP address for KVM IP address for Ethernet Switch IP address for PDUs Mgmt Subnet Total Client Access Database Servers VIPs for Database Servers SCAN Addresses (per Cluster) Client Access Total Total

57 57