Data Center Facility Analysis Project. Final Presentation Wednesday, July 22, :30 3:30 PM

Transcription

1 Data Center Facility Analysis Project Final Presentation Wednesday, July 22, :30 3:30 PM 1 Commonwealth Technology Center Conference Rooms 1 & 2

2 Enterprise Server Farm (ESF) Facility Data: 1 Technology Park, Harrisburg Built in 1991 Data Center = 12,369 sf Current Server Count = 862 servers Current Uptime Institute Rating: Tier II (redundant capacity components) Growth Capacity: At current build: +131 servers At full build out: +612 servers 2

3 ESF General Observations General Notes and Areas for Improvement: Facility is presently operating above the recommended operating limit of the UPS system Power infrastructure expansion project does not provide for comparable increase of HVAC capacity Power infrastructure expansion project does not provide for additional secondary power distribution capacity (PDU) Facility is Tier II Resilient (All HVAC is powered from a single ATS) Resilient HVAC capacity is less than resilient capacity of power expansion project Several HVAC elements are beyond their expected service life and should be evaluated for replacement Facility Recommendations: A primary hosting site for mission critical and other Commonwealth applications Current and future growth capacity suited for consolidation of systems from other sites 3

4 ESF Capacity Summary Enterprise Server Farm Data Center Operation at UPS limit with Power Expansion 2N Utility 2N 81% 89% 131% N+1 Space (RUs) 150% 0% 82% 79% 80% 80% 2N 134% N+1 UPS 2N 123% Notes: 1. Space limit presumes sw ing space reserved for 5 year refresh cycle + 5% contingency 2. Space consumption presumes historical policy deployment w ithout compaction or virtualization. % Consumed Operating Limit % of Rating System Enterprise Server Farm Resiliency Resilient Capacity Current Consumption UPS 1 N 450kW 406kW UPS 1 PDUs 2N 650kVA 392kVA UPS 2 N 117kW 70kW UPS 2 PDUs 2N 300kVA 71kVA CRAC N+1 170T 138T Air-cooled Condensers N+1 192T 152T Utility Transformer N 1500kVA 1233kVA Generator N 1000kW 838kW Temporary Generator N 500kW 79.kW Limiting Infrastructure Resiliency Resilient Capacity Assumed Operating Limit (%) Enterprise Server Farm Assumed Operating Limit Current Consumption Growth Potential (expressed as kw of IT Compute load) Growth Potential (expressed as % of current consumption) CRAC N T 90% 153 T 138 T 53 11% Condensers N T 90% T 152 T 66 14% Expansion PDU config 2N 800 kva 80% 640 kva 463 kva % Expansion UPS 2N 900 kw 80% 720 kw 476 kw % Approximate Growth - Number of Discrete Servers (+ supporting storage & network) 131 Comments Presumes power expansion project complete 4

5 ESF Tier III Conformance Electrical HVAC Site Subsystem Utility Service Lightning Protection & TVSS Primary Power Disrtibution UPS Systems UPS Batteries Engine Generators Load Bank Secondary (Critical) Power Distribution Grounding Description Comments & Related Subsystems Remediation Dual Utility Feed Data center building equipped with passive lightning protection Multi-tier TVSS deployed UPS powered at 2N; All HVAC powered from single ATS 2N UPS 2N Batteries sized to support rated load for 15 minutes 2N Generators but "Tie" can not be maintained in service No load bank 2N to IT load with some static switch PDUs deployed Counterpoise, pedistal grounding & perimeter ground in power room; no counterpoise around expansion generator housings CRAC N+1 at current load; Less than N at resilient Raised Floor Equipment rating of UPS Cooling Conveyance Not Applicable- Units contain compressors Present UPS room cooling is non-redundant; Power Equipment UPS Room Cooling Expansion electrical room e/w ~ N CRAC capacity Conveyance Not Applicable- Units contain compressors Eqpt- Data Cnter Condenser capacity < N for power expansion Dist'n. & Condensers paired 1:1 with CRAC units- design concept is Conv- Data concurrently maintainable but capacity is non-resilient Heat Rejection Eqpt- Pwr Power Expansion e/w < N condenser capacity Exp Bldg Dist'n. & Condensers paired 1:1 with CRAC units- design concept is Conv- Pwr concurrently maintainable but capacity is non-resilient Curret Uptime Tier Classification Notes: 1 Evaluation presumes power expansion project completion 2 Tier 3 considers generators to be the primary electrical source Tier II Concurrent Maintainability Scorecard Enterprise Server Farm Feeds are aerial to site & non-diverse on-site UL listing status unknown Main-Tie-Main configuration does not permit out-of-service maintenance of "Tie"; Utility & generator share bus Main-Tie-Main configuration does not permit out-of-service See Primary distribution remediation maintenance of "Tie" as utility & generator share bus Single starting batteries on existing generator Repaired equipment is testable only into production load Requires rigorous circuit management to ensure single corded IT kit is connected to static switched PDU; risk level elevated due to use of bulk STS Presently no EPO- to be implemented as p/o power expansiondetails unknown 1. Convert Main-Tie-Main configuration to Main-Tie-Tie-Main 2. On a phased basis, replace vintage CRAC units with units equipped with manually selectable, dual source capability 3. Furnish & install new 1200A HVAC distribution board 4. Furnish & install ten (10) manual transfer switches (MTS) 5. Disconnect circuits to all server room condenser units 6. Disconnect circuits to one (1) 30T condenser unit, one (1) UPS condenser unit, and two (2) 30T UPS CRAC units 7. Add circuits between one (1) 30T condenser unit, one (1) UPS condenser unit, and two (2) 30T server room condensers and the new HVAC distribution board 8. Add circuits between dual source CRAC units to New HVAC distribution board 9. Connect nine (9) MTS to both HVAC distribution boards 10. Connect one (1) MTS to both HVAC distribution boards and the 30T enterprise server room CRAC 11. Connect nine (9) MTS to condenser units Required to provide resiliency to support growth only; furnish & install 1, 10T CRAC in Enterprise Systems room and 2, 20T CRAC units in main server room Additionally, non-redundant exhaust fans to be deployed in wet Required to provide resiliency to support growth only; furnish & install 1, 30T CRAC in new cell battery rooms electrical room All HVAC powered from single ATS, et al Required to provide resiliency to support growth only; furnish & install 1, 10T Condenser unit and 2, 20T Condenser units Required to provide resiliency to support growth only; part of CRAC/Condenser installation Required to provide resiliency to support growth only; furnish & install 1, 30T Condenser unit Required to provide resiliency to support growth only; part of CRAC/Condenser installation Total estimated resiliency remediation cost = $1,566,000 + HVAC resiliency to support growth = $845,000 Rating Legend Concurrently Maintainable Increased Risk Shutdown Required Not Applicable 5

6 Facility Data: 43 North Circle Drive, Harrisburg Built in 1983 Data Center = 22,000 sf Current Server Count = 330 servers Current Uptime Institute Rating: Tier I (basic site infrastructure)* Growth Capacity: At current build: +210 servers At full build out: +668 servers Data PowerHouse (DPH) 6

7 DPH General Observations General Notes and Areas for Improvement: DPH is currently Tier I rated due to non-redundant cooling of the UPS and battery rooms; otherwise, it is Tier II resilient most notably owing to HVAC powering Input power restrictions prevent utilization of full UPS capacity The building is not equipped with a lightning protection system Facility grounding is not documented nor evident Data center fire suppression is via wet pipe system Primary electrical backbone equipment is within 5 years of its expected service life The terminals of the wet cell batteries are exposed and subject to accidental contact (although they are in a restricted access area) Staff expects the data center to be at capacity limits by mid 2010 Facility Recommendations: A primary hosting site for mission critical and other Commonwealth systems requiring high availability and 24x7 monitoring Current and future growth capacity suited for consolidation of systems from other sites 7

8 DPH Capacity Summary N+1 Utility N+1 Data Power House Data Center Full UPS Rating 59% 75% Space (RUs) 100% 58% 0% 47% 43% 65% 2N 81% 2N Notes: 1.Space consumption presumes historical policy deployment w ithout compaction or virtualization. 2. Pow er consumption adjusted for summer peak 3. UPS output limited by input pow er available via 1200A distribution System Data Power House Resiliency Resilient Capacity Current Consumption UPS 2N 1000kW 545kW PDU 2N 900kVA 608kVA CRAC N+1 440T 157T Drycoolers N+1 440T 173T Utility Transformers N kVA 1299kVA Generator N kW 1199kW N+1 N+1 (22/28) % Consumed Operating Limit Limiting Infrastructure Resiliency Resilient Capacity Assumed Operating Limit (%) Data PowerHouse Assumed Operating Limit Current Consumption Growth Potential (expressed as kw of IT Compute load) Growth Potential (expressed as % of current consumption) UPS 2N 1000 kw 65% 650 kw 545 kw % PDU 2N 900 kva 90% 810 kva 608 kva % Generator N kw 90% 1800 kw 1199 kw % Utility Transformer N kva 90% 2250 kva 1299 kva % Approximate Growth - Number of Discrete Servers (+ supporting storage & network) 210 Comments UPS supply circuit restricts UPS output to 65% of full resilient rating 8

9 DPH Tier III Conformance Electrical HVAC 9 Site Subsystem Utility Service Lightning Protection & TVSS Primary Power Disrtibution UPS Systems UPS Batteries Engine Generators Load Bank Secondary (Critical) Power Distribution Grounding Raised Floor Cooling UPS Room Cooling Equipment Conveyance Equipment Conveyance Equipment- Data Ctr. Dist'n. & Conv- Data Ctr. Heat Rejection Equipment- UPS Room Dist'n. & Conveyance- UPS Room Curret Uptime Tier Classification Description Comments & Related Subsystems Remediation Dual Utility Feed from two substations Data center building not equipped with lightning protection; Multi-tier TVSS deployed UPS powered at 2N; All HVAC powered from two ATS; HVAC is < 2N resilient 2N UPS 2N Batteries sized to support rated load for 15 minutes N+1 Generators but common paralleling bus; Generators share single radiator for heat rejection; Redundant starting batteries No load bank 2N to IT load via static switch PDUs Overall facility grounding is indeterminate- documentation specifies data center grid referenced to utility transformer neutral CRAC N+1 (22/28) at full UPS rating Not Applicable- Units contain compressors UPS room cooling is non-redundant- single 40T air handler/chiller Not Applicable- Units contain compressors Dry N+1 (22/28) redundancy Glycol conveyance comprised of 5 independent risers matching the 22/28 equipment resiliency deployment UPS room cooling is non-redundant- single 40T air handler/chiller Notes: 1 Tier 3 considers generators to be the primary electrical source Not Applicable- Packaged Unit with air distribution only Tier I Concurrent Maintainability Scorecard Data PowerHouse Feeds are aerial to site & non-diverse on-site Out-of-service maintenance of "Tie" in Main-Tie-Main configuration possible with emergency power system configuration Input source restrictions limit UPS to ~650kW output Must use utility as unprotected power source during bus & radiator maintenance Repaired equipment is testable only into production load Risk level elevated due to use of bulk rather than local STS for single cord devices Wet cell battery racks bonded to building steel Additionally, non-redundant exhaust fans in battery rooms Non-redundant cooling of UPS room Rating Legend Concurrently Maintainable Increased Risk Shutdown Required Not Applicable 1. Furnish & install one (1) automatic transfer switch (ATS) 2. Furnish & install three (3) manual transfer switches (MTS) 3. Furnish & install one (1) distribution panel for CRAC & drycooler system A 4. Install circuits from new ATS to distribution panel and generator distribution bus 5. Install circuits from new ATS to both main distribution panels 6. Install circuits between new ATS and three (3) new MTS 7. Disconnect circuit distribution panel CR2 Furnish & install 40-ton air conditioning unit + redundant fans for UPS room (current project) See UPS room equipment remediation Total estimated resiliency remediation cost = $1,590,000

10 Dept. of Env. Protection (DEP) Facility Data: 400 Market Street, Harrisburg Built in 1992 Data Center = 2,200 sf Current Server Count = 100 servers Current Uptime Institute Rating: Tier II (redundant capacity components) Growth Capacity: At current build: +68 servers At full build out: +160 servers 10

11 DEP General Observations General Notes and Areas for Improvement: Facility is Tier II resilient (non-redundant primary power backbone and non-redundant glycol conveyance) Data center fire suppression is via wet pipe system The center is protected by a single, non-redundant generator Facility grounding is not documented nor evident (although data center pedestals and cabinets are grounded The data center floor slab is rated for 100 pounds per square foot live loading TVSS is not deployed in the electrical distribution system Facility is located near an active railroad Per FEMA map # A dated May 2, 1977, the building is located on the boundary of the 100 and 500 year flood plains Facility Recommendations: Continue its role supporting critical agency systems Should be considered as a potential consolidation site for systems from other agencies that do not require high uptime or 24x7 monitoring 11

12 DEP Capacity Summary Dept. of Environmental Protection Data Center Operation at UPS Rating N Building Utility 2N Air Cooled Condensers 58% 35% Space (RUs) 100% 79% 55% 80% 2N 80% 0% 37% 59% 57% 80% Panel UP 2N Notes: 1. Space limit presumes sw ing space reserved for 5 year refresh cycle + 5% contingency 2. Operating limits presumed to be 90% of nameplate rating except as noted 3. Pow er consumption adjusted for summer peak 4. DCNR load on Panel UP presumed to be 152 A (100kVA rating) System Department of Environmental Protection Resiliency Resilient Capacity Current Consumption UPS 2N 104kW 39kW RPP 2N 900kVA 608kVA Panel UP N 600A 241A CRAC N+1 40T 11T Air-cooled Condensers N+1 94T 12T Building Utility Transformers 2N 2000kW 893kW Generator N 1100kW 384 N+1 % Consumed Operating Limit Limiting Infrastructure Resiliency Resilient Capacity Department of Environmental Protection Assumed Operating Limit (%) Assumed Operating Limit Current Consumption Growth Potential (expressed as kw of IT Compute load) Growth Potential (expressed as % of current consumption) RPP 2N 225 A 90% 203 A 108 A 34 88% UPS 2N 104 kw 80% 83 kw 39 kw % CRAC N+1 40 T 90% 36 T 11 T % Approximate Growth - Number of Discrete Servers (+ supporting storage & network) 68 Comments 12

13 DEP Tier III Conformance Electrical Site Subsystem Utility Service Lightning Protection & TVSS Primary Power Disrtibution UPS Systems UPS Batteries Engine Generators Load Bank Concurrent Maintainability Scorecard Department of Environmental Protection Description Comments & Related Subsystems Remediation Dual Utility Feed from two PP&L substations Data center building equipped with passive lightning protection; TVSS not noted nor indicated on drawings Single circuit powers UPS through panel UP1; All HVAC powered from single panel Feeds are underground to facility UL listing status unknown 1. Reconfigure power sources to Main-Tie-Tie-Main configuration by adding a 4000A breaker 2. Furnish & install one (1) 600A automatic transfer switch (ATS) 3. Furnish & install one (1) 400A HVAC distribution panel 4. Furnish & install one (1) distribution panel (UPB) 5. Furnish & install one (1) 100A manual transfer switch (MTS) 6. Install circuits from new 600A ATS to switchboard, new generator, new HVAC distribution panel, & new panel UPB 7. Install circuits one (1) new 2.5-ton CRAC unit and one (1) new condenser unit to each HVAC distribution panel 8. Install circuits from the new 50A MTS to each distribution panel, one (1) new CRAC unit, & one (1) new condenser 9. Install circuits from the new 100A MTS to each distribution panel 10. Remove circuits between one existing 20-ton CRAC unit and condenser and the existing HVAC distribution panel 11. Install circuits between one (1) 20-ton CRAC and condenser and the new HVAC distribution panel 2N UPS 2N Batteries sized to support rated load for 15 minutes Single Generator, non-redundant starting battery Not dedicated to technology operations 1. Furnish & install one (1) 500kW emergency generator No load bank Repaired equipment is testable only into production load Secondary (Critical) Power Distribution 2N to IT load; static switches not deployed Circuits powering single cord IT devices can not be maintained; UPS outputs via common panel with tie 1. Install circuits from one (1) existing UPS to panel UPB and to new UPS output panel; remove circuits to existing UP panel 2. Remove circuits from one (1) UPS to the UPS output panel 3. Install circuit between the new UPS output panel and the existing 225A distribution panel HVAC 13 Grounding Raised Floor Cooling UPS Room Cooling Dist'n. & Conv.- Data Ctr Heat Rejection Equipment- UPS Overall facility grounding is indeterminate- data center ground grid formed via pedistal bonding Equipment CRAC N+1 (2/3) at full UPS rating Conveyance Not Applicable- Units contain compressors Equipment UPS room cooling is N+1 redundant- 3 x 2.5T glycol units Conveyance Not Applicable- Units contain compressors Glycol pumps are redundant Equipment- Data Ctr Condensers paired 1:1 with 2 CRAC; Drycoolers are N+1 Dist'n. & Conv- UPS Two of 3 units are dual source units- refrigerant + glycol; third unit is glycol only Drycoolers are N+1 Linear glycol system with single riser to rooftop drycoolers Curret Uptime Tier Classification Tier II Notes: 1 Tier 3 considers generators to be the primary electrical source Glycol pumps are redundant Non-redundant primary power backbone, et al Rating Legend Concurrently Maintainable Increased Risk Shutdown Required Not Applicable 1. Remove one (1) vintage 20-ton CRAC unit & associated conveyance from server room 2. Furnish & install one (1) 20-ton CRAC unit in server room paired with condenser on 4th floor roof 3. Install circuits from the new 100A MTS to the new 20-ton CRAC & new 20-ton condenser 1. Remove three (3) 2.5-ton vintage (glycol) CRAC units & associated conveyance from the UPS room 2. Furnish & install three (3) 2.5-ton CRAC units in UPS room paired with condenser on 4th floor roof Total estimated resiliency remediation cost = $1,555,000

14 Facility Data: 651 Boas Street, Harrisburg Built in 1954 Data Center Renovation in 1997 Data Center = 7,000 sf Current Server Count = 593 servers Labor & Industry (L&I) Current Uptime Institute Rating: Tier II (redundant capacity components) Growth Capacity: At current build: +65 servers At full build out: +65 servers 14

15 L&I General Observations General Notes and Areas for Improvement: The L&I building is a multi-purpose, multi-tenant facility hosting over 30 L&I lines of business Data center is located adjacent to building lobby; access to building is not controlled by security Facility is Tier II resilient (non-redundant primary power backbone and non-redundant glycol conveyance) 11 raised floor height is not conducive to high density computing The center is protected by a single, non-redundant generator The UPS are located in the data center, contributing to the heat load TVSS is not deployed in the electrical distribution system Although located outside of the flood plain, flooding in surrounding areas could render the site unapproachable by critical staff Facility Recommendations: Continue its role supporting critical L&I systems Although expansion is limited, it should be considered as a potential consolidation site for systems from other agencies that do not require high uptime or 24x7 monitoring 15

16 L&I Capacity Summary N ATS N Substation N 85% Labor & Industry Data Center Operation at UPS Limit 49% 38% ATS N Space (RUs) 100% 78% 79% 0% 55% 45% 45% 48% 52% 57% 2N Substation N N+1 N+1 Notes: 1. UPS depicted as % of full rating with 45% policy deployment limit. 2. Space consumption presumes historical policy deployment without compaction or virtualization. 3. Power consumption adjusted for summer HVAC peak. % Consumed System Resiliency Resilient Capacity Current Consumption UPS 2N 440kW 172kW CRAC N+1 120T 54T Dry Coolers N+1 120T 59T Substation 8 Department of Labor & Industry N 500kVA 207kVA ATS 8 N 1200A 575A Substation 9 N 750kVA 243kVA ATS 9 N 1200A 675A Generator N 1500kW 421kW Operating Limit (90% unless otherwise noted) Department of Labor & Industry Limiting Infrastructure Resiliency Resilient Capacity Assumed Operating Limit (%) Assumed Operating Limit Current Consumption Growth Potential (expressed as kw of IT Compute load) Growth Potential (expressed as % of current consumption) Approximate Growth - Number of Discrete Servers (+ supporting storage & network) Comments UPS 2N 440 kw 45% 198 kw 172 kw 26 15% ATS 9 (HVAC) N 1200 A 90% 1080 A 675 A 53 46% Substation 9 N 750 kva 90% 675 kva 243 kva % 65 UPS operating limit expressed as % of total capacity 16

17 L&I Tier III Conformance Electrical HVAC 17 Site Subsystem Utility Service Lightning Protection & TVSS Primary Power Disrtibution UPS Systems UPS Batteries Engine Generators Load Bank Secondary (Critical) Power Distribution Grounding Raised Floor Cooling UPS Room Cooling Description Comments & Related Subsystems Remediation Dual Utility Feed from Capitol Campus Substation Roof containing critical drycoolers e/w passive lightning protection Single circuit via Power Center 8 powers UPS; Single circuit via Power Center 9 powers all critical HVAC Feeds are underground to facility; Substation double-ended form 4 PP&L circuits UL listing status unknown; TVSS not noted 2N UPS UPS system comprised of 8 independent in-row UPS Dedicated batteries for each UPS sized to support rated load for >15 minutes Single Generator e/w non-redundant starting batteries Single Load Bank dedicated to generator only 2N to IT load; static switches not deployed Overall facility grounding is indeterminate Equipment CRAC N+1 (6/8) at full UPS rating Conveyance Not Applicable- Units contain compressors Equipment Not Applicable- UPS located in Data Ctr Conveyance Not Applicable- UPS located in Data Ctr Eqpt- Data Ctr N+1 (6/8) at full UPS rating Dist'n. & Conv.- Data Linear glycol conveyance with single riser to roof Ctr Heat Rejection Cooling for Critical Power Centers 8 & 9 is non-redundant on Eqpt- Pwr Ctr base building system Dist'n. & Conv.- Pwr Linear chilled water conveyance to critical power center room Ctr Curret Uptime Tier Classification Notes: 1 Tier 3 considers generators to be the primary electrical source Concurrent Maintainability Scorecard Department of Labor & Industry Dedicated to technology operations Can not test UPS post maintenance Circuits powering single cord IT devices can not be maintained Per documentation, data center ground bar connected to telephone main ground via daisy chain; telco ground connected to main electrical ground in main electrical room- This ground was not evident Rating Legend Concurrently Maintainable Increased Risk Shutdown Required Not Applicable 1. Furnish & install one (1) 2000A power center 9b (including 750kVA transformer) 2. Furnish & install one (1) 800A power center 8b (including 250kVA transformer) 3. Furnish & install one (1) distribution board 4. Furnish & install six (6) manual transfer switches (MTS) 5. Furnish & install one (1) ATS 6. Disconnect circuits from seven (7) CRACs and their associated drycoolers 7. Install circuits from the new 800A ATS and the new 800A power center and UPS 5 and UPS 6 8. Add circuits from six (6) new MTS to both distribution boards and to six (6) CRACs and their associated drycoolers 9. Add circuit from one (1) CRAC and associated drycooler to the new distribution board 10. Disconnect circuits between the 1600A power center and UPS 1, UPS 2, UPS 5, UPS 6, and UPS Install circuits between the new 800A power center and UPS 1, UPS 2, and UPS 7 1. Furnish & install one (1) 1000kW generator 2. Furnish & install one (1) 2000A automatic transfer switch (ATS) 3. Furnish & install one (1) 800A ATS 4. Add circuits from the new generator to the new 800A and the new 2000A ATS 5. Add circuits from the new 2000A and the new 2000A ATS to the switchgear and the new power centers Not addressed due to operational disruption required to loop & add isolation to glycol conveyance; utilization of refrigerant system requires detailed engineering analysis Not addressed due to disruptive affect on building operations Not addressed due to disruptive affect on building operations Tier II Non-redundant primary power backbone, et al Total estimated resiliency remediation cost = $3,383,000