Practical Lessons from Data Centre Design and Testing Zac Potts

Transcription

1 Practical Lessons from Data Centre Design and Testing Zac Potts Associate Director of Data Centre Design

2 Introduction and Overview of Data Centre Design and Engineering Principles Tier Ratings and Standards Examples from Design, Implementation, and Testing Key Points

3 Introduction and Overview of Data Centre Design and Engineering Principles

4 An Introduction to Data Centres What is a Data Centre? A data centre is a critical space dedicated to housing and supporting the ICT Infrastructure that supports a business. Where is the line between Sever Room and Data Centre? Are all Data Centres Equally Critical?

5 An Introduction to Data Centres What is a Data Centre? 2U Server 10U Blade Enclosure 1U Network Switch

6 An Introduction to Data Centres What is a Data Centre? 42U IT Rack 24 Rack Contained POD

7 An Introduction to Data Centres What is a Data Centre?

8 An Introduction to Data Centres What is a Data Centre? A view Inside a Google Data Centre Facility Source: Google.

9 An Introduction to Data Centres What is a Data Centre? An ever increasing number of systems rely on Data Centres, IT Networks and Digital Infrastructure. Each with a varying level of integration and reliance on it

10 An Introduction to Data Centres How Critical is a Data Centre? Various Levels of Criticality Dependent on the IT Function and Topology. Consequences of failure are clear but varied. E.g o o o o o Financial Impact Brand Damage Information Loss Lost Time Unusable Resource

11 An Introduction to Data Centres Engineering Principles Keep the IT Systems within Design Operating Conditions Cooling and Humidification Keep the IT Systems Powered UPS / Standby Generation Secure and Protect Systems Fire Detection and Suppression / CCTV Access Control / Intruder Alarms Physical Security Minimise Cost and Environmental Impact System Selection and Engineering

12 An Introduction to Data Centres Components of Design Data Centres are complex, containing Multiple Systems each with Multiple Interactions and Dependencies

13 An Introduction to Data Centres Components of Design Mechanical Systems Various approaches to Cooling: CRAC Units, CRAH Units, In-Row, Indirect Air, Direct Air Various approaches to Heat Rejection: Condensers, Chillers, Dry Coolers, Cooling Towers, Air Various Specialist or Ancillary Systems: CHW Flow Control Systems, Room / Aisle Pressure Control Systems, Humidification, Leak Detection, Ventilation

14 An Introduction to Data Centres Components of Design Electrical Systems UPS Systems / Flywheel Systems Standby Generation LV Switchgear HV Switchgear Distribution Systems

15 An Introduction to Data Centres Components of Design Other Specialist Systems BMS and SCADA Systems Fire Detection, Suppression, and VESDA Leak Detection Comm s

16 An Introduction to Data Centres Bringing Everything Together Data Centre Design is generally about delivering: Reliability, Availability, and Confidence Generally by using a combination of Redundancy, Resiliency, and Conservative Design However. Every Facility is Different. Different Levels of Criticality Different Acceptable Modes of Failure Different Attitudes to Risk Different Attitudes to Maintenance

17 Introducing Data Centre Resiliency and Efficiency Standards

18 Data Centre Standards Why do we need them? Where do we draw the line What are the chances of.. Do you really think that we need to plan for.. You ll never need to replace.. If. happens, we have bigger problems We ve bought the best so the risk is less

19 Data Centre Standards Why do we need them? To simplify the assessment of multiple, complex M&E Systems Clarifies what could be an endless discussion Provides a Baseline and Benchmark Progressive Standards building one level upon another A clear measure of resilience

20 Data Centre Standards Available Standards Established Standards Uptime Institute TIA 942 BICSI EN Plenty of Similarities and Differences between one another. Definitions of Different Tiers/Classes/Ratings ASHRAE Guidelines and Standards EU Code of Conduct The Green Grid

21 Data Centre Standards Common Pitfalls Against M&E Standards (e.g Tier/Class etc) Design Capacity of Equipment at Peak Conditions Within Design Equipment Derating E.g Ambient Temperatures, Elevation, Power Factor Manufacturing Tolerances Compartmentalisation and Diversity Failure modes of Sub-systems and Sensors Concurrent Maintainability and Plant Removal Approach to Redundancy and Diversity of Mains Continuous Cooling or Impact of Interruption

22 Examples from Design, Implementation and Testing

23 Examples from Design Implementation and Testing Overview Selected Examples from Design, Auditing and Industry Discussions Not all are Issues - Focus is on Intent Representative of many Engineering Challenges Changing Industry Approaches lead to new Experiences

24 Examples from Design Implementation and Testing Air Handling Units Located Outside the Whitespace/Building Increasingly common for A/C units to be outside the critical space either due to aesthetics or engineering need Engineering Challenges: Fire Boundaries / Rating / Suppression Leakage Pressure Mitigations Unit Air Tightness Definition of Fire Zone Boundary and/or Use of Dampers Note: Consider across operational band; not simply at design load

25 Examples from Design Implementation and Testing Correct Understanding of Circuit Derating and Thermal Affects Consideration and understanding of circuit grouping effects and thermal derating May Affect Room Based PDU Units and Rack PDUs High Load Circuits are often adjacent Result: Exact provision often misunderstood Fundamental derating of capacity Discrimination may be affected Mitigations Clear Communication and Co-ordination

26 Examples from Design Implementation and Testing Interlinks between Redundant Systems Badly Designed or Low Cost Interlinks can reduce resilience. Perceived Advantage to Interlink Long Term Failure Planning e.g. Catastrophic TX Failure Perceived Low Risk of Interlink Failure Often Introduces a SPOF, compromising overall rating. May be justified based on a non-standard probabilistic approach

27 Examples from Design Implementation and Testing Understanding of Generator Start Conditions Generators Start Conditions often Misunderstood Typically triggered by a sustained event (e.g 5s) Are there any scenarios outside of your control which can result in generators not starting? Nuisance Starts Mitigations Consider requirement and scope Reduce time to start, remove proving period Consider additional generator start conditions

28 Examples from Design Implementation and Testing BMS and Controls Systems Control Systems clearly need to be resilient and match the design philosophy of the overall facility Monitoring Systems with no control are expected to be unable to affect operation Not always true BMS and Controls systems may still be vulnerable to both malicious attack and fault Physical separation from control functions required

29 Examples from Design Implementation and Testing Mismatched Airflow and/or dt Contained Cooling Systems need to match Air Flow across the operational range Design dt is critical 10 C / 12 C / 15 C are commonly specified ASHARE Reports a low end estimate of 15 C-20 C In practice lower temperatures are still often seen Why? Volumetric Control Systems Over Supplying Air IT Team Overriding Fan Speeds Leakage from RAF or Containment Leakage into Return Plenum Hot Aisle Containment Plan Resulting Separation

30 Examples from Design Implementation and Testing Choice of Load Banks The wrong load bank can undermine an otherwise good test IT Loads vs typical Load Bank: Power Supply Redundancy Exhaust Temperatures and dt Dynamic response of fan speed to environment Additional Considerations Any part of the distribution which is bypassed, is not tested Be aware of Power Factor, kw and KVA

31 Examples from Design Implementation and Testing Awareness of IT Load Awareness of Leading kvar loads Function of IT Utilisation and Electrical System Capacity Requires close consideration at electrical design stage IT Users and Specifiers are not Electrical Engineers Consider: Non-Standard Hardware A&B Supplies Connected to Single Path (e.g via STS) Paths without UPS or UPS in Bypass Mechanical UPS Systems, Filters, Power Factor Correction Risk of Cascade Failure

32 Key Points

33 Key Point No. 1 The Importance of a Clear Specification

34 Key Point No. 1 The Importance of a Clear Specification Applicable to End Users, Service Providers, Consultants and Contractors Common Pitfalls Unclear and ambiguous specification results in responses which cannot be compared to each other Designing for all eventualities often results in over specified solutions and investments in infrastructure which may never be needed Describing something which doesn t exist

35 Key Point No. 1 The Importance of a Clear Specification Consider: Specification Requirement: 100 IT Racks with capacity up-to 20kW per Rack. A 2 MW Facility with HD cooling and three phase rack power?

36 Key Point No. 1 The Importance of a Clear Specification What was actually wanted: 100 IT Racks with an aisle averaged rack density of 4kW per Rack, with individual rack capacity up to 7kW Individual Rack Capacity should be able to be increased up to 20kW per rack with no changes to the central cooling systems or main electrical infrastructure An allowance for the final distribution to any 20 Racks to be upgraded to three phase circuits with no downtime and minimal disruption

37 Key Point No. 1 The Importance of a Clear Specification Two very different end results, each quite valid Often a need to target a balance Result 1 will likely run out of footprint before capacity Result 2 will likely run out of capacity before footprint There is a strategic business decision which is right It should not be left down to the interpretation of others Specify what is needed separate from what is preferable or anticipated Be conscious of the level of (un)certainty surrounding predictions

38 Key Point No. 1 The Importance of a Clear Specification Competitive Tendering often avoids specifying a product Approach with caution specification of features: Mixing features of different products Specifying features not strictly needed Specifying features which do not affect outcome DCIM is a common culprit. Watch out for vague sales terminology can do, could do, capable of Required Features

39 Key Point 2 Minimise Assumptions

40 Key Point No. 2 Minimise Assumptions Assumptions will exist but remove and minimise where possible. Common Assumptions: That there is a common understanding of a concept That a system will operate like another system e.g because common supplier or technology A system will operate in the most economical, efficient, or accurate manner Something has been done before so it is fully understood, and immune from fault or misunderstanding

41 Key Point No. 2 Minimise Assumptions Simple Example: Planned Run and Standby Rotation Should one unit switch off before telling another unit to start? Overlap of Air/Water Flow vs Break in Supply Time to Resume, Risk of Fault? Relatively simple operation without a single right answer

42 Key Point No. 2 Minimise Assumptions As previously, specify expectations clearly Request and Compile thorough Descriptions of Operations for expected operation Don t overlook simple operations e.g Unit Rotation or Normal Operation Witness Testing Question and Challenge

43 Key Point 3 Well Established Concepts are not Immune from Error

44 Key Point No. 3 Well Established Concepts are not Immune e.g. Hot and Cold Aisles / Rack Blanking Well established do not assume that it will be present nor that it is fully understood by all parties Mismatched airflow cannot operate with 100% air segregation Consequences of both Undersupply and Oversupply Can we control to zero net difference? if not, which is better, should we accept a level of bypass? Interaction with system design dt

45 Key Point 4 The Importance of Testing

46 Key Point No. 4 The Importance of Testing Only final testing happens at the end A good testing regime starts at project inception Client requirements Outcome Objectives Engineering Proof of Operation Testing Continual Confidence Ongoing testing All affect the total cost of ownership of a solution

47 Key Point No. 4 The Importance of Testing Considering how a system will be tested can uncover shortfalls in design: Early Identification = Easier and Cheaper Resolution Considering how a system will be tested improves safety and allows risk to be minimised Considering how a system will be tested can improve results with less intrusive methods, and/or greater accuracy

48 Key Point No. 4 The Importance of Testing Consider the progressive types of testing Design stage validation Factory visits and witnessing Factory acceptance Manufacturers commissioning Systems acceptance testing Integrated systems testing Ongoing planned and preventative testing

49 Key Point No. 4 The Importance of Testing Consider what testing is specified, required, needed or desired Anticipate and expect that the bulk of testing occurs at the end of a project; expect restricted time and additional pressure Consider and appreciate the dependencies and interactions required for site testing We test because there is a chance it will fail Consider the importance of load simulation Consider the requirements for ongoing testing

50 Key Point 5 Shared Experience

51 Key Point No. 4 Shared Experience All good engineering is built on the experiences and understanding of others There is a general secrecy around Data Centre Engineering Limited publication of faults and vulnerabilities - for good reason We can work with experienced specialists in the field, e.g Designers, Manufacturers, Installers, Testers, etc Attend Industry presentations and join professional networks. Build two way relationships and share experiences, lessons, short falls and pitfalls without specifics or blame

52 Questions and Discussion Zac Potts MEng Associate Director Sudlows Critical Infrastructures Tel: Mob: