Practical Data Centre Management Part 1

Transcription

1 Practical Data Centre Management Part 1 David Cuthbertson BSc MBCS MIOD Mobile david.cuthbertson@squaremilesystems.com 1

2 Housekeeping Fire evacuation Toilets Refreshments 11:15 Mid morning break 1pm Lunch 15:15 Afternoon break Mobile phones Student packs 2

3 Square Mile Overview UK based Cirencester, Glos Focus on applying asset & configuration management techniques to large infrastructures & data centres Develop toolsets for end to end systems and service mapping Integrate existing CMDB / knowledge sources with other toolsets Training, design, data capture, process development All technologies! Business Processes Departmental, Company Services End user, infrastructure, supplier Applications PC, server, mainframe, SOA Virtual Infrastructure PCs, Network, Servers, Storage, DBMS Hardware Infrastructure PCs, Network, Servers, UPS, Storage, Other Fixed Infrastructure (Cabling, Power, Cabinets, Rooms, Buildings) 3

4 Data Centre Management The following organisations have contributed information to this course, directly or indirectly, and their trademarks and cooperation are all acknowledged: Cisco, Panduit, APC, University of California, Texas Department of Information Resources, The Data Center Journal, BCS Data Centre Specialist Group, BICSI, OGC (Office of Government & Commerce), AFCOM ITSMF (IT Services Management Forum), Microsoft 4

5 Introduction, Agenda and Objectives Managing the Data Centre Facilities Introduction and objectives The TIA 942 framework Handover of the data centre facility Establishing the design limits Controlling the environment Maintenance practices Data centre inventory and documentation Installation practices Monitoring and ongoing administration of the facility Meeting the needs of regulators and 3rd parties Testing for recovery and resilience Backup and equipment recovery Access & Security Practices 5

6 Typical Data Centre Issues Heat and power Operating costs Speed of provisioning Optimising space Decommissioning Forward planning Reacting to failures / emergencies Communication with other teams 6

7 Defining a Data Centre A building or portion of a building whose primary function is to house a computer room and its support areas, according to TIA 942 7

8 Data Centre Standards TIA 942 Telecommunications Infrastructure Standard for Data Centers, April 2005 Substantial, available, all-american EN Information technology data centres Smaller scope, European perspective BICSI 942, complementary to TIA 942, expected 1Q2010 ISO/IEC NP Information technology -- Generic cabling for data-centres 8

9 Tier Classifications - The Uptime Institute Tier 1 Tier II Tier III Tier IV Site availability % % % % Downtime (hours/yr) Operations Center Not required Not required Required Required Redundancy for power, cooling N N+1 N+1 2(N+1) or S+S Gaseous fire FM200 or FM200 or suppression system Not required Not required Inergen Inergen Redundant backbone pathways Not required Not required Required Required 9

10 Classifying Data Centres Construction $/ft2 raised floor $450 $600 $900 $1,

11 Redundancy 11

12 TIA 942 Good at specifying how a facility should be designed and installed to meet requirements Ongoing management is often left to technology vendors to recommend O&M manuals for HVAC, UPS, Fire suppression Operational management is the customer s responsibility. 12

13 Technology Evolves Before A Monitor, keyboard and mouse to control servers 3 of 4 Cabinets have Servers 36 deep cabinets Ventilation Along Window Window Front Door 13

14 Technology Evolves After A Monitor, keyboard and mouse moved to remote location All 4 Cabinets have Servers cabinets Entire front ventilated, no window Routing, protecting cable Enclosing equipment More equipment and more types of equipment being put into datacenter 14

15 Changing Requirements BEFORE AFTER No. of Servers per cabinet Power Disipated per cab W 3kW - 25kW Current service to cabinet 16A 32 A or 3 phase Types of Equipment Servers Blade Servers Monitor Power Distribution Units KVMs MidSpan Boxes Power Strips UPS Disk Arrays (Storage) Smart Power Strips Regular Power Strips Network types 100Base-T 1G, 10G, SAN No. of Cables Power 1 or 2 2 to 6 (per server) Network 1 or 2 5 to 10 Total

16 New Technology Challenges Sun Blade 8000 Blade Chassis 4 Power supplies (N+1) 9kW 3 chassis per rack HP C7000 Blade Chassis Up to 6 Power Supplies 13kW 4 chassis per rack Cisco Nexus Data Centre Switch 3 Power Supplies 12kW 16

17 Data Centre Management We will address typical operational management issues required to ensure that data centres (and equipment rooms) continue to work as facilities to house computing devices. Covering: Organisation Facilities Systems Services Staffing Funding 17

18 Data Centre Management We will develop the concept of best practice - industry acknowledged practices, processes and guidelines - what you find to be practical Handout material covers 70% of the course 30% is interactive. 18

19 Management Impact In staff, managing incidents a year In staff managing 9000 incidents a year 32 times improvement in productivity or employ 450 staff Cisco How? 19

20 Inadequate Management Avoidable failures Uncertainty Tasks take too long Lack of ownership Costs are too high compared to others Reacting to events rather than controlling And many others. What do you regard as bad management (in a Data Centre context) 20

21 Best Practice Management For Data Centres No accepted framework In other areas: Project Management Prince 2 Service Management - the IT Infrastructure Library (ITIL) - ISO Information Security - ISO Business Continuity BS

22 Different Working Practices 22

23 ISO20000/ ITIL V2 Framework Security Management Service Continuity & Availability Management Release Processes Release Management Service Delivery Processes Service Level Management Service Reporting Control Processes Configuration Management Change Management Resolution Processes Incident Management Problem Management Capacity Management Financial Management Relationship Processes Business Relationship Management Supplier Management 23

24 Examples Best Practice Procuring a new server Policies - sign off, payment Ordering process life cycle Purchase orders common reference Roles and responsibilities specify, order and approve 24

25 Examples Best Practice Backup Tapes Regular schedule Taken offsite Checked on a regular basis 1. Policies or rules 2. Processes within and across teams 3. Unique reference for lifecycle management 4. Roles and responsibilities 25

26 Defining Management Planning what needs to be done to achieve a particular result Organising and directing appropriate resources. Controlling and making adjustments as needed Motivating all those involved. 26

27 Common Issues (for the manager) Personal objectives Culture and working practices Accurate and relevant information Toolsets, audits, reviews Available resources Organisational structures Difference in understanding of risk at different levels 27

28 Provisioning a Server You have received a request to move 10 existing servers into a data centre. What tasks might have to be done to fulfil this request? 28

29 Exercise 1 - Provisioning Concentrate on these two and the tasks appropriate to each Assess Plan Implement Test Completion 29

30 Data Centre Rack Space %Full % 50-75% <50% 30

31 Power Loading >6kW 2-6kW <2kW 31

32 Cooling Rated % 50-75% <50% 32

33 At the Environmental Level Overall design limits of rooms Power, Cooling, Loading Manage current and future needs Power by Room, PDU, Phase, Cabinet, Strip, Port, Cooling by Room, Zones, Cabinets Floor loading by Room, Zones Space by Room, Zones, Cabinets Maintained list of devices Connectivity of device power Don t assume a spare cabinet can be used! 33

34 Reducing Provisioning Times Reduce discovery time Management toolsets Remove the physical aspect Virtualisation Optimise Processes Formalise workflow Use common knowledge bases 34

35 Starting Well A New Data Centre 1. We have used TIA 942 to determine requirements and build specifications for a new data centre What handover information could you specify from the contractors to make it easier to manage when in service? 2. It has been decided to use a hosting provider to house some of the mission critical computing. How would you assess their management practices so that you would be confident that there is minimal risk? 35

36 Handover of a New Data Centre Typical handover information sets Design documents Test documents O&M manuals Support contracts & conditions Training programme Support systems - BMS 36

37 Understanding Existing Data Centres Environment limits Information sets - formal and informal Working practices - formal and informal Roles / responsibilities Current issues Establish priorities 37

38 Establishing a Baseline Know your environment design limits Understand the gaps Roles Knowledge Practices Decide on priorities and actions 38

39 Establish Design Limits Room Architectural and Structural - Weight Mechanical - Cooling, fire detection /suppression Electrical - Power 39

40 Typical Power Distribution Incoming feeds UPS 32 & 16 Amp Sockets 3 Phase PDU 100A 100A 100A Remote Controlled Power Strips Circuit Breakers 24A 32A 16A Power Strips Power Strips Power Strips 10A 16A Ethernet 40

41 Power Monitoring at Strip Level Totals for Power Strip Individual Port Status and controls 41

42 Exercise 2 Power Trips Your first objective as the new data centre manager is to prevent circuit breakers tripping unexpectedly due to loading issues. What would you do and in what order? 42

43 Controlling the Environment Known design limits Project approval process Regular reviews against limits Forward planning to avoid overload Customer / own kit can cause issues Maintenance practices 43

44 Maintenance Practices Failover testing Optimisation of power, cooling, cabling Maintenance contract records Servicing and maintenance requirements as covered by BS5266 pt1 Batteries Fire detection systems PAT testing CFD Modelling 44

45 CFD Modelling 45

46 Data Centre Inventory and Documentation It s common sense to know what is in your data centre and how it is configured But achieving this is not so easy Why is it so difficult? 46

47 Exercise 3 Data Centre Records The 10 servers have been successfully installed in the data centre What records or systems would you expect to have been updated or modified as a result of the additional servers? 47

48 Recommended Information Sets Space Environment (power, cooling) Connectivity (power, networks) Asset and Inventory controls Device management Service management 48

49 Space Management Space management systems Floor layouts and plans Rack layouts Floor loading Cabinet functions (customer, comms, servers etc.) Spreadsheets 49

50 Environment Building management systems Fire, cooling, temperature, power, humidity Power management systems PDU, UPS, power strips Access controls Power distribution diagrams and lists Current and projected power / cooling 50

51 Connectivity IP addressable power strips Cable management systems Intelligent cable management systems Network provisioning systems Fixed infrastructure diagrams Network diagrams Lists / databases of connectivity between devices and fixed infrastructure. 51

52 Cabling Overview Diagram 3C 3E 2E 2W 1E 1W 0E 0C 0W 0E N1E Fitzwilliam Street London Head Office Fibre Copper UTP 52

53 Cabling Fixed Infrastructure Second 2E-A 2E-A 2E-A 2E-A 2E-B 2E-C First 1E-A 1E-A 1E-A 1E-A 1E-B 1E-C Ground 0E-A 0E-A 0E-A 0E-A Head Office Computer Room 0C-A 0C-A 0C-A 0C-A 0C-B 0C-B 0C-B 0C-B Second 2W-A 2W-A 2W-A 2W-A 2W-B 2W-C First 1W-A 1W-A 1W-A 1W-A 1W-B 1W-C Ground 0W-A 0W-A 0W-A 0W-A 3C-A 3C-A 3C-A 3C-A 3C-A 3C-A 3C-A 3C-A 3C-B 3C-B 3C-B 3C-B 3C-B 3C-B 3C-C 3E-A 3E-A 3E-A 3E-A 3E-A 3E-A 3E-A 3E-A 3E-B 3E-B 0E-B 0E-C 0C-C 0C-C 0C-D 0C-D 0W-B 0W-C 0E-A 0E-A 0E-B Basement 0C-V 0C-V 0C-V 0C-V 0C-F 0C-F 0C-G Fitzwilliam Street 0E-B 0E-B 0E-B N1E-B N1E-B N1E-C 0C-H 0C-J 0C-K 0C-M 0C-L 0C-N 53

54 Asset Controls Lists of all devices and assets Their current status and location Previous history and audit trail Often combined with maintenance and procurement data Auto-discovery can help, but often limited in value in data centres. 54

55 Device Management Network, server, storage monitoring Configuration systems Deployment / provisioning systems Network and other architecture diagrams Automated discovery and scanning Backup and failover 55

56 Service Management Help or service desk system Project control or management system Services maps Devices mapped to critical services Service monitoring tools Recovery planning and testing Billing and charging 56

57 For Example Data centre project server cabinets: 6 servers per cabinet, 3600 servers 4 network connections per server 14,400 switch ports 7200 power connections (dual supplies) Plus SAN storage, network hardware, cabling etc. 57

58 Data Chaos Power feeds to cabinets User locations Network Inventory Chassis/card layouts IP addressing spreadsheet Rack diagrams Network diagrams Maintenance contracts Floor plans Patching spreadsheets Server asset list Building wiring diagrams PAT testing results Departmental billing/charging Service desk inventory Hot/cold aisle Project plans Computer room layout Naming conventions Power phase summation Equipment room maps Labelling standards PABX port mapping SAN Architecture 58

59 Summary of Information Sets There is potentially a huge amount of data, so focus is required Knowledge of components is duplicated, so focus on reducing the number of possible data sets Start with information held by specific teams, before looking at information sets across teams. 59

60 Installation Practices The Data Centre was designed and implemented against various criteria which will be compromised if the installation practices are inconsistent When should you define your own best practices - before you take over the facility, or afterwards? 60

61 Good Installation Practices Original design rules and limitations should be understood by all Keep naming and labelling consistent Updates to O&M manuals must be carried out to keep them current Separate build room can save a lot of time when staging new installs They will evolve, so review regularly. 61

62 Monitoring and Administration of the Facility Improve management information Toolsets, audits, incident and problem reviews Optimise people resources Centralise expertise and workflow Multi-disciplinary staff Culture change Review organisational structures 62

63 Typical Management Data Downtime or availability figures Component, service, customer Incidents Equipment faults, change related Changes Successful, failed or backed out Capacity Existing, planned and full Charging and funding 63

64 Charging and Funding Space Power Cooling Network Ports used Shared Infrastructure Costs and Support Hardware Maintenance Costs and Support Operations Costs and Support 64

65 Exercise 4 - Reducing Costs Without buying new technology, or changing staffing, you are targeted with improving the cost base of the data centre. What management initiatives can you undertake that will reduce direct costs? 65

66 Meeting the Needs of 3rd Parties Insurance SOX, FSA, auditors, etc. Planning permission Legislation Disability Health and safety Electricity at work And others All require evidence that you are complying with their policies, requirements or laws. What would be sufficient evidence in most cases? 66

67 Evidence of Conformance Policies Evidence of processes that support the policies Change records Build and test records Written material or trails Communications Incident reviews If someone was accidentally electrocuted, what would the Health and Safety Inspector want to see? 67

68 Testing for Resilience Resilience may be found in: Incoming power supply HVAC systems Networks Equipment clustering or primary/standby Data storage Used for fault tolerance and maintenance 68

69 Typical Issues with Resilience Is meant to prevent disruption, not cause it! Detailed understanding needed of primary to secondary failover, with evidence Should be tested offline before going live Doesn t work if device configurations are corrupted and functionality is lost Testing is meant to prove the resilience would work if required. 69

70 Backup Backup process Weekly dumps, daily incremental backups Tapes often rotated every 6 weeks Off site storage SAN systems don t impact need for tape backups Most financial data must be retained for up to 6 years, pensions 40 years plus 70

71 Practical Backup Issues Recovery of data may be dependent on media type, recovery software and platform. Password and encryption is more common to protect data, so improved records required to keep passwords available. Organisations must check on physical security of backups e.g. credit card frauds. June 2, 2008 (Computerworld) Bank of New York Mellon Corp. late last week said it has launched a new policy to encrypt data held on storage devices and to limit the amount of confidential client data stored on tape drives. The policy was launched after unencrypted backup data tapes were lost twice by third-party couriers this year. The company late last week also announced that it will provide two years of free credit monitoring, credit-freeze benefits and a $25,000 identity theft insurance policy to those affected by the missing tapes. 71

72 Exercise 5 Data Centre Authority Give examples where initiatives or tasks driven by the data centre team will result in live systems being switched off (planned and unplanned) 72

73 Access and Security Practices Keep the bad guys out and the good guys honest! Two thirds of security attacks involve individuals wanting to damage a company, rather than making a profit 73

74 Access and Security Practices Secure access is a common feature in data centres Access control will depend on organisational issues Building Access Room Access Cabinet, Equipment Layering - multiple needs to authenticate access Many types What you have Card or token What you know Pass code, keypad Who you are Biometric (finger, eye) 74

75 Access Controls Issue Maintenance Recall Authority to enter (internal teams) Authority to enter (external teams) Surveillance issues 75

76 Some Tips Embed in HR terms of employment Confidentiality, adherence to procedures, communicate concerns Applying for access to central system or Operations Centre, rather than local site Criminal checks on those with regular access Guards, Cleaners, Working in pairs or under supervision External auditors useful to support management concerns 76

77 Data Center Job Spec - Yahoo Performs periodic physical security penetration testing. Reviews security logs to monitor unauthorized system access attempts, both internal and external. Investigates and follows up on security violations and incidents. Serve as a member of Yahoo! Paranoids operations security team. Evaluate new data center build-outs/initiatives from a physical security perspective and provide guidance. Develop physical security monitoring standards and ensure those standards are consistently enforced across all locations. BA/BS or experience related military background preferred. Thorough understanding and experience related to Data Center Security policies and practices a must (prefer 5+ years.) Excellent interpersonal and organizational skills. *Excellent judgment, discretion and diplomacy. Knowledge of laws and practices of employee safety. Knowledge of current security technology including access control, CCTV, identification and special applications. Familiar with investigative tactics including interviewing and documentation. Experience administering a large contractor staff of security officers and control center operations. 77

78 Data Centre Management Planning what needs to be done to achieve a particular result. Organising and directing appropriate resources Controlling and make adjustments as needed. Motivating all those involved. 78

79 Key Learning Points One each 79

80 Thank you for attending Have a good journey back home! 80