Evolution of IT Infrastructure For Fusion Control Systems

Transcription

1 Evolution of IT Infrastructure For Fusion Control Systems Presentation to 14 th International Conference on Accelerator & Large Experimental Physics Control Systems (ICALEPCS) October 6-11, 2013 Tim Frazier Chief Information Officer NIF & Photon Science LLNL-PRES

2 NIF s IT architecture is based on four principles Individual component failure should not cause infrastructure failure Separate workloads & have more than one running at all times Technology should be easily replaceable Avoid an attachment to physical things Achieving self-similarity should guide the selection of technology Like model numbers wherever possible Use data to forecast resource consumption Create repositories of long-term metrics for analysis NIF s2.ppt Frazier - ICALEPCS Conference San Francisco, October 6-11,

3 NIF has consolidated its server footprint by 40% 423 servers 330 servers 228 blade servers 123 blade servers Partnership with key technology providers has been integral to our success Frazier - ICALEPCS Conference San Francisco, October 6-11,

4 Our physical footprint has been reduced by 50% SPARC-to-Intel migration made possible by port from Ada to Java High-density, virtualized servers Single-purpose to multi-purpose servers made possible by Virtualization (Xen) Frazier - ICALEPCS Conference San Francisco, October 6-11,

5 We have kept up with customer demand despite the 40% consolidation in footprint Low cost of ownership virtual machines enable single-purpose hosts Frazier - ICALEPCS Conference San Francisco, October 6-11,

6 Virtualization of our Integrated Computer Control System (ICCS) is nearly complete [1] Shot Director Supervisory Layer Bundle 1 Bundle 2 Bundles 3-24 NIF Common [50] [450] Collaboration Server Subsystem Shot Supervisors Collaboration Server Subsystem Shot Supervisors... Collaboration Server Subsystem Shot Supervisors Collaboration Framework Servers Subsystem Analysis Analysis Shot Servers Supervisors Servers Injection Laser Injection Laser Target Diagnostics Device Control Layer [1300] Beam Control Beam Control Complete Alignment LPOM... Laser Diagnostics... Laser Diagnostics In Progress Inspection Systems Industrial Industrial Controls Controls (Consoles) NIF s2.ppt 6

7 To build an infrastructure, you need building blocks Ethernet Switch Cisco 5548 Ethernet Switch Cisco 6509 Diskless Blade Servers HP BL460c Fiber DCX Switch Filer with disks NetApp 3250 Frazier - ICALEPCS Conference San Francisco, October 6-11,

8 Proto-type a single environment Ethernet Switch Cisco Gb Fiber network 10 Gb Ethernet network Diskless blades Blade Servers HP BL460c Ethernet Switch Cisco 5548 Fiber DCX Switch Filer with disks NetApp 3250 Frazier - ICALEPCS Conference San Francisco, October 6-11,

9 Create a segmented infrastructure Ethernet Switch Cisco 6509 Sandbox Dev Int - QA Production Controls Services Fiber DCX Switch Frazier - ICALEPCS Conference San Francisco, October 6-11,

10 Computational workload is very densely packed onto hypervisors 33% utilization 27% utilization 17% utilization 17% utilization Controls2 General02 : Control Development/Integration/QA system virtual machines virtual machines ShotProd2 Production02 : : Production, non-control system system virtual virtual machines Frazier - ICALEPCS Conference San Francisco, October 6-11,

11 Memory limits the packing factor for hypervisors Non-controls environments run with less margin Controls2 General02 : Control Development/Integration/QA system virtual machines virtual machines ShotProd2 Production02 : : Production, non-control system system virtual virtual machines Frazier - ICALEPCS Conference San Francisco, October 6-11,

12 We rely on many tools to manage our infrastructure Storage Infrastructure Network Infrastructure Brocade Manager Active Directory IPAM (netping) 3PAR desk manager Statseeker NetApp Manager Asset F5 Manager AssetDB DNS Server Infrastructure Enterprise Manager Splunk

13 We have developed metrics to measure discrepancies between tools & performance outliers Systemic problems can be revealed by trending metrics over time Frazier - ICALEPCS Conference San Francisco, October 6-11, 2013

14 Two tools are used monitor & manage our server infrastructure Oracle Enterprise Manager Performance, Configuration Management & Incident Management See John Fisher s Poster Monitoring of the National Ignition Facility Control System THPCC082 Splunk Log file mining & alerting Frazier - ICALEPCS Conference San Francisco, October 6-11,

15 Agent-based monitoring provides a wealth of information beyond host performance management Configuration Management Management Database Database metrics Application metrics Management Service User-Defined Metrics Management Agent Database Application Host Frazier - ICALEPCS Conference France, October 10-14,

16 Final thoughts (informed by hindsight) Self-similarity, more than any other quality, has enabled us to grow our infrastructure by 100% without a corresponding increase in staff Segmentation, more than any other design principle, has enabled us to increase reliability by isolating performance degradation & component failures Virtualization, more than any other technology, has enabled us to grow our infrastructure by 100% while at the same time consolidating our physical footprint by 50% Data provided by tools, specifically agent-based collection, more than any other asset, have provided the knowledge needed to manage our infrastructure It is time for your questions! Frazier - ICALEPCS Conference San Francisco, October 6-11,

17