Tesla Cluster Monitoring & Management

Transcription

1 Tesla Cluster Monitoring & Management

2 Introductions Robert Alexander CUDA Tools Software Engineer at NVIDIA Tesla Software Group

3 Overview Management and Monitoring APIs Kepler Tesla Power Management Features Health and Diagnostics

4 Monitoring and Management NVIDIA Display Driver pynvml nvidia-smi Command line NVML C API Python API nvidia::ml Perl API

5 NVML Supported OSes Windows: Windows 7 Windows Server 2008 R2 64 bit only Linux: All supported by driver 32-bit 64-bit

6 NVML Supported GPUs NVIDIA Tesla Brand: All NVIDIA Quadro Brand: Kepler All Fermi , 5000, 6000, 7000, M2070-Q NVIDIA VGX Brand: All Supported in the hypervisor

7 NVML Queries Board serial number, GPU UUID PCI Information GPU utilization, memory utilization, pstate GPU compute processes, PIDs Power draw, temperature, fan speeds Clocks ECC errors Events API

8 NVML Commands Enable or Disable ECC Change Compute mode Applies only to CUDA Default multiple contexts Exclusive Process Exclusive Thread Prohibited Change persistence mode (Linux) Keeps NVIDIA driver loaded

9 nvidia-smi nvidia-smi -q ==============NVSMI LOG============== Timestamp : Wed May 16 11:24: Driver Version : Attached GPUs : 1 GPU 0000:02:00.0 Product Name Display Mode Persistence Mode : Tesla C2050 : Disabled : Disabled

10 nvidia-smi Driver Model Current Pending Serial Number GPU UUID : N/A : N/A : xxxxxxxxxx : GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx VBIOS Version : Inforom Version OEM Object : 1.0 ECC Object : 1.0 Power Management Object : N/A PCI Bus Device Domain Device Id : 0x02 : 0x00 : 0x0000 : 0x06D110DE

11 nvidia-smi Bus Id : 0000:02:00.0 Sub System Id : 0x077110DE GPU Link Info PCIe Generation Max : 2 Current : 2 Link Width Max Current : 16x : 16x Fan Speed : 30 % Performance State : P0 Memory Usage Total Used Free : 2687 MB : 6 MB : 2681 MB

12 nvidia-smi Compute Mode : Default Utilization Gpu : 0 % Memory : 0 % Ecc Mode Current Pending : Enabled : Enabled ECC Errors Volatile Single Bit Device Memory : 0 Register File : 0 L1 Cache : 0 L2 Cache : 0 Total : 0

13 nvidia-smi Double Bit Device Memory : 0 Register File : 0 L1 Cache : 0 L2 Cache : 0 Total : 0 Aggregate Single Bit Device Memory Register File L1 Cache L2 Cache : N/A : N/A : N/A : N/A Total : 0

14 nvidia-smi Double Bit Device Memory Register File L1 Cache L2 Cache : N/A : N/A : N/A : N/A Total : 0 Temperature Gpu : 56 C Power Readings Power Management Power Draw Power Limit : N/A : N/A : N/A

15 nvidia-smi Clocks Graphics SM Memory : 573 MHz : 1147 MHz : 1494 MHz Max Clocks Graphics SM Memory Compute Processes : 573 MHz : 1147 MHz : 1500 MHz : None

16 nvidia-smi - XML ralexander@ralexander-test:~> nvidia-smi -q -x <?xml version="1.0"?> <!DOCTYPE nvidia_smi_log SYSTEM "nvsmi_device_v3.dtd"> <nvidia_smi_log> <timestamp>wed May 16 11:33: </timestamp> <driver_version>295.54</driver_version> <attached_gpus>1</attached_gpus> <gpu id="0000:02:00.0"> <product_name>tesla C2050</product_name> <display_mode>disabled</display_mode> <persistence_mode>disabled</persistence_mode> <driver_model> <current_dm>n/a</current_dm> <pending_dm>n/a</pending_dm>

17 pynvml Example $ python >>> from pynvml import * >>> nvmlinit() >>> count = nvmldevicegetcount() >>> for index in range(count):... h = nvmldevicegethandlebyindex(index)... print nvmldevicegetname(h) Tesla C2075 Tesla C2075

18 pynvml Example Continued >>> gpu = nvmldevicegethandlebyindex(0) >>> print nvmldevicegetclockinfo(gpu, NVML_CLOCK_SM) 101 >>> print nvmldevicegetmaxclockinfo(gpu, NVML_CLOCK_SM) 1147 >>> print nvmldevicegetpowerusage(gpu) In megahertz In milliwatts >>> nvmlshutdown() Also: NVML_CLOCK_GRAPHICS NVML_CLOCK_MEM

19 Downloads NVML SDK Python NVML Bindings Perl NVML Bindings

20 Third Party Software Platform Computing Bright Computing Ganglia Adaptive Computing Penguin Computing Univa

21 Ganglia NVML plug-in Data from

22 Out of Band API Why Out of band? Doesn t use CPU or operating system Lights out management Minimize performance jitter Subset of in band functionality ECC Power Draw Temperature Static info Serial number, UUID

23 Out of Band API Requires system vendor integration BMC can control and monitor GPU Control system fans based on GPU temperature IPMI may be exposed

24 Kepler Power Management New Kepler Only APIs Set Power Limit Set Fixed Maximum Clocks Query Performance Limiting Factors

25 Set Power Limit Limit the amount of power GPU can consume Exposed in NVML and nvidia-smi Set power budgets and power policies

26 Set Fixed Maximum Clocks From a set of supported clocks Will be overridden by over power or over thermal events Fixed performance when multiple GPUs operate in lock step Equivalent Performance Reliable Performance Save Power

27 Lockstep with Dynamic Performance GPU0 GPU1 GPU2 GPU3 Time

28 Lockstep with Dynamic Performance Thermal Event GPU0 GPU1 GPU2 GPU3 Time

29 Lockstep with Dynamic Performance GPU0 GPU1 GPU2 GPU3 Time

30 Lockstep with Dynamic Performance GPU0 GPU1 GPU2 GPU3 Time

31 Query Performance Limiting Factors GPU clocks will adjust based on environment Over thermal or over power limits GPU will reduce performance

32 Health and Diagnostics nvidia-healthmon Quick health check Suggest remedies to SW and system configuration problems Help users help themselves

33 nvidia-healthmon What it s not Not a full HW diagnostic Not comprehensive

34 nvidia-healthmon Feature Set Basic CUDA and NVML sanity check Diagnosis of GPU failure-to-initialize problems Check for conflicting drivers (I.E. VESA) InfoROM validation Poorly seated GPU detection Check for disconnected power cables ECC error detection and reporting Bandwidth test

35 nvidia-healthmon Use Cases Cluster scheduler s prologue / epilogue script Heath and diagnostic suites Designed to integrate into third party tools After provisioning cluster node Run directly, manually

36 nvidia-healthmon Loading Config: SUCCESS Global Tests: NVML Sanity: SUCCESS Tesla Devices Count: SKIPPED Result: 1 success, 0 errors, 0 warnings, 1 did not run GPU 0000:02:00.0 #0 : Tesla C2050 (Serial: xxxxxxxxxxx): NVML Sanity: SUCCESS

37 nvidia-healthmon InfoROM: SKIPPED GEMENI InfoROM: SKIPPED ECC: SUCCESS CUDA Sanity: SUCCESS PCIe Maximum Link Generation: SKIPPED PCIe Maximum Link Width: SKIPPED PCI Seating: SUCCESS PCI Bandwidth: SKIPPED Result: 4 success, 0 errors, 0 warnings, 5 did not run 5 success, 0 errors, 0 warnings, 6 did not run WARNING: One or more tests didn't run.

38 Contact Info and Links Robert Alexander developer.nvidia.com/nvidia-management-library-nvml forums.nvidia.com

39 Thanks!

40 Questions?

41 Tesla Cluster Monitoring & Management