The CIME Case Control System

Transcription

1 The CIME Case Control System An Object Oriented Python Data Driven Workflow Control System for Earth System Models Jim Edwards 22 nd Annual Community Earth System Model Workshop Boulder, CO June 2017

2 What is CCS? CCS is the workflow control code for creating an experiment in CIME. Creates and configures a case (experiment) Handles access to input data and disposition of output data Configuration and compilation of the model and supporting software. Batch interface with HPC systems Tests of model validity

3 Data and Control in CCS Data describing all aspects of interactions with CIME are stored in XML format files. These files contain all the information required to run a given experiment on a given machine using a given compiler. Code in python interprets these XML files to create a case. Code contains no client specific information. Users can change all aspects of an experiment via manipulation of data in XML files and should never need to modify code.

4 The CCS: Written in python (2.7 core) with a coordinated, consistent look and feel Object oriented approach allows for much greater code reuse Each XML file type has a corresponding python module A large and growing number of internal tests help to assure changes are backward compatible and consistent

5 User Interface scripts in CSS o create_newcase: Create an experiment (case) directory for a given machine, compiler and model configuration. o case.setup: Generate batch scripts in a case for a given pe-layout o case.build: Compile model code and supporting libraries o case.submit: Submit a job or set of jobs to the batch system o xmlquery: examine case data settings o xmlchange: modify case data settings

6 Templates for batch jobs <job name= case.run > <template> template.case.run</template> <task_count>default</task_count> <prereq> $BUILD_COMPLETE and not $TEST</prereq> </job> (from config_batch.xml)

7 Templates generate batch scripts #!/usr/bin/env python {{batchdirectives}} import os os.chdir( {{caseroot}} ) #!/usr/bin/env python # Batch system directives #PBS -N SMS.T62_g16.C.cheyenne_intel _101428_s4bcr8.run #PBS -r n #PBS -j oe #PBS -m ae #PBS -V #PBS -S /bin/bash #PBS -l select=2:ncpus=36:mpiprocs=36:ompthreads=1 os.chdir( '/glade/p/work/jedwards/sandboxes/cesm2_0_alpha/cime/scripts/sms.t62_g 16.C.cheyenne_intel _101428_s4bcr8')

8 CCS design overview Python classes and scripts user interface scripts input classes case classes abstract classes The Case class acts as the central conduit for interaction between objects. Abstract classes contain

9 CCS user interface All user scripts access and extend a common set of command line options. User scripts operate on the case object and through it manage and interpret properties defined in XML

10 Input classes Machines Input and output directories Supported compilers and libraries Batch queueing system MPI execution Compilers Options for linking and building Batch system Options for submitting jobs Components Compsets Grids Pe layouts

11 Case classes Contain variables to control configuration Each case XML file has a corresponding python object These objects inherit common properties from abstract classes

12 Testing CCS includes internal regression tests to assure functionality is maintained as the code evolves. These include pylint, xmllint, python unit tests, internal functionality tests. CCS supports a wide variety of tests of functionality of earth system models. This facilitates development of those components external to CIME. These include model restart tests, baseline comparisons, and model consistency tests.

13 pylint and xmllint Pylint assures that python code follows a community accepted standard and that exceptions to that standard are well documented. xmllint assures that XML files follow (locally) defined standards for that file format. XML file standards for CIME are defined in cimeroot/config/xml_schemas

14 Python doctest def parse_test_name(test_name): """ Given a CIME test name TESTCASE[_CASEOPTS].GRID.COMPSET[.MACHINE_COMPILER[.TESTMODS]], return each component of the testname with machine and compiler split >>> parse_test_name('ers') ['ERS', None, None, None, None, None, None] >>> parse_test_name('ers.fe12_123') ['ERS', None, 'fe12_123', None, None, None, None] >>> parse_test_name('ers.fe12_123.jgf') ['ERS', None, 'fe12_123', 'JGF', None, None, None] >>> parse_test_name('ers_d.fe12_123.jgf') ['ERS', ['D'], 'fe12_123', 'JGF', None, None, None] >>> parse_test_name('ers_d_p1.fe12_123.jgf') ['ERS', ['D', 'P1'], 'fe12_123', 'JGF', None, None, None] >>> parse_test_name('sms_d_ln9_mmpi-serial.f19_g16_rx1.a') ['SMS', ['D', 'Ln9', 'Mmpi-serial'], 'f19_g16_rx1', 'A', None, None, None] >>> parse_test_name('ers.fe12_123.jgf.machine_compiler') ['ERS', None, 'fe12_123', 'JGF', 'machine', 'compiler', None] >>> parse_test_name('ers.fe12_123.jgf.machine_compiler.test-mods') ['ERS', None, 'fe12_123', 'JGF', 'machine', 'compiler', 'test/mods'] """

15 System Tests DAE: Data assimilation test ERI: hybrid/branch/exact restart test ERIO: tests restart with different PIO methods ERP: pes counts hybrid (open-mp/mpi) restart bfb test from startup ERR: Restart with short-term archiving ERS: exact restart from startup NCK: multi-instance validation vs single instance PEA: single pe bfb test PEM: modified pe counts mpi bfb test PET: modified threading openmp bfb test (seq tests) PFS: system performance test PRE: Implementation of the CIME pause/resume test SMS: smoke startup

16 Component Specific Tests CIME allows component developers to add component specific tests to CIME capability while maintaining the test with the component model. As an example, the CLM model includes tests specific to that component but integrated with the CIME software. LII: CLM initial condition interpolation test SSP: CLM spinup test.

17 scripts_regression_tests.py daily cdash test output Name Status Time Summary N_TestUnitTest Passed 2m 22s 670ms Stable Z_FullSystemTest Passed 21m 33s Stable O_TestTestScheduler Passed 1m 17s 90ms Stable A_RunUnitTests Passed 3s 380ms Stable P_TestJenkinsGenericJob Passed 890ms Stable G_TestMacrosBasic Passed 2s 10ms Stable Q_TestBlessTestResults Passed 2m 58s 810ms Stable I_TestCMakeMacros Passed 10m 5s 810ms Stable K_TestCimeCase Passed 1m 3s 560ms Stable X_TestSingleSubmit Passed 3s 690ms Stable T_TestRunRestart Passed 9m 42s 870ms Stable S_TestManageAndQuery Passed 1s 250ms Stable R_TestUpdateACMETests Passed 870ms Stable L_TestSaveTimings Passed 7m 25s 20ms Stable J_TestCreateNewcase Passed 6m 41s 130ms Stable H_TestMakeMacros Passed 3s 50ms Stable B_CheckCode Passed 25s 910ms Stable M_TestWaitForTests Passed 29s 430ms Stable

18

19 CIME source availability and contributions CIME is open source and available from the github repository at: Contributions, issues and bug reports are welcome. Thanks to our collaborators: Alice Bertini, Michael Deakin, Chris Fischer, Jim Foucar, Steve Goldhaber, Rob Jacob, Bill Sacks, Jason Sarich, Mariana Vertenstein, Francis Vitt, Andreas Wilke Questions?

20 Disclaimer This talk is not intended to be a tutorial on how to use CIME/CCS It offers a brief exploration of the code design and structure. You really don t need to know any of this to use the model. A Look under the hood

21 What is a CIME object? Properties: Machine Compiler Grid Compset Events: setup_complete build_complete run_complete Methods: setup build submit postprocess

22 CCS Directory layout cime scripts Tools utils python CIME SystemTests XML tests

23 What is Object Oriented Design? Properties: Properties Machine Events: Compiler On_start setup_complete Make Grid On_stop build_complete Compset On_parked run_complete Model Color Year Price Methods: Methods setup Start build Stop submit Park postprocess

24 The CIME Case Control System Code Structure

25 The CIME Case Control System Introduction

26 Motivation for CCS The CESM1 scripts: Written in several languages and styles Inconsistent Look & Feel of user interface Little or no coordination between scripts or code reuse Changing an input XML file may require changes in several scripts Very difficult to modify, extend and support. Tests for functionality and correctness of scripts are few and far between Communication between scripts is through environment variables and files

27 CESM hub and spoke coupling MOSART Data CISM CLM Data WW3 Data MCT Driver/ Mediator CICE Data CAM Data POP Data

28 Common Infrastructure for Modeling Infrastructure code Freely available open source the Earth CIME Supporting Active component client models Driver-Coupler Code Data Models Case Control System System/Unit Testing Mapping Utilities CESM ACME NOAA/NEMS Post Processing and Workflow Tools Optimized math and communication utilities

29 Outline What is CIME? Case Control System (CCS) CCS Design Overview

30 The CIME Case Control System Code Design