Common Persistent Memory POSIX* Runtime (CPPR) API Reference (MS21) API Reference High Performance Data Division

Transcription

1 Common Persistent Memory POSIX* Runtime (CPPR) API Reference High Performance Data Division INTEL FEDERAL, LLC PROPRIETARY December 2017

2 Generated under Argonne Contract number: B DISTRIBUTION STATEMENT: None Required Disclosure Notice: This presentation is bound by Non-Disclosure Agreements between Intel Corporation, the Department of Energy, and DOE National Labs, and is therefore for Internal Use Only and not for distribution outside these organizations or publication outside this Subcontract. USG Disclaimer: This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof. Export: This document contains information that is subject to export control under the Export Administration Regulations. Intel Disclaimer: Intel makes available this document and the information contained herein in furtherance of CORAL. None of the information contained therein is, or should be construed, as advice. While Intel makes every effort to present accurate and reliable information, Intel does not guarantee the accuracy, completeness, efficacy, or timeliness of such information. Use of such information is voluntary, and reliance on it should only be undertaken after an independent review by qualified experts. Access to this document is with the understanding that Intel is not engaged in rendering advice or other professional services. Information in this document may be changed or updated without notice by Intel. This document contains copyright information, the terms of which must be observed and followed. Reference herein to any specific commercial product, process or service does not constitute or imply endorsement, recommendation, or favoring by Intel or the US Government. Intel makes no representations whatsoever about this document or the information contained herein. IN NO EVENT SHALL INTEL BE LIABLE TO ANY PARTY FOR ANY DIRECT, INDIRECT, SPECIAL OR OTHER CONSEQUENTIAL DAMAGES FOR ANY USE OF THIS DOCUMENT, INCLUDING, WITHOUT LIMITATION, ANY LOST PROFITS, BUSINESS INTERRUPTION, OR OTHERWISE, EVEN IF INTEL IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Company Name: Intel Federal, LLC Company Address: 4100 Monument Corner Drive, Suite 540 Fairfax, VA Copyright , Intel Corporation. Technical Lead: (Name, , Phone) Al Gara, Contract Administrator: (Name, , Phone) Aaron Matzkin, Program Manager: (Name, , Phone) Jacob Wood, Intel Federal, LLC Proprietary Page 2 of 30

3 Document Revision History Revision Number Date Comments 1.0 December 2017 Report of the MS21 prototype demonstrations. Intel Federal, LLC Proprietary Page 3 of 30

4 Contents 1 CPPR API Reference Manual Module Index Modules Class Index Class List Module Documentation CPPR API Classes Macros Typedefs Enumerations Functions Detailed Description Macro Definition Documentation Typedef Documentation Enumeration Type Documentation Function Documentation Class Documentation cppr_op_info Struct Reference Public Attributes Detailed Description Member Data Documentation Example Documentation mover.c Index Intel Federal, LLC Proprietary Page 4 of 30

5 1 CPPR API Reference Manual This manual describes the CPPR API. Applications may use this API to request that files be copied from global filesystems to local storage on compute nodes and back again. CPPR provides two namespaces storing data locally: a local per-node namespace for file-per-node data and the cooperative cache, a scalable read-only cache of files from the global file system. The API supports two basic mover operations: Move Copy a file from one location to another, if the file systems are supported Cooperative Get Copy from a global file system to the local namespace, but optimizing for the case where the same file requested by many compute nodes. The basic operations are non-blocking but blocking versions are provided. 5

6 2 Module Index 2.1 Modules Here is a list of all modules: CPPR API

7 3 Class Index 3.1 Class List Here are the classes, structs, unions and interfaces with brief descriptions: cppr_op_info

8 4 Module Documentation 4.1 CPPR API Classes struct cppr_op_info Macros #define CPPR_HANDLE_FMT "CPPR Handle(%llx)" #define CPPR_HANDLE_VALUE(info) ULL(((info)->handle & ~((cppr_handle_t)0x ))) #define CPPR_INVALID_HANDLE #define CPPR_INFINITE (0) #define CPPR_DEFAULT_STRIPE_SIZE 0 #define CPPR_DEFAULT_STRIPE_OFFSET ((uint32_t)-1) Typedefs typedef uint64_t cppr_handle_t Enumerations enum cppr_return_t { CPPR_SUCCESS, CPPR_NO_PREFIX, CPPR_NO_CC, CPPR_INCOMPATIBLE_VERSION, CPPR_NO_SERVICES, CPPR_INSUFFICIENT_ARGS, CPPR_INVALID_ARG, CPPR_NOT_IMPLEMENTED, CPPR_INTERNAL_ERROR, CPPR_OUT_OF_MEMORY, CPPR_QUOTA_LIMIT, CPPR_OP_EXECUTING, CPPR_UNKNOWN_HANDLE, CPPR_UNSUPPORTED_MOVE_TYPE, CPPR_SINGLE_WAITER_VIOLATION, CPPR_INVALID_PATH, CPPR_SOURCE_MISSING, CPPR_NOT_A_DIRECTORY, CPPR_INVALID_DEST_FS, CPPR_NO_READ_PERMISSION, CPPR_NO_WRITE_PERMISSION, CPPR_INSUFFICIENT_SPACE, CPPR_PATH_EXISTS, CPPR_NAMESPACE_CONFLICT, CPPR_THREAD_CREATE_FAILURE, CPPR_FAILED, CPPR_UNDEFINED, CPPR_BAD_DATA } enum cppr_op_status_t { CPPR_STATUS_UNKNOWN, CPPR_STATUS_PENDING, CPPR_STATUS_COMPLETE } 8

9 enum cppr_flag_t { CPPR_FLAG_OVERWRITE = 0x1 } enum cppr_prefix_t Functions const char * cppr_err_to_str (cppr_return_t code) const char * cppr_err_to_desc (cppr_return_t code) const char * cppr_lver_to_str (void) cppr_return_t cppr_sver_to_str (const char **version) cppr_return_t cppr_prefix_get (cppr_prefix_t type, const char **prefix) cppr_return_t cppr_status (void) cppr_return_t cppr_mv (struct cppr_op_info *op_info, uint32_t flags, const char *dest_prefix, const char *src_prefix, const char *pathname) cppr_return_t cppr_mv_wait (uint32_t flags, const char *dest_prefix, const char *src_prefix, const char *pathname) cppr_return_t cppr_mv_cget (struct cppr_op_info *op_info, uint32_t flags, const char *local_prefix, const char *global_prefix, const char *pathname) cppr_return_t cppr_mv_cget_wait (uint32_t flags, const char *local_prefix, const char *global_prefix, const char *pathname) cppr_return_t cppr_mv_revalidate (const char *pathname) cppr_return_t cppr_wait (struct cppr_op_info *info, uint32_t timeoutms) cppr_return_t cppr_wait_all (uint32_t count, struct cppr_op_info info[], uint32_t timeoutms) cppr_return_t cppr_test (struct cppr_op_info *info) cppr_return_t cppr_test_all (uint32_t count, struct cppr_op_info info[]) Detailed Description 9

10 4.1.7 Macro Definition Documentation #define CPPR_DEFAULT_STRIPE_OFFSET ((uint32_t)-1) Constant to select the default stripe offset #define CPPR_DEFAULT_STRIPE_SIZE 0 Constant to select the default stripe size #define CPPR_HANDLE_FMT "CPPR Handle(%llx)" Macro for printing a handle format string #define CPPR_HANDLE_VALUE( info) ULL(((info)->handle & ~((cppr_handle_t)0x ))) Macro for printing a handle value #define CPPR_INFINITE (0) Infinite timeout period Examples: mover.c #define CPPR_INVALID_HANDLE Identifies a cppr_handle_t as invalid Typedef Documentation typedef uint64_t cppr_handle_t A handle to a CPPR operation Enumeration Type Documentation enum cppr_flag_t Flags affecting behavior of CPPR operations 10

11 Enumerator CPPR_FLAG_OV ERWRITE Description Overwrite existing files. If flag isn't set, corresponding operation will fail if destination file already exists enum cppr_op_status_t CPPR operation status Enumerator CPPR_STATUS_ UNKNOWN CPPR_STATUS_ PENDING CPPR_STATUS_ COMPLETE Description The default operation status if something went wrong getting the status Operation is still pending completion. Operation is complete. Check operation status for errors enum cppr_prefix_t Identifies prefix type used in cppr_prefix_get API enum cppr_return_t Return code for CPPR API routines Enumerator CPPR_SUCCESS CPPR_NO_PREFI X Function executed correctly CNSS_PREFIX is not set Description CPPR_NO_CC CPPR_INCOMPA TIBLE_VERSION CPPR_NO_SERVI CES CPPR_INSUFFICI ENT_ARGS CPPR_INVALID_ ARG CPPR_NOT_IMP LEMENTED CPPR_INTERNAL _ERROR No cooperative cache is present Incompatible service version Services not running Insufficient arguments Invalid argument The requested operation is not implemented Internal error 11

12 Enumerator CPPR_OUT_OF_ MEMORY CPPR_QUOTA_L IMIT CPPR_OP_EXEC UTING CPPR_UNKNOW N_HANDLE CPPR_UNSUPPO RTED_MOVE_TY PE CPPR_SINGLE_ WAITER_VIOLAT ION CPPR_INVALID_ PATH CPPR_SOURCE_ MISSING CPPR_NOT_A_DI RECTORY CPPR_INVALID_ DEST_FS CPPR_NO_READ _PERMISSION CPPR_NO_WRIT E_PERMISSION CPPR_INSUFFICI ENT_SPACE CPPR_PATH_EXI STS CPPR_NAMESPA CE_CONFLICT CPPR_THREAD_ CREATE_FAILUR E CPPR_FAILED CPPR_UNDEFIN ED Description Insufficient memory to complete request The requested operation hit the quota limit Operation is still executing Unknown handle Unsupported move type Single waiter violation Source path does not exist Source path does not exist A directory name in the destination path is not a not a directory. Unsupported filesystem in destination path No read permission on source path No write permission on destination file Insufficient space on destination path Destination path already exists Multiple processes requested the same destination path Thread create failed Requested action failed to complete Undefined value. The value can't be determined because the operation is still in progress 12

13 Enumerator CPPR_BAD_DAT A Description Bad data. Data provided was not valid Function Documentation const char* cppr_err_to_desc (cppr_return_t code) Return a short description of an error code Parameters: in code A return code to translate Returns: A pointer to a constant string describing the return code Examples: mover.c const char* cppr_err_to_str (cppr_return_t code) Return a string representation of an error code Parameters: in code A return code to translate Returns: A string representation of the enumeration const char* cppr_lver_to_str (void ) Return the client library version as a string. Examples: mover.c. 13

14 cppr_return_t cppr_mv (struct cppr_op_info * op_info, uint32_t flags, const char * dest_prefix, const char * src_prefix, const char * pathname) Create a handle for a CPPR transfer and return. Individual files are copied in their entirety to the destination or not at all. If pathname is a directory, the same semantic applies to each individual file in the subtree. Directories are created as needed on the destination. Parameters: out op_info A handle for the transfer. in flags Configuration flags for the transfer. See cppr_flag_t. in dest_prefix Path prefix on the destination filesystem in src_prefix Path prefix on the source filesystem in pathname Path relative to the prefix of a file or directory to be copied Return values: CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBL E_VERSIONS CPPR_OUT_OF_MEM ORY CPPR_INVALID_ARG Operation handle is successfully created The services process is not available Library initialization failed due to incompatibility between the session services and the library. CPPR has insufficient memory to handle the request. Bad argument passed to routine Examples: mover.c cppr_return_t cppr_mv_cget (struct cppr_op_info * op_info, uint32_t flags, const char * local_prefix, const char * global_prefix, const char * pathname) Identical to cppr_mv for a single file except that the transfer is optimized using scalable communication and aggressive caching for the case where many or all CNs are requesting the same file to be transfered. 14

15 Parameters: out op_info A handle for the transfer. in flags Configuration flags for the transfer. See cppr_flag_t. in local_prefix Path prefix on local filesystem supporting op in global_prefix Path prefix on the global filesystem in pathname Path relative to the prefix of a file to be copied Return values: CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBL E_VERSIONS CPPR_INVALID_ARG Operation handle is successfully created The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine Examples: mover.c cppr_return_t cppr_mv_cget_wait (uint32_t flags, const char * local_prefix, const char * global_prefix, const char * pathname) Equivalent to calling cppr_mv_cget followed by cppr_wait with an infinite timeout. Returns the operation return code. Parameters: in flags Configuration flags for the transfer. See cppr_flag_t. in local_prefix Path prefix on local filesystem supporting op in global_prefix Path prefix on the global filesystem in pathname Path relative to the prefix of a file to be copied Return values: CPPR_SUCCESS CPPR_NO_SERVICES Routine executed successfully. See info for status of transfer The services process is not available 15

16 CPPR_INCOMPATIBL E_VERSIONS CPPR_INVALID_ARG Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine cppr_return_t cppr_mv_revalidate (const char * pathname) Searches the cooperative cache entries in which global namespace files match pathname directory. If the pathname is NULL, all cached files are matched The function only marks the files for revalidation on a subsequent call to cppr_mv_cget for matching files. No local files are deleted. Parameters: in pathname directory to revalidate. Any file in a global namespace matching the pathname will be marked for revalidation. Return values: CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBL E_VERSIONS CPPR_INVALID_ARG All cached copies have been marked for revalidation The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine cppr_return_t cppr_mv_wait (uint32_t flags, const char * dest_prefix, const char * src_prefix, const char * pathname) Equivalent to calling cppr_mv followed by cppr_wait with an infinite timeout. Returns the operation return code If conflict detection is enabled, transfers using it will be checked globally for namespace conflicts. Otherwise, it is ignored. Parameters: in flags Configuration flags for the transfer. See cppr_flag_t. in dest_prefix Path prefix on the destination filesystem in src_prefix Path prefix on the source filesystem in pathname Path relative to the prefix of a file or directory to be copied 16

17 Return values: CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBL E_VERSIONS CPPR_OUT_OF_MEM ORY CPPR_INVALID_ARG Routine executed successfully. See info for status of transfer The services process is not available Library initialization failed due to incompatibility between the session services and the library. CPPR has insufficient memory to handle the request. Bad argument passed to routine cppr_return_t cppr_prefix_get (cppr_prefix_t type, const char ** prefix) Get prefix of the specified type Parameters: in type Type of the prefix requested in,out prefix Returns pointer to constant prefix string. If CPPR is not initialized, returns NULL Return values: CPPR_SUCCESS CPPR_NO_SERVICES CPPR_NO_CC CPPR_INCOMPATIBL E_VERSIONS Success The service process is not available No CC namespace present Library initialization failed due to incompatability between the session services and the library. Examples: mover.c cppr_return_t cppr_status (void ) Returns the status of CPPR services Return values: CPPR_SUCCESS CPPR_NO_SERVICES Success The services process is not available 17

18 CPPR_INCOMPATIBL E_VERSIONS Library initialization failed due to incompatibility between the session services and the library. Examples: mover.c cppr_return_t cppr_sver_to_str (const char ** version) Return the services process version as a string. Parameters: in,out version Returns pointer to constant version string. If CPPR is not initialized, returns NULL Return values: CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBL E_VERSIONS Success The services process is not available Library initialization failed due to incompatibility between the session services and the library. Examples: mover.c cppr_return_t cppr_test (struct cppr_op_info * info) Test the current status of an operation Parameters: in,out info User sets the handle field. Function populates the other fields according to the status of the operation Return values: CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_ VERSIONS CPPR_INVALID_ARG Routine executed correctly. See info for status of the operation The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine 18

19 Note: On return, if the operation status is CPPR_STATUS_COMPLETE, then the handle is no longer valid. However, the handle gets modified such that subsequent calls will be a noop if the same structure is used cppr_return_t cppr_test_all (uint32_t count, struct cppr_op_info info[]) Test the current status of all of the specified operations Parameters: in count Number of operations in,out info Array of size count. User populates the handle field for all entries. Function populates the remaining fields according to the status of the operations. Return values: CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBL E_VERSIONS CPPR_INVALID_ARG Routine executed correctly. See info for status of the operations The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine Note: On successful return, the handles for operations with status CPPR_STATUS_COMPLETE are no longer valid. However, the handle gets modified such that subsequent calls will be a noop for that entry cppr_return_t cppr_wait (struct cppr_op_info * info, uint32_t timeoutms) Wait for the asynchronous operation complete for a specified timeout interval Parameters: in,out info User sets the handle field. Function populates the other fields according to the status of the operation in timeoutms A timeout interval in milliseconds. Pass CPPR_INFINITE to block indefinitely. 19

20 Return values: CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBL E_VERSIONS CPPR_INVALID_ARG Routine executed correctly. See info for status of the operation The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine Note: On return, if the operation status is CPPR_STATUS_COMPLETE, then the handle is no longer valid. However, the handle gets modified such that subsequent calls will be a noop if the same structure is used cppr_return_t cppr_wait_all (uint32_t count, struct cppr_op_info info[], uint32_t timeoutms) Wait for all of the operations to complete or for a specified timeout interval Parameters: in count Number of operations in,out info Array of size count. User populates the handle field for all entries. Function populates the remaining fields according to the status of the operations. in timeoutms A timeout interval in milliseconds. Pass CPPR_INFINITE to block indefinitely. Return values: CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBL E_VERSIONS CPPR_INVALID_ARG Routine executed correctly. See info for status of the operations The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine 20

21 Note: On successful return, the handles for operations with status CPPR_STATUS_COMPLETE are no longer valid. However, the handle gets modified such that subsequent calls will be a noop for that entry. Examples: mover.c. 21

22 5 Class Documentation 5.1 cppr_op_info Struct Reference #include <cppr_types.h> Public Attributes cppr_handle_t handle cppr_op_status_t status cppr_return_t retcode uint64_t byte_progress Detailed Description Information about the current status of an operation Examples: mover.c Member Data Documentation uint64_t cppr_op_info::byte_progress Indicates the number of bytes copied so far for the operation cppr_handle_t cppr_op_info::handle The handle of the operation to query cppr_return_t cppr_op_info::retcode If the operation is complete, this will contain either CPPR_SUCCESS or an appropriate error if the operation was not successful. If the operation is still 22

23 pending, the value is CPPR_OP_EXECUTING. In the case of a wait request, this indicates that the timeout period expired. Examples: mover.c cppr_op_status_t cppr_op_info::status Completion status of the operation Examples: mover.c The documentation for this struct was generated from the following file: C:/Users/wcdurfey/Documents/HPDD/CPPR/New folder/new folder/src/include/cppr_types.h 23

24 6 Example Documentation 6.1 mover.c Demonstrates movement of multiple input files from global storage and overlapping computation with output I/O to move files to global storage. /* * All rights reserved. This program and the accompanying materials * are made available under the terms of the GNU Lesser General Public License * (LGPL) version 2.1 which accompanies this distribution, and is available at * * * This library is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. * * GOVERNMENT LICENSE RIGHTS-OPEN SOURCE SOFTWARE * The Government's rights to use, modify, reproduce, release, perform, * display, or disclose this software are subject to the terms of the LGPL * License as provided in Contract No. B * Any reproduction of computer software, computer software documentation, or * portions thereof marked with this legend must also reproduce the markings. * * (C) Copyright Intel Corporation. * * This file is a usage example for the CPPR API * * example/mover.c * * Author: Jeff Olivier <jeffrey.v.olivier@intel.com> */ #include <stdlib.h> #include <stdio.h> #include <assert.h> #include <cppr.h> #define NUM_ITERATIONS 10 24

25 #define NUM_SETUP_OPS 3 #define NUM_CHECKPOINT_OPS 2 int main(void) { struct cppr_op_info info[num_setup_ops]; cppr_return_t ret_code; int i; const char *nl_prefix; ret_code = cppr_status(); if (ret_code == CPPR_INCOMPATIBLE_VERSION) { const char *server_version; } cppr_sver_to_str(&server_version); printf("cppr couldn't be initialized due to version mismatch\n" " header: %s\n" " service: %s\n", cppr_lver_to_str(), server_version); return -1; /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Get the node local namespace prefix */ ret_code = cppr_prefix_get(cppr_prefix_nl, &nl_prefix); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Assume file1 is a file-per-process data file */ ret_code = cppr_mv(&info[0], 0, "/mnt/lustre", nl_prefix, "file1"); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Assume file2 is a some read-only file that we want on all * compute nodes. * This function should be used to ensure that CNs cooperate with 25

26 * eachother and don't create a distributed denial of service * attack on the global filesystem. */ ret_code = cppr_mv_cget(&info[1], 0, "/mnt/lustre", nl_prefix, "file2"); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Do some other things */ ret_code = cppr_wait_all(num_setup_ops, info, CPPR_INFINITE); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); assert(info[0].status == CPPR_STATUS_COMPLETE); assert(info[1].status == CPPR_STATUS_COMPLETE); assert(info[2].status == CPPR_STATUS_COMPLETE); if (info[0].retcode!= CPPR_SUCCESS) { printf("transfer of file1 failed: %s\n", cppr_err_to_desc(info[0].retcode)); return -1; } if (info[1].retcode!= CPPR_SUCCESS) { printf("transfer of file2 failed: %s\n", cppr_err_to_desc(info[1].retcode)); return -1; } if (info[2].retcode!= CPPR_SUCCESS) { printf("transfer of file3 failed: %s\n", cppr_err_to_desc(info[2].retcode)); return -1; } for (i = 0; i < NUM_ITERATIONS; i++) { /* Do computations...creating new outputs for each * iteration */ char output1[32]; char output2[32]; 26

27 sprintf(output1, "output1_%03d", i); sprintf(output2, "output2_%03d", i); /* Do computation and write output1 and output2 */ /* Wait for a previous transfer before starting the next */ if (i!= 0) { ret_code = cppr_wait_all(num_checkpoint_ops, info, CPPR_INFINITE); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); assert(info[0].status == CPPR_STATUS_COMPLETE); assert(info[1].status == CPPR_STATUS_COMPLETE); } printf("node local transfer %d code: %d (%s)\n", i-1, info[0].retcode, cppr_err_to_desc(info[0].retcode)); printf("shared file transfer %d code: %d (%s)\n", i-1, info[1].retcode, cppr_err_to_desc(info[1].retcode)); ret_code = cppr_mv(&info[0], 0, nl_prefix, "/mnt/lustre", output1); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); } /* wait for the last transfer */ ret_code = cppr_wait_all(num_checkpoint_ops, info, CPPR_INFINITE); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); assert(info[0].status == CPPR_STATUS_COMPLETE); assert(info[1].status == CPPR_STATUS_COMPLETE); printf("transfer of node local output (%d) return code: %d (%s)\n", info[0].retcode, i-1, cppr_err_to_desc(info[0].retcode)); printf("transfer of sf output (%d) return code: %d (%s)\n", 27

28 info[1].retcode, i-1, cppr_err_to_desc(info[1].retcode)); } return 0; 28

29 7 Index byte_progress cppr_op_info CPPR API... 8 CPPR_BAD_DATA CPPR_DEFAULT_STRIPE_OFFSET CPPR_DEFAULT_STRIPE_SIZE cppr_err_to_desc cppr_err_to_str CPPR_FAILED CPPR_FLAG_OVERWRITE cppr_flag_t CPPR_HANDLE_FMT cppr_handle_t CPPR_HANDLE_VALUE CPPR_INCOMPATIBLE_VERSION CPPR_INFINITE CPPR_INSUFFICIENT_ARGS CPPR_INSUFFICIENT_SPACE CPPR_INTERNAL_ERROR CPPR_INVALID_ARG CPPR_INVALID_DEST_FS CPPR_INVALID_HANDLE CPPR_INVALID_PATH cppr_lver_to_str cppr_mv cppr_mv_cget cppr_mv_cget_wait cppr_mv_revalidate cppr_mv_wait CPPR_NAMESPACE_CONFLICT CPPR_NO_CC CPPR_NO_PREFIX CPPR_NO_READ_PERMISSION CPPR_NO_SERVICES CPPR_NO_WRITE_PERMISSION CPPR_NOT_A_DIRECTORY CPPR_NOT_IMPLEMENTED CPPR_OP_EXECUTING cppr_op_status_t CPPR_OUT_OF_MEMORY CPPR_PATH_EXISTS cppr_prefix_get cppr_prefix_t CPPR_QUOTA_LIMIT cppr_return_t CPPR_SINGLE_WAITER_VIOLATION CPPR_SOURCE_MISSING cppr_status CPPR_STATUS_COMPLETE CPPR_STATUS_PENDING CPPR_STATUS_UNKNOWN CPPR_SUCCESS cppr_sver_to_str cppr_test cppr_test_all CPPR_THREAD_CREATE_FAILURE CPPR_UNDEFINED CPPR_UNKNOWN_HANDLE CPPR_UNSUPPORTED_MOVE_TYPE cppr_wait cppr_wait_all CPPR_BAD_DATA CPPR API CPPR_DEFAULT_STRIPE_OFFSET CPPR API CPPR_DEFAULT_STRIPE_SIZE CPPR API cppr_err_to_desc CPPR API cppr_err_to_str CPPR API CPPR_FAILED CPPR API CPPR_FLAG_OVERWRITE CPPR API cppr_flag_t CPPR API CPPR_HANDLE_FMT CPPR API cppr_handle_t CPPR API CPPR_HANDLE_VALUE CPPR API CPPR_INCOMPATIBLE_VERSION CPPR API CPPR_INFINITE CPPR API CPPR_INSUFFICIENT_ARGS CPPR API CPPR_INSUFFICIENT_SPACE CPPR API CPPR_INTERNAL_ERROR CPPR API CPPR_INVALID_ARG 29

30 CPPR API CPPR_INVALID_DEST_FS CPPR API CPPR_INVALID_HANDLE CPPR API CPPR_INVALID_PATH CPPR API cppr_lver_to_str CPPR API cppr_mv CPPR API cppr_mv_cget CPPR API cppr_mv_cget_wait CPPR API cppr_mv_revalidate CPPR API cppr_mv_wait CPPR API CPPR_NAMESPACE_CONFLICT CPPR API CPPR_NO_CC CPPR API CPPR_NO_PREFIX CPPR API CPPR_NO_READ_PERMISSION CPPR API CPPR_NO_SERVICES CPPR API CPPR_NO_WRITE_PERMISSION CPPR API CPPR_NOT_A_DIRECTORY CPPR API CPPR_NOT_IMPLEMENTED CPPR API CPPR_OP_EXECUTING CPPR API cppr_op_info byte_progress handle retcode status cppr_op_status_t CPPR API CPPR_OUT_OF_MEMORY CPPR API CPPR_PATH_EXISTS CPPR API cppr_prefix_get CPPR API cppr_prefix_t CPPR API CPPR_QUOTA_LIMIT CPPR API cppr_return_t CPPR API CPPR_SINGLE_WAITER_VIOLATION CPPR API CPPR_SOURCE_MISSING CPPR API cppr_status CPPR API CPPR_STATUS_COMPLETE CPPR API CPPR_STATUS_PENDING CPPR API CPPR_STATUS_UNKNOWN CPPR API CPPR_SUCCESS CPPR API cppr_sver_to_str CPPR API cppr_test CPPR API cppr_test_all CPPR API CPPR_THREAD_CREATE_FAILURE CPPR API CPPR_UNDEFINED CPPR API CPPR_UNKNOWN_HANDLE CPPR API CPPR_UNSUPPORTED_MOVE_TYPE CPPR API cppr_wait CPPR API cppr_wait_all CPPR API handle cppr_op_info retcode cppr_op_info status cppr_op_info