Common Persistent Memory POSIX Runtime (CPPR) API Reference Manual. Reference Manual High Performance Data Division

Transcription

1 Common Persistent Memory POSIX Runtime (CPPR) Reference Manual High Performance Data Division INTEL FEDERAL, LLC PROPRIETARY October 2016

2 Generated under Argonne Contract number: B DISTRIBUTION STATEMENT: None Required Disclosure Notice: This presentation is bound by Non-Disclosure Agreements between Intel Corporation, the Department of Energy, and DOE National Labs, and is therefore for Internal Use Only and not for distribution outside these organizations or publication outside this Subcontract. USG Disclaimer: This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof. Export: This document contains information that is subject to export control under the Export Administration Regulations. Intel Disclaimer: Intel makes available this document and the information contained herein in furtherance of CORAL. None of the information contained therein is, or should be construed, as advice. While Intel makes every effort to present accurate and reliable information, Intel does not guarantee the accuracy, completeness, efficacy, or timeliness of such information. Use of such information is voluntary, and reliance on it should only be undertaken after an independent review by qualified experts. Access to this document is with the understanding that Intel is not engaged in rendering advice or other professional services. Information in this document may be changed or updated without notice by Intel. This document contains copyright information, the terms of which must be observed and followed. Reference herein to any specific commercial product, process or service does not constitute or imply endorsement, recommendation, or favoring by Intel or the US Government. Intel makes no representations whatsoever about this document or the information contained herein. IN NO EVENT SHALL INTEL BE LIABLE TO ANY PARTY FOR ANY DIRECT, INDIRECT, SPECIAL OR OTHER CONSEQUENTIAL DAMAGES FOR ANY USE OF THIS DOCUMENT, INCLUDING, WITHOUT LIMITATION, ANY LOST PROFITS, BUSINESS INTERRUPTION, OR OTHERWISE, EVEN IF INTEL IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Company Name: Intel Federal, LLC Company Address: 4100 Monument Corner Drive, Suite 540 Fairfax, VA Copyright , Intel Corporation. Technical Lead: (Name, , Phone) _Al Gara, Contract Administrator: (Name, , Phone) Aaron Matzkin, Program Manager: (Name, , Phone) Jacob Wood, Intel Federal, LLC Proprietary Page 2 of 40

3 Document Revision History Revision Number Date Comments 0.3 October 2016 Initial version or add short informative comments about changes made; add most recent revision in new row at bottom of table Intel Federal, LLC Proprietary Page 3 of 40

4 Contents 1 CPPR Module Index Modules Class Index Class List Module Documentation CPPR API Detailed Description Macro Definition Documentation Typedef Documentation Enumeration Type Documentation Function Documentation Class Documentation cppr_op_info Struct Reference Public Attributes Detailed Description Member Data Documentation Example Documentation group.c mover.c redundancy.c Intel Federal, LLC Proprietary Page 4 of 40

5 1 CPPR This manual describes the CPPR API. Applications may use this API to request that files be copied from global filesystems to local storage on compute nodes and back again. CPPR provides two namespaces storing data locally: a local per-node namespace for file-per-node data and a global shared file namespace. The API supports 2 basic mover operations: Move - Copy a file from one location to another, if the file systems are supported Cooperative Get - Copy from a global file system to the local namespace, but optimizing for the case where the same file requested by many compute nodes. The basic operations are non-blocking but blocking versions are provided. Additionally, the API supports limited global transactions to enable multi-process applications to know whether all operations in the transaction have successfully completed. A flag file is written upon successful completion of all operations in the transaction. Should at least one failure occur, no flag file is generated but no files are removed. Intel Federal, LLC Proprietary Page 5 of 40

6 2 Module Index 2.1 Modules Here is a list of all modules: CPPR API... 7 Intel Federal, LLC Proprietary Page 6 of 40

7 3 Class Index 3.1 Class List Here are the classes, structs, unions and interfaces with brief descriptions: cppr_op_info Intel Federal, LLC Proprietary Page 7 of 40

8 4 Module Documentation 4.1 CPPR API Classes struct cppr_op_info Macros #define CPPR_GLOBAL_COOKIE 0xffffffff #define CPPR_INVALID_HANDLE #define CPPR_INFINITE (0) #define CPPR_DEFAULT_STRIPE_SIZE 0 #define CPPR_DEFAULT_STRIPE_OFFSET ((uint32_t)-1) Typedefs typedef uint64_t cppr_handle_t typedef char cppr_attr_t[16] Enumerations Functions const char cppr_err_to_str (cppr_return_t code) const char cppr_err_to_desc (cppr_return_t code) const char cppr_lver_to_str (void) cppr_return_t cppr_sver_to_str (const char version) cppr_return_t cppr_nl_prefix_get (const char prefix) cppr_return_t cppr_cc_prefix_get (const char prefix) cppr_return_t cppr_sf_prefix_get (const char prefix) cppr_return_t cppr_status (void) cppr_return_t cppr_mv (cppr_handle_t handle, uint32_t cookie, uint32_t flags, cppr_attr_t attributes, const char dest_prefix, const char src_prefix, const char pathname) cppr_return_t cppr_mv_wait (uint32_t cookie, uint32_t flags, cppr_attr_t attributes, const char dest_prefix, const char src_prefix, const char pathname) cppr_return_t cppr_set_sf_attr (cppr_attr_t attributes, uint32_t stripe_size, uint32_t stripe_offset) Intel Federal, LLC Proprietary Page 8 of 40

9 cppr_return_t cppr_mv_cget (cppr_handle_t handle, uint32_t cookie, uint32_t flags, const char local_prefix, const char global_prefix, const char pathname) cppr_return_t cppr_mv_cget_wait (uint32_t cookie, uint32_t flags, const char local_prefix, const char global,_prefix, const char pathname) cppr_return_t cppr_mv_revalidate (const char pathname) cppr_return_t cppr_wait (struct cppr_op_info info, uint32_t timeoutms) cppr_return_t cppr_wait_all (uint32_t count, struct cppr_op_info info[ ], uint32_t timeoutms) cppr_return_t cppr_test (struct cppr_op_info info) cppr_return_t cppr_test_all (uint32_t count, struct cppr_op_info info[ ]) cppr_return_t cppr_op_info_print (uint32_t count, struct cppr_op_info info[ ]) cppr_return_t cppr_grp_create (cppr_handle_t handle, uint32_t cookie, const char flagpath, uint32_t num_,procs) cppr_return_t cppr_grp_commit (cppr_handle_t handle) cppr_return_t cppr_rs_grp_create (cppr_handle_t set_handle, cppr_redundancy_scheme_t scheme, cppr_attr_t attributes, const char set_id, uint32_t version, uint32_t num_procs) cppr_return_t cppr_rs_add_file (cppr_handle_t set_handle, uint32_t rank, const char pathname) cppr_return_t cppr_rs_query_last (const char set_id, uint32_t version) cppr_return_t cppr_rs_rebuild (cppr_handle_t set_handle, const char set_id, uint32_t version, uint32_t rank) cppr_return_t cppr_rs_remove (cppr_handle_t set_handle, const char set_id, uint32_t version, uint32_t rank) Detailed Description Macro Definition Documentation #define CPPR_DEFAULT_STRIPE_OFFSET ((uint32_t)-1) Constant to select the default stripe offset #define CPPR_DEFAULT_STRIPE_SIZE 0 Constant to select the default stripe size Intel Federal, LLC Proprietary Page 9 of 40

10 #define CPPR_GLOBAL_COOKIE 0xffffffff A cookie reserved for global conflict detection mover.c #define CPPR_INFINITE (0) Infinite timeout period group.c, mover.c, and redundancy.c #define CPPR_INVALID_HANDLE Identifies a cppr_handle_t as invalid Typedef Documentation typedef char cppr_attr_t[16] Opaque structure to store operation attributes typedef uint64_t cppr_handle_t A handle to a CPPR operation Enumeration Type Documentation enum cppr_flag_t Flags affecting behavior of CPPR operations Enumerator CPPR_FLAG_OVERWRITE Overwrite existing files CPPR_FLAG_DETECT_SOURCE_MODIFICATION Detect modification of the source file during transfer. If modification is detected, an error will be reported as CPPR_SOURCE_MODIFIED. CPPR_FLAG_TRACK_PROGRESS Track progress of operation CPPR_FLAG_DISABLE_CONFLICT_DETECTION Disable default conflict detection. Conflict detection requires coordination and communication between nodes and incurs a small memory overhead. This flag disables it for the current operation CPPR_FLAG_LIMIT_CONFLICT_DETECTION_SCOPE Limit conflict detection to operations in transaction. This enables more efficient memory usage at the cost of not detecting conflicts globally. Intel Federal, LLC Proprietary Page 10 of 40

11 CPPR_FLAG_REVALIDATE Perform revalidation on the file enum cppr_op_status_t CPPR operation status Enumerator CPPR_STATUS_UNKNOWN The default operation status if something went wrong getting the status CPPR_STATUS_PENDING Operation is still pending completion. CPPR_STATUS_COMPLETE Operation is complete. Check operation status for errors enum cppr_redundancy_scheme_t Identifies the type of redundancy scheme to apply to a set Enumerator CPPR_RS_XOR Copy to a partner node CPPR_RS_REED_SOLOMON XOR parity enum cppr_return_t Return code for CPPR API routines Enumerator CPPR_SUCCESS Function executed correctly CPPR_NO_PREFIX CPPR_PREFIX is not set CPPR_NO_COMM Communication failure CPPR_NO_SF No CPPR shared file present CPPR_NO_CC No cooperative cache is present CPPR_INCOMPATIBLE_VERSION Incompatible service version CPPR_NO_SERVICES Services not running CPPR_INSUFFICIENT_ARGS Insufficient arguments CPPR_INVALID_ARG Invalid argument CPPR_USAGE_ERROR Usage error CPPR_NOT_IMPLEMENTED The requested operation is not implemented CPPR_INTERNAL_ERROR Internal error CPPR_OUT_OF_MEMORY Insufficient memory to complete request Intel Federal, LLC Proprietary Page 11 of 40

12 CPPR_UNKNOWN_HANDLE Unknown handle CPPR_UNSUPPORTED_MOVE_TYPE Unsupported move type CPPR_SINGLE_WAITER_VIOLATION Single waiter violation CPPR_INVALID_PATH Source path does not exist CPPR_SOURCE_MISSING Source path does not exist CPPR_NOT_A_DIRECTORY A directory name in the destination path is not a not a directory. CPPR_INVALID_SOURCE_FS Unsupported filesystem in source path CPPR_INVALID_DEST_FS Unsupported filesystem in destination path CPPR_NO_READ_PERMISSION No read permission on source path CPPR_NO_WRITE_PERMISSION No write permission on destination file CPPR_INSUFFICIENT_SPACE Insufficient space on destination path CPPR_PATH_EXISTS Destination path already exists CPPR_SOURCE_MODIFIED The source path was modified during transfer CPPR_NAMESPACE_CONFLICT Multiple processes requested the same destination path CPPR_THREAD_CREATE_FAILURE Thread create failed CPPR_FAILED Requested action failed to complete CPPR_DEAD_NODE Specified node is dead CPPR_UNDEFINED Undefined value. The value can't be determined because the operation is still in progress Function Documentation cppr_return_t cppr_cc_prefix_get ( const char prefix ) Get cooperative cache namespace prefix. in,out prefix Returns pointer to constant prefix string. If CPPR is not initialized, returns NULL Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS Success The services process is not available Library initialization failed due to incompatibility between the session services and the library. Intel Federal, LLC Proprietary Page 12 of 40

13 const char cppr_err_to_desc ( cppr_return_t code ) Return a short description of an error code in code A return code to translate Returns A pointer to a constant string describing the return code mover.c const char cppr_err_to_str ( cppr_return_t code ) Return a string representation of an error code in code A return code to translate Returns A string representation of the enumeration cppr_return_t cppr_grp_commit ( cppr_handle_t handle ) Indicates that the calling process will not be starting any new operations in the group. If a flagpath file is requested, the file created only after all involved processes have called commit and only when all associated operations successfully execute. If any operation fails, the operation group is marked complete with an error status and the file is not created. If for some reason, the flag file exists at the time of commit, the file will be removed and an error will be reported upon querying the operation group status. Memory associated with the group is not deallocated until a call to cppr_wait, cppr_wait_all, cppr_test, cppr_test_all indicates that the operation group has completed. in handle Handle returned by cppr_grp_create Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS CPPR_INVALID_ARG Routine executed correctly. The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine Intel Federal, LLC Proprietary Page 13 of 40

14 group.c, and redundancy.c cppr_return_t cppr_grp_create ( cppr_handle_t handle, uint32_t cookie, const char flagpath, uint32_t num_procs) Registers a user-defined cookie globally. The cookie will be used to identify all operations involved in an operation group. An application should not use the same cookie for more than one operation group. Each rank involved in the group must call this routine with the same value for cookie and ranks. out handle Returns a handle that can be used to query the status of the operation group. in cookie A user defined value identifying the operation group in flagpath Path to file to create on successful completion of all operations in the group in num_procs The number of processes involved in the operation group. Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS CPPR_INVALID_ARG CPPR_USAGE_ERROR Routine executed correctly. The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine Invalid usage of routine NOTE: If a process specifies a different value for num_procs for an existing operation group, CPPR_INVALID_ARG will be returned. Any additional processes beyond num_procs will get CPPR_USAGE_ERROR as a return code. If flagpath is NULL, no flag file will be created after commit. If a file exists at the specified location, CPPR_INVALID_ARG will be returned. Should any error occur, no new operation group is created. group.c const char cppr_lver_to_str ( void ) Return the client library version as a string. group.c, mover.c, and redundancy.c. Intel Federal, LLC Proprietary Page 14 of 40

15 cppr_return_t cppr_mv ( cppr_handle_t handle, uint32_t cookie, uint32_t flags, cppr_attr_t attributes, const char dest_prefix, const char src_prefix, const char pathname ) Create a handle for a CPPR transfer and return. Individual files are copied in their entirety to the destination or not at all. If pathname is a directory, the same semantic applies to each individual file in the subtree. Directories are created as needed on the destination. out handle A handle for the transfer. in cookie Identifies an operation group to which the operation belongs. Use CPPR_GLOBAL_COOKIE if not using operation groups. Conflict detection is done globally unless the flag CPPR_FLAG_LIMIT_CONFLICT_DETECTION_SCOPE is set. If a user-defined cookie doesn't identify a valid operation group (see cppr_grp_create), CPPR_INVALID_ARG is returned. in flags Configuration flags for the transfer. See cppr_flag_t. in attributes Optional attributes for passing addtional configuration for the transfer in dest_prefix Path prefix on the destination filesystem in src_prefix Path prefix on the source filesystem in pathname Path relative to the prefix of a file or directory to be copied Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS CPPR_OUT_OF_MEMORY CPPR_INVALID_ARG Operation handle is successfully created The services process is not available Library initialization failed due to incompatibility between the session services and the library. CPPR has insufficient memory to handle the request. Bad argument passed to routine group.c, and mover.c cppr_return_t cppr_mv_cget ( cppr_handle_t handle, uint32_t cookie, uint32_t flags, const char local_prefix, const char global_prefix, const char pathname ) Identical to cppr_mv for a single file except that the transfer is optimized using scalable communication and aggressive caching for the case where many or all CNs are requesting the same file to be transfered. out handle A handle for the transfer. in cookie If conflict detection is enabled, transfers using it will be checked globally for namespace conflicts. Otherwise, it is ignored. Intel Federal, LLC Proprietary Page 15 of 40

16 in flags Configuration flags for the transfer. See cppr_flag_t. in local_prefix Path prefix on local filesystem supporting op in global_prefix Path prefix on the global filesystem in pathname Path relative to the prefix of a file to be copied Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS CPPR_INVALID_ARG Operation handle is successfully created The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine mover.c cppr_return_t cppr_mv_cget_wait ( uint32_t cookie, uint32_t flags, const char local_prefix, const char global_prefix,const char pathname ) Equivalent to calling cppr_mv_cget followed by cppr_wait with an infinite timeout. Returns the operation return code. in cookie in flags in local_prefix in global_prefix in pathname If conflict detection is enabled, transfers using it will be checked globally for namespace conflicts. Otherwise, it is ignored. Configuration flags for the transfer. See cppr_flag_t. Path prefix on local filesystem supporting op Path prefix on the global filesystem Path relative to the prefix of a file to be copied Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS CPPR_INVALID_ARG Routine executed successfully. See info for status of transfer The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine cppr_return_t cppr_mv_revalidate ( const char pathname ) Searches the cooperative cache entries in which global namespace files match pathname directory. If the pathname is NULL, all cached files are matched The function only marks the files for revalidation on a subsequent call to cppr_ mv_cget for matching files. No local files are deleted. Intel Federal, LLC Proprietary Page 16 of 40

17 in pathname Directory to revalidate. Any file in a global namespace matching the pathname will be marked for revalidation. Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS CPPR_INVALID_ARG All cached copies have been marked for revalidation The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine cppr_return_t cppr_mv_wait ( uint32_t cookie, uint32_t flags, cppr_attr_t attributes, const char dest_prefix, const char src_prefix, const char pathname ) Equivalent to calling cppr_mv followed by cppr_wait with an infinite timeout. Returns the operation return code in cookie in flags in attributes in dest_prefix in src_prefix in pathname If conflict detection is enabled, transfers using it will be checked globally for namespace conflicts. Otherwise, it is ignored. Configuration flags for the transfer. See cppr_flag_t. Optional attributes for passing addtional configuration for the transfer Path prefix on the destination filesystem Path prefix on the source filesystem Path relative to the prefix of a file or directory to be copied Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS CPPR_OUT_OF_MEMORY CPPR_INVALID_ARG Routine executed successfully. See info for status of transfer The services process is not available Library initialization failed due to incompatibility between the session services and the library. CPPR has insufficient memory to handle the request. Bad argument passed to routine cppr_return_t cppr_nl_prefix_get ( const char prefix ) Get node-local namespace prefix in,out prefix Returns pointer to constant prefix string. If CPPR is not initialized, returns NULL Intel Federal, LLC Proprietary Page 17 of 40

18 Return values CPPR_SUCCESS Success Return values CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS The services process is not available Library initialization failed due to incompatibility between the session services and the library. group.c, mover.c, and redundancy.c cppr_return_t cppr_op_info_print ( uint32_t count, struct cppr_op_info info[ ] ) Pretty print the current status of operations as returned from previous calls to status inquiry APIs such as cppr_wait, cppr_wait_all, cppr_test, and cppr_test_all. in count in info Number of elements in info Array of size count containing status information retrieved from status inquiry APIs Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS CPPR_INVALID_ARG Routine executed correctly. The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine cppr_return_t cppr_rs_add_file ( cppr_handle_t set_handle, uint32_t rank, const char pathname ) Adds a file to a redundancy set in set_handle in rank in pathname A handle for the redundancy set The rank of the process Full pathname of a local file redundancy.c. Intel Federal, LLC Proprietary Page 18 of 40

19 cppr_return_t cppr_rs_grp_create ( cppr_handle_t set_handle, cppr_redundancy_scheme_t scheme, cppr_attr_t attributes, const char set_id, uint32_t version, uint32_t num_procs ) Starts a new, unique operation group to calculate redundancy on a set of files. out set_handle A handle for the redundancy set in scheme Redundancy scheme to use in attributes Optional attributes for modifying behavior of a scheme in set_id A unique identifier for all versions of the set of files in version The version of the set. New sets should use increasing versions in num_procs Number of processes involved redundancy.c cppr_return_t cppr_rs_query_last ( const char set_id, uint32_t version ) Returns the version of the last set created that can be rebuilt. in set_id The name used to build the redundancy set out version Returns the id of the last available version redundancy.c cppr_return_t cppr_rs_rebuild ( cppr_handle_t set_handle, const char set_id, uint32_t version, uint32_t rank ) Rebuild a specific version of a redundancy set, if available out set_handle A handle for the redundancy set in set_id The name used to build the redundancy set in version The numeric version of the set to rebuild in rank The rank of the process The function must be called by the same number of ranks involved in creating the set. Use cppr_wait/cppr_test to query the status of the set regeneration. Once complete and successful, the user can access the local files. Intel Federal, LLC Proprietary Page 19 of 40

20 redundancy.c cppr_return_t cppr_rs_remove ( cppr_handle_t set_handle, const char set_id, uint32_t version, uint32_t rank ) Starts removing a redundancy set version out set_handle A handle for the redundancy set in set_id The name used to build the redundancy set in version The numeric version of the set to remove in rank The rank of the process Use cppr_wait/cppr_test to query the status of the set regeneration. Once complete and successful, the user can access the local files cppr_return_t cppr_set_sf_attr ( cppr_attr_t attributes, uint32_t stripe_size, uint32_t stripe_offset ) Sets cppr_attr_t to pass flags relating to creation of files copied into CPPR shared file namespace via cppr_mv in,out attributes Pointer to structure to populate in stripe_size Stripe size in KB in stripe_offset Rank offset for stripe cppr_return_t cppr_sf_prefix_get ( const char prefix ) Get shared file namespace prefix in,out prefix Returns pointer to constant prefix string. If CPPR is not initialized, returns NULL Return values CPPR_SUCCESS CPPR_NO_SF CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS Success No shared file namespace present The services process is not available Library initialization failed due to incompatibility between the session services and the library. Intel Federal, LLC Proprietary Page 20 of 40

21 mover.c cppr_return_t cppr_status ( void ) Returns the status of CPPR services Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS Success The services process is not available Library initialization failed due to incompatibility between the session services and the library. group.c, mover.c, and redundancy.c cppr_return_t cppr_sver_to_str ( const char version ) Return the services process version as a string. in,out version Returns pointer to constant version string. If CPPR is not initialized, returns NULL Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS Success The services process is not available Library initialization failed due to incompatibility between the session services and the library. group.c, mover.c, and redundancy.c cppr_return_t cppr_test ( struct cppr_op_info info ) Test the current status of an operation in,out info User sets the handle field. Function populates the other fields according to the status of the operation Return values CPPR_SUCCESS CPPR_NO_SERVICES Routine executed correctly. See info for status of the operation The services process is not available Intel Federal, LLC Proprietary Page 21 of 40

22 CPPR_INCOMPATIBLE_VERSIONS CPPR_INVALID_ARG Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine Note: On return, if the operation status is CPPR_STATUS_COMPLETE, then the handle is no longer valid. However, the handle gets modified such that subsequent calls will be a noop if the same structure is used cppr_return_t cppr_test_all ( uint32_t count, struct cppr_op_info info[ ] ) Test the current status of all of the specified operations in count Number of operations in,out info Array of size count. User populates the handle field for all entries. Function populates the remaining fields according to the status of the operations. Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS CPPR_INVALID_ARG Routine executed correctly. See info for status of the operations The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine Note: On successful return, the handles for operations with status CPPR_STATUS_COMPLETE are no longer valid. However, the handle gets modified such that subsequent calls will be a noop for that entry cppr_return_t cppr_wait ( struct cppr_op_info info, uint32_t timeoutms ) Wait for the asynchronous operation complete for a specified timeout interval in,out info User sets the handle field. Function populates the other fields according to the status of the operation in timeoutms A timeout interval in milliseconds. Pass CPPR_INFINITE to block indefinitely. Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS Routine executed correctly. See info for status of the operation The services process is not available Library initialization failed due to incompatibility between the session services and the library. Intel Federal, LLC Proprietary Page 22 of 40

23 CPPR_INVALID_ARG Bad argument passed to routine Note: On return, if the operation status is CPPR_STATUS_COMPLETE, then the handle is no longer valid. However, the handle is modified such that subsequent calls will be a noop if the same structure is used. redundancy.c cppr_return_t cppr_wait_all ( uint32_t count, struct cppr_op_info info[ ], uint32_t timeoutms) Wait for all of the operations to complete or for a specified timeout interval in count Number of operations in,out info Array of size count. User populates the handle field for all entries. Function populates the remaining fields according to the status of the operations. in timeoutms A timeout interval in milliseconds. Pass CPPR_INFINITE to block indefinitely. Return values CPPR_SUCCESS CPPR_NO_SERVICES CPPR_INCOMPATIBLE_VERSIONS CPPR_INVALID_ARG Routine executed correctly. See info for status of the operations The services process is not available Library initialization failed due to incompatibility between the session services and the library. Bad argument passed to routine Note: On successful return, the handles for operations with status CPPR_STATUS_COMPLETE are no longer valid. However, the handle is modified such that subsequent calls will be a noop for that entry. group.c, mover.c, and redundancy.c. Intel Federal, LLC Proprietary Page 23 of 40

24 5 Class Documentation 5.1 cppr_op_info Struct Reference #include <cppr_types.h> Public Attributes cppr_handle_t handle cppr_op_status_t status cppr_return_t retcode uint32_t progress Detailed Description Information about the current status of an operation group.c, mover.c, and redundancy.c Member Data Documentation cppr_handle_t cppr_op_info::handle The handle of the operation to query group.c, mover.c, and redundancy.c uint32_t cppr_op_info::progress Indicates percent of operation completed [0-100] so far if the flag CPPR_FLAG_TRACK_PROGRESS is set for the operation cppr_return_t cppr_op_info::retcode If the operation is complete, this will contain either CPPR_SUCCESS or an appropriate error if the operation was not successful. If the operation is still pending, the value is CPPR_OP_EXECUTING. In the case of a wait request, this indicates that the timeout period expired. mover.c cppr_op_status_t cppr_op_info::status Completion status of the operation Intel Federal, LLC Proprietary Page 24 of 40

25 mover.c. Intel Federal, LLC Proprietary Page 25 of 40

26 6 Example Documentation 6.1 group.c Demonstrates usage of CPPR operation group API /* * All rights reserved. This program and the accompanying materials * are made available under the terms of the GNU Lesser General Public License * (LGPL) version 2.1 which accompanies this distribution, and is available at * * * This library is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. * * GOVERNMENT LICENSE RIGHTS-OPEN SOURCE SOFTWARE * The Government s rights to use, modify, reproduce, release, perform, * display, or disclose this software are subject to the terms of the LGPL * License as provided in Contract No. B * Any reproduction of computer software, computer software documentation, or * portions thereof marked with this legend must also reproduce the markings. * * (C) Copyright Intel Corporation. * * This file is a usage example for the CPPR API * * example/group.c * * Author: Jeff Olivier <jeffrey.v.olivier@intel.com> */ #include <stdlib.h> #include <stdio.h> #include <assert.h> Intel Federal, LLC Proprietary Page 26 of 40

27 #include <linux/limits.h> #include <mpi.h> #include <cppr.h> #define NUM_ITERATIONS 10 #define ITER 10 int main(int argc, char **argv) { struct cppr_op_info info[iter*2]; cppr_return_t ret_code; char pathname[path_max + 1]; char flag[path_max + 1]; int i; const char *nl_prefix; int num_procs; int rank; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &num_procs); MPI_Comm_rank(MPI_COMM_WORLD, &rank); ret_code = cppr_status(); Intel Federal, LLC Proprietary Page 27 of 40

28 if (ret_code == CPPR_INCOMPATIBLE_VERSION) { const char *server_version; } cppr_sver_to_str(&server_version); printf("cppr couldn t be initialized due to version mismatch\n" " header: %s\n" " service: %s\n", cppr_lver_to_str(), server_version); return -1; /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Get the node local namespace prefix */ ret_code = cppr_nl_prefix_get(&nl_prefix); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* All ranks execute this loop which generates a checkpoint for * every iteration. Each checkpoint is executed as a * CPPR operation group. The presence of the flag file indicates * that a particular operation group executed successfully. */ for (i = 0; i < ITER; i++) { sprintf(pathname, "%s/file-%d.%d", nl_prefix, rank, i); sprintf(flag, "/mnt/lustre/grpflag.%d", i); /* Computation to produce file at pathname */ /* Register this process with operation group #i */ ret_code = cppr_grp_create(&info[i].handle, i, flag, num_procs); Intel Federal, LLC Proprietary Page 28 of 40

29 /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Start the transfer */ ret_code = cppr_mv(&info[i+iter].handle, i, 0, NULL, "/mnt/lustre", nl_prefix, "file1"); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Notify that this process has started all operations in * the group */ ret_code = cppr_grp_commit(info[i].handle); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); } /* Wait for all of the operations to complete */ ret_code = cppr_wait_all(iter*2, info, CPPR_INFINITE); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* At this point, all flag files should exist */ return 0; } Intel Federal, LLC Proprietary Page 29 of 40

30 6.2 mover.c Demonstrates movement of multiple input files from global storage and overlapping computation with output I/O to move files to global storage. /* * All rights reserved. This program and the accompanying materials * are made available under the terms of the GNU Lesser General Public License * (LGPL) version 2.1 which accompanies this distribution, and is available at * * * This library is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. * * GOVERNMENT LICENSE RIGHTS-OPEN SOURCE SOFTWARE * The Government s rights to use, modify, reproduce, release, perform, * display, or disclose this software are subject to the terms of the LGPL * License as provided in Contract No. B * Any reproduction of computer software, computer software documentation, or * portions thereof marked with this legend must also reproduce the markings. * * (C) Copyright Intel Corporation. * * This file is a usage example for the CPPR API * * example/mover.c * * Author: Jeff Olivier <jeffrey.v.olivier@intel.com> */ #include <stdlib.h> #include <stdio.h> #include <assert.h> Intel Federal, LLC Proprietary Page 30 of 40

31 #include <cppr.h> #define NUM_ITERATIONS 10 #define NUM_SETUP_OPS 3 #define NUM_CHECKPOINT_OPS 2 int main(void) { struct cppr_op_info info[num_setup_ops]; cppr_return_t ret_code; int i; const char *nl_prefix; const char *sf_prefix; ret_code = cppr_status(); if (ret_code == CPPR_INCOMPATIBLE_VERSION) { const char *server_version; } cppr_sver_to_str(&server_version); printf("cppr couldn t be initialized due to version mismatch\n" " header: %s\n" " service: %s\n", cppr_lver_to_str(), server_version); return -1; /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); Intel Federal, LLC Proprietary Page 31 of 40

32 /* Get the node local namespace prefix */ ret_code = cppr_nl_prefix_get(&nl_prefix); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Get the node local namespace prefix */ ret_code = cppr_sf_prefix_get(&sf_prefix); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Assume file1 is a file-per-process data file */ ret_code = cppr_mv(&info[0].handle, CPPR_GLOBAL_COOKIE, 0, NULL, "/mnt/lustre", nl_prefix, "file1"); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Assume file2 is a some read-only file that we want on all * compute nodes. * This function should be used to ensure that CNs cooperate with * eachother and don t create a distributed denial of service * attack on the global filesystem. */ ret_code = cppr_mv_cget(&info[1].handle, CPPR_GLOBAL_COOKIE, 0, "/mnt/lustre", nl_prefix, "file2"); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); ret_code = cppr_mv(&info[2].handle, CPPR_GLOBAL_COOKIE, Intel Federal, LLC Proprietary Page 32 of 40

33 0, NULL, "/mnt/lustre", sf_prefix, "file3"); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Do some other things */ ret_code = cppr_wait_all(num_setup_ops, info, CPPR_INFINITE); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); assert(info[0].status == CPPR_STATUS_COMPLETE); assert(info[1].status == CPPR_STATUS_COMPLETE); assert(info[2].status == CPPR_STATUS_COMPLETE); if (info[0].retcode!= CPPR_SUCCESS) { printf("transfer of file1 failed: %s\n", cppr_err_to_desc(info[0].ret code)); } return -1; if (info[1].retcode!= CPPR_SUCCESS) { printf("transfer of file2 failed: %s\n", } cppr_err_to_desc(info[1].r etcode)); return -1; if (info[2].retcode!= CPPR_SUCCESS) { printf("transfer of file3 failed: %s\n", } cppr_err_to_desc(info[2].r etcode)); return -1; Intel Federal, LLC Proprietary Page 33 of 40

34 for (i = 0; i < NUM_ITERATIONS; i++) { /* Do computations...creating new outputs for each * iteration */ char output1[32]; char output2[32]; sprintf(output1, "output1_%03d", i); sprintf(output2, "output2_%03d", i); /* Do computation and write output1 and output2 */ /* Wait for a previous transfer before starting the next */ if (i!= 0) { ret_code = cppr_wait_all(num_checkpoint_ops, info, CPPR_INFINITE); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); assert(info[0].status == CPPR_STATUS_COMPLETE); assert(info[1].status == CPPR_STATUS_COMPLETE); printf("node local transfer %d code: %d (%s)\n", i-1, info[0].retcode, cppr_err_to_desc(info[0].ret code)); printf("shared file transfer %d code: %d (%s)\n", i-1, info[1].retcode, cppr_err_to_desc(info[1].retcode)); Intel Federal, LLC Proprietary Page 34 of 40

35 } ret_code = cppr_mv(&info[0].handle, CPPR_GLOBAL_COOKIE, 0, NULL, nl_prefix, "/mnt/lustre", output1); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); ret_code = cppr_mv(&info[1].handle, CPPR_GLOBAL_COOKIE, 0, NULL, sf_prefix, "/mnt/lustre", output2); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); } /* wait for the last transfer */ ret_code = cppr_wait_all(num_checkpoint_ops, info, CPPR_INFINITE); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); assert(info[0].status == CPPR_STATUS_COMPLETE); assert(info[1].status == CPPR_STATUS_COMPLETE); Intel Federal, LLC Proprietary Page 35 of 40

36 printf("transfer of node local output (%d) return code: %d (%s)\n", info[0].retcode, i-1, cppr_err_to_desc(info[0].retcode)); printf("transfer of sf output (%d) return code: %d (%s)\n", info[1].retcode, i-1, cppr_err_to_desc(info[1].retcode)); return 0; } 6.3 redundancy.c Demonstrates usage of CPPR redundancy API /* * All rights reserved. This program and the accompanying materials * are made available under the terms of the GNU Lesser General Public License * (LGPL) version 2.1 which accompanies this distribution, and is available at * * * This library is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. * * GOVERNMENT LICENSE RIGHTS-OPEN SOURCE SOFTWARE * The Government s rights to use, modify, reproduce, release, perform, * display, or disclose this software are subject to the terms of the LGPL * License as provided in Contract No. B * Any reproduction of computer software, computer software documentation, or * portions thereof marked with this legend must also reproduce the markings. * * (C) Copyright Intel Corporation. * * This file is a usage example for the CPPR API * Intel Federal, LLC Proprietary Page 36 of 40

37 * example/redundancy.c * * Author: Jeff Olivier <jeffrey.v.olivier@intel.com> */ #include <stdlib.h> #include <stdio.h> #include <assert.h> #include <linux/limits.h> #include <mpi.h> #include <cppr.h> #define NUM_ITERATIONS 10 #define ITER 10 int main(int argc, char **argv) { struct cppr_op_info info[iter]; cppr_return_t ret_code; char pathname[path_max + 1]; int i; const char *nl_prefix; int num_procs; int rank; uint32_t count; MPI_Init(&argc, &argv); Intel Federal, LLC Proprietary Page 37 of 40

38 MPI_Comm_size(MPI_COMM_WORLD, &num_procs); MPI_Comm_rank(MPI_COMM_WORLD, &rank); ret_code = cppr_status(); if (ret_code == CPPR_INCOMPATIBLE_VERSION) { const char *server_version; } cppr_sver_to_str(&server_version); printf("cppr couldn t be initialized due to version mismatch\n" " header: %s\n" " service: %s\n", cppr_lver_to_str(), server_version); return -1; /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Get the node local namespace prefix */ ret_code = cppr_nl_prefix_get(&nl_prefix); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); count = 0; /* Retrieve the last available version of the set */ if (cppr_rs_query_last("rs_set", &count) == CPPR_SUCCESS) { /* Start rebuilding the files */ ret_code = cppr_rs_rebuild(&info[0].handle, "rs_set", count, Intel Federal, LLC Proprietary Page 38 of 40

39 rank); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); ret_code = cppr_wait(&info[0], CPPR_INFINITE); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Insert code to read data from * nl_prefix/file-<rank>.<count> */ /* Increment counter to next checkpoint */ count++; } /* Else there are no previous checkpoints so initialize * data accordingly */ /* All ranks execute this loop which generates a checkpoint for * every iteration. Redundancy is calculated on each checkpoint. */ for (i = 0; i < ITER; i++) { sprintf(pathname, "%s/file-%d.%d", nl_prefix, rank, i+count); /* Computation to produce file at pathname */ /* Register a new redundancy set for checkpoint #i */ ret_code = cppr_rs_grp_create(&info[i].handle, CPPR_RS_REED_SOLOMON, NULL, "rs_set", i+count, num_procs); Intel Federal, LLC Proprietary Page 39 of 40

40 /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Start the transfer */ ret_code = cppr_rs_add_file(info[i].handle, rank, pathname); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* Notify that this process has added all files to the set */ ret_code = cppr_grp_commit(info[i].handle); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); } /* Wait for all of the operations to complete */ ret_code = cppr_wait_all(iter, info, CPPR_INFINITE); /* Assume better error handling */ assert(ret_code == CPPR_SUCCESS); /* At this point, all flag files should exist */ return 0; } Intel Federal, LLC Proprietary Page 40 of 40