ISILON FILENAME BASED PREFETCH

Transcription

1 White Paper ISILON FILENAME BASED PREFETCH UNDERSTANDING AND CONFIGURING FILENAME BASED PREFETCH WITH DELL EMC ISILON ONEFS 8.1 Abstract This document examines the OneFS feature of Filename based Prefetch. Filename-based Prefetch can yield improved performance for workflows where multiple files are read in sequence by an application - for example, image sequences in Media and Entertainment content creation workflows. It is intended that architects and system integrators use this document as a guide to both understanding the potential performance benefits of using Filename Based Prefetch, and how to configure and use the feature. February Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners. Reference Number: H16951 Page 1 of 15

2 Table of Contents TABLE OF CONTENTS 2 VERSION CONTROL 3 1 FILENAME BASED PREFETCH 4 OVERVIEW 4 CONFIGURING FILENAME-BASED PREFETCH 6 ACCESS PATTERN TEMPLATES 6 REVIEW EXISTING ACCESS PATTERN TEMPLATES 7 CONFIGURE A CUSTOM ACCESS PATTERN TEMPLATE 8 Export the Existing Streaming Access Pattern Template to a Temporary Text File. 8 Edit the Temporary File. 8 fnprefetch.fn_enabled 9 fnprefetch.l1_all 9 fnprefetch.size_limit 9 fnprefetch.lookahead 10 prefetch_by_cluster 10 Create a New Access Pattern Template 10 APPLY THE NEW ACCESS PATTERN TEMPLATE 13 SUMMARY 15 Page 2 of 15

3 Version Control Date Version January Notes Initial Release Page 3 of 15

4 Filename Based Prefetch Overview Filename-based Prefetch is a feature of OneFS that detects when a connected client reads in sequence multiple files with sequential file names. When a sequence is detected, data is proactively prefetched (or staged) into L1 cache in advance of it being requested. For more information about L1 cache (and other OneFS caches), review the Isilon OneFS SmartFlash White Paper. This document also covers Access Patterns - which are referred to below. Filename-based Prefetch can yield improved performance for workflows where multiple files are read in sequence by an application - for example, image sequences in Media and Entertainment content creation workflows. The algorithm will detect decimal and hexadecimal incrementing names. Incrementing filenames are detected, but decrementing names are not. Embedded punctuation and filename extensions are both ignored. In Media and Entertainment content creation workflows, an image sequence (such as a DPX sequence) represents each frame of video with a separate file. The files for a sequence typically are stored in the same directory, have a fixed base name with a decimal or hexadecimal incrementing suffix, and a.dpx extension. Different applications use different naming structures and standards within this form, but some illustrative examples are: filebasename_scene-01_00056.dpx filebasename_scene-01_00057.dpx filebasename_scene-01_00058.dpx filebasename_scene-05_0fc09.dpx filebasename_scene-05_0fc0a.dpx filebasename_scene-05_0fc0b.dpx Page 4 of 15

5 For more information about image sequences in Media and Entertainment workflows, review the Uncompressed High Resolution Workflows White Paper available at: Whilst filename-based prefetch can yield improved performance when used with directories containing sequential files, there can be a detrimental performance impact if the feature is inappropriately applied. This is because system resources are consumed by the the prefetches. When filename-based prefetch is enabled, any read from the beginning of a file that is seen to contain an embedded number will cause the system to attempt to prefetch the subsequent files. Additional resource consumption will not occur if the file name coincidentally contains a number but is not accompanied in its directory by other sequentially-named files. However, if sequentially-named files do exist, but the application generally does not read them in order, a great deal of unnecessary prefetching will occur. The recommended best practice for enabling filename-based prefetch is to: Configure applications to read and write sequential files to filesystem directories that are separate, and not used by other parts of the workflow. Apply filename-based prefetch functionality only to those directories. This guide details the steps required to configure, enable, and use filename-based prefetch. The steps are validated with OneFS version 8.1. The steps require connecting to the Isilon cluster via SSH and working at the OneFS command line whilst authenticated as root. The tasks should be undertaken only by suitably qualified and authorised system administrators. Before proceeding with any of the steps, please consult Dell EMC Isilon Customer Support. Page 5 of 15

6 Configuring Filename-based Prefetch Filename-based Prefetch is a configurable property of OneFS Access Patterns. Access Patterns are defined with preset or user-configured templates. To enable Filenamebased Prefetch, a template must be configured and then assigned to the required filesystem directories. Access Pattern Templates Each Access Pattern Template is a collection of sysctl settings that together optimise the performance of OneFS according to particular use cases. There are available a number of Access Pattern Templates: Streaming Random Default custom1 custom2 custom3 custom4 custom5 The Streaming, Random and Default Access Pattern Templates are pre-defined and should not be modified. The Access Pattern Templates: custom1 to custom5 are available to be modified and used as required. Filename-based Prefetch is not enabled by default in any template. To enable Filename-based Prefetch, the recommended best practice is to configure a Custom Access Pattern Template with the appropriate settings. Given that the aim is to optimise streaming performance, the new custom template for Filename-based Prefetch should be based on the existing Streaming Access Pattern Template. The configuration of a Custom Access Pattern Template with the correct settings will make Filename-based Prefetch available to be used. To use the template, it must be applied to the directories where the behaviour is required. Files that are subsequently created in a containing directory will inherit their Access Pattern Template from the containing directory. Files that already exist in the directory must have the Access Pattern Template explicitly applied to them. It must be emphasised that Access Pattern Templates should not be confused with Layout Settings - particularly when instances of the different types may have the same identifying name. An Access Pattern Template affects how data is retrieved from the storage - for example, by enabling Filename-based Prefetch. Page 6 of 15

7 A Layout Setting affects how data is laid out on the underlying storage devices - for example by optimising for different workflows the distribution of data across multiple underlying storage devices. There are three Layout Settings: Streaming Concurrency Random Typically, Access Pattern Templates and Layout Settings are configured in conjunction to optimise performance for any given workflow. Review Existing Access Pattern Templates Before proceeding, it is useful to review the existing Access Pattern Templates of the cluster. This can be achieved at the OneFS command line whilst logged in as the root user. To review the Streaming Access Pattern Template, issue the command: sysctl isi.access.streaming For reference, the Access Pattern Template for Streaming in OneFS version 8.1 has the following settings (though these may change in future releases of OneFS): isi.access.streaming.read_batch: 0 isi.access.streaming.prefetch.read_cluster_limit: 256 isi.access.streaming.prefetch.aread: 0 isi.access.streaming.prefetch.offload: 1 isi.access.streaming.prefetch.adaptive: 0 isi.access.streaming.prefetch.l1_window_blocks: 2048 isi.access.streaming.prefetch.l1_cluster_blocks: 256 isi.access.streaming.prefetch.l2_window_blocks: isi.access.streaming.prefetch.l2_cluster_blocks: 8192 isi.access.streaming.prefetch.l2_ssd_disable: 0 isi.access.streaming.prefetch.l2_by_run: 1 isi.access.streaming.prefetch.l2_unconditional: 1 isi.access.streaming.prefetch.l2_batch: 1 isi.access.streaming.prefetch.prefetch_by_cluster: 0 isi.access.streaming.prefetch.meta_pgs: 8 isi.access.streaming.coalescer.cregion_min_size: isi.access.streaming.coalescer.cregion_max_forced: 16 isi.access.streaming.coalescer.coal_falloc: 1 isi.access.streaming.fnprefetch.fn_enabled: 0 isi.access.streaming.fnprefetch.lookahead: 9 Page 7 of 15

8 isi.access.streaming.fnprefetch.l1_all: 1 isi.access.streaming.fnprefetch.size_limit: isi.access.streaming.write.realloc_blocks: 0 isi.access.streaming.write.writealloc_blocks: 0 isi.access.streaming.write.prealloc: 1 Configure a Custom Access Pattern Template The user-configurable Access Pattern Templates custom1 to custom5 are configured by adding the appropriate sysctl settings to the OneFS sysctl.conf over-ride file. As noted earlier, the recommended best practice is to base the Custom Access Pattern Template on the existing Streaming Access Pattern Template. The recommended procedure for accomplishing this task is: Write the existing Streaming Access Pattern Template to a temporary text file. Change streaming to the desired template name. Modify the relevant sysctl settings. Create the sysctl.conf over-ride file. Add the sysctl settings to the sysctl.conf over-ride file. Export the Existing Streaming Access Pattern Template to a Temporary Text File. Working at the command line of OneFS, and authenticated as the root user, use the command: sysctl -e isi.access.streaming > /tmp/custom_access.txt Edit the Temporary File. Using a text editor or text manipulation tool of personal choice, edit the temporary file: For each sysctl setting, change the template name of streaming to custom1 (or the preferred template name). Change the appropriate sysctl settings for Filename-based Prefetch. For OneFS version 8.1 (and assuming the template name custom1), the lines to be changed are shown below. Note that it is recommended not to copy-and-paste this example without first reviewing the existing Streaming Access Pattern Template of the instance of OneFS being optimised. Page 8 of 15

9 isi.access.custom1.fnprefetch.fn_enabled=1 isi.access.custom1.fnprefetch.l1_all=1 isi.access.custom1.fnprefetch.size_limit= isi.access.custom1.fnprefetch.lookahead=9 isi.access.custom1.prefetch_by_cluster=1 fnprefetch.fn_enabled This control enables Filename-based Prefetch functionality for the Access Pattern Template. Set to =1 to enable. Set to =0 to disable. fnprefetch.l1_all This control affects the target cache for the pre-fetched data. + Set to 1 to set enabled to true. + Set to 0 to set enabled to false. When l1_all is set to true, the entirety of each file, up to the limit specified in fnprefetch.size_limit (detailed below), is prefetched to L1 cache. When the control is set to false, only a portion of the file is prefetched. Specifically, only the portion that would be prefetched as part of standard prefetch. The number of blocks prefetched to L1 and L2 differ and are configurable with the following sysctls. isi.access.custom1.prefetch.l1_window_blocks=2048 isi.access.custom1.prefetch.l1_cluster_blocks=256 isi.access.custom1.prefetch.l2_window_blocks=32768 isi.access.custom1.prefetch.l2_cluster_blocks=8192 The first.prefetch.l1_cluster_blocks +.prefetch.l1_window_blocks are prefetched to L1 The first.prefetch.l2_cluster_blocks +.prefetch.l2_window_blocks are prefetched to L2. The recommended value for l1_all is 1 (enabled). The remainder of this document assumes this setting, and that all prefetched data will be sent to L1 cache. fnprefetch.size_limit This control limits the number of Bytes of each file that are prefetched into L1 cache when l1_all is enabled. The recommended default value for this control is (64 MiB). Page 9 of 15

10 fnprefetch.lookahead This control sets the number of files ahead that the Filename-based Prefetch algorithm will cause OneFS to attempt to read. Where the current file is number N in the sequence, and the lookahead is set to X: When an application starts to read a sequence of files, OneFS will attempt to read immediately into L1 cache all files with numbers N, N+1, N+2, N+3 N+X. Whilst an application is in the steady state of reading files in sequence, on reading file number N, OneFS will attempt to read into L1 cache file number N+X. The recommended value for this control is 9. prefetch_by_cluster Set to =1 to enable. Set to =0 to disable. It should be emphasised that in this context, a cluster is a logically-contiguous sequence of data blocks being prefetched, not an interconnected set of Isilon nodes. This control optimises the interaction between Filename-based Prefetch and the standard prefetch behavior of OneFS. During standard file data prefetch triggered by a read, OneFS determines which (if any) clusters should be prefetched. For each block in the cluster, if it is not already in L1 cache, the operation to prefetch it is triggered. If a required block has already been prefetched, the operation (which would be redundant) is skipped. When Filename-based Prefetch is additionally active as part of the Access Pattern Template (and where the client is reading a supported sequence of files), OneFS is already instructed to prefetch to L1 cache the entire file (see above: fnprefetch.l1_all) - making additional checks redundant. If a check on the first block of data in a cluster of blocks finds it to be in cache already, it is considered safe to assume that all remaining blocks in that cluster will have the same status. When prefetch_by_cluster is enabled, the system treats the status of the first block in the cluster as a proxy for the status of the remaining blocks in the cluster. If the first block is found in cache, it is not necessary to perform in-cache checks on the remaining blocks. As a result, fewer system resources are saved, and performance is improved. The recommended default value for this control is 1 (enabled). Create a New Access Pattern Template Once the work to edit the temporary file is complete, its contents will be similar to the example below. Note that there may be variations dependent on the particular version Page 10 of 15

11 of OneFS and other optimisation work that may previously have been undertaken. For this reason, it is important not to copy-and-paste the example. ### DO NOT COPY AND PASTE THESE CONTROLS! ### ### INSTEAD, FOLLOW THE INSTRUCTIONS TO CREATE ### A SET OF CONTROLS BASED ON THE EXISTING ### STREAMING *Access Pattern Template*. ### isi.access.custom1.read_batch=0 isi.access.custom1.prefetch.read_cluster_limit=256 isi.access.custom1.prefetch.aread=0 isi.access.custom1.prefetch.offload=1 isi.access.custom1.prefetch.adaptive=0 isi.access.custom1.prefetch.l1_window_blocks=2048 isi.access.custom1.prefetch.l1_cluster_blocks=256 isi.access.custom1.prefetch.l2_window_blocks=32768 isi.access.custom1.prefetch.l2_cluster_blocks=8192 isi.access.custom1.prefetch.l2_ssd_disable=0 isi.access.custom1.prefetch.l2_by_run=1 isi.access.custom1.prefetch.l2_unconditional=1 isi.access.custom1.prefetch.l2_batch=1 isi.access.custom1.prefetch.prefetch_by_cluster=1 isi.access.custom1.prefetch.meta_pgs=8 isi.access.custom1.coalescer.cregion_min_size= isi.access.custom1.coalescer.cregion_max_forced=16 isi.access.custom1.coalescer.coal_falloc=1 isi.access.custom1.fnprefetch.fn_enabled=1 isi.access.custom1.fnprefetch.lookahead=9 isi.access.custom1.fnprefetch.l1_all=1 isi.access.custom1.fnprefetch.size_limit= isi.access.custom1.write.realloc_blocks=0 isi.access.custom1.write.writealloc_blocks=0 isi.access.custom1.write.prealloc=1 To enable Filename-based Prefetch, the controls must be added to the sysctl.conf over-ride file located at: /etc/mcp/override/sysctl.conf Page 11 of 15

12 By making changes to the global over-ride sysctl.conf file, consistent functionality across the whole cluster is maintained. Any client connecting to any node will be able to access Filename-based Prefetch functionality. To enable the feature only on specific Isilon nodes, the local copy of the sysctl.conf file on each node may instead be edited, and not the over-ride file. Only clients that connect to the nodes where Filename-based Prefetch is enabled will be able to access Filename-based Prefetch functionality. The local copy of the sysctl.conf file is located on each node at: /etc/local/sysctl.conf This information is included in this document for completeness, but it is strongly recommended that changes are made to the Isilon cluster as a whole, and not to individual nodes. There could be an unexpected impact on performance characteristics if clients are mistakenly configured to connect to different nodes, if the configuration is not homogenous across the whole cluster. System Administrators and users generally expect the characteristics of an Isilon cluster to be consistent across nodes. The behaviour and persistence of changes made only to individual nodes has not been tested through a OneFS upgrade process. Making changes to individual nodes will be supported only in specific circumstances and by agreement. Before proceeding, please consult Dell EMC Isilon Customer Support. Before making any changes to a sysctl.conf file, it can be useful to review the existing settings for the Access Pattern Template to be edited. It will then be possible to make a comparison and verify that changes have been made. To review the existing template, use the command: sysctl isi.access.custom1 Using a text editor or text manipulation tool of personal choice, copy the contents of the temporary file created earlier to the sysctl.conf file. Depending on the status of the cluster and previous actions or configurations, the sysctl.conf file may or may-not Page 12 of 15

13 already exist. If it doesn t exist, it will need to be created. The following command will copy the contents of the file, creating the new file automatically if required: cat /tmp/custom_access.txt >> /etc/mcp/override/sysctl.conf Changes to this file will be mirrored across the cluster and put into effect on the running system automatically. Finally, verify that the new template has been created by reviewing the settings. Use same command again, and compare with the previous result: sysctl isi.access.custom1 On completion, it is recommended that the temporary file is copied and archived to a location of personal choice, and then deleted. Apply the New Access Pattern Template Once the new Access Pattern Template containing the controls to enable Filenamebased Prefetch has been enabled, it must be applied to the filesystem directories and/or files where the feature is required. The purpose of enabling Filename-based Prefetch is to improve streaming performance. Therefore, as well as applying the Access Pattern Template to the required directories, it is strongly advised to also set the Layout Setting for those directories to Streaming. As noted above, a Layout Setting is different from an Access Pattern Template, even though instances of either type can have the same identifying name. To apply the Streaming Layout Setting recursively to a directory and its contained subdirectories, use the command: isi set -R -l streaming /ifs/data/path/to/directory Note that if the directory already contains a substantial amount of data, applying the Streaming Layout Setting could take a significant amount of time. This is because data is being moved around the underlying storage devices and laid out across those devices in a manner that is optimised for streaming access. isi set is a singlethreaded command that will by default run in the foreground of the shell. Page 13 of 15

14 The recommended best practice is to: If possible, apply the Layout Setting, and Access Pattern Template to the required directories whilst they are empty - before any files are written. This will significantly reduce the use of system resources. If that is not possible, care should be taken to consider the impact on the system as a whole of the resources being consumed whilst the layout changes are taking place. It may be appropriate to schedule the work to occur at off-peak times. Before proceeding further, verify that the Layout Setting has been applied correctly to the required directories. At the command line, use the command: isi get /ifs/data/path/to/directory Check each file and directory as appropriate, to verify that /@18 follows the Access Pattern Template name in the PERFORMANCE column of the console output. To apply the new Access Pattern Template recursively to the same directory and its contained sub-directories, use the command: isi set -R -a custom1 /ifs/data/path/to/directory Note that a side-effect of applying the Layout Setting using the isi tool is that the Access Pattern Template will automatically be modified as well. It is important therefore that the Access Pattern Template be applied after the Layout Setting is applied. Now, verify that the Access Pattern Template has been applied correctly to the required directories. At the command line, use the same command: isi get /ifs/data/path/to/directory Check each file and directory as appropriate, to verify that custom1 is listed in the PERFORMANCE column of the console output. As an example, the console output should look something like this: POLICY LEVEL PERFORMANCE COAL FILE default 4+2/2 custom1/@18 on./ default 4+2/2 custom1/@18 on../ Note that it is not possible to verify the status of the Access Pattern Template at the OneFS Administration Web User Interface. Page 14 of 15

15 Summary Filename-based Prefetch is a powerful feature of OneFS that can yield improved performance for workflows where multiple files are read in sequence by an application - for example, image sequences in Media and Entertainment content creation workflows. This guide details the steps required to configure, enable, and use filename-based prefetch. The steps are validated with OneFS version 8.1. Page 15 of 15