Plexistor SDM User Guide

Transcription

1 Plexistor SDM User Guide Latest version: m1fs v1.7.1 February 2016 The latest version of this document is available online at Plexistor SDM Community Edition (CE) is provided for free with limited support. That said, Plexistor respects your time and will be happy to guide and support you, so do not hesitate to contact our support team: Table of contents: Background Plexistor SDM Installation & Setup Installation Script The Basic configuration Uninstall System Information & Analytics Installed packages & version information Configuration Capacity Advanced Insight Performance sanity test Reporting a Bug Known Issues Appendix A: Configuring LVM2 to support multiple FLASH based devices Appendix B: Upgrading Ubuntu Appendix C: Using Plexistor s Automatic Installation Script Appendix D: Re mount All rights reserved to Plexistor 1

2 Background Plexistor s SDM is exposed as the m1fs file system. m1fs uses memory and FLASH as equally persistent tiers, which is perfect for either: 1. Ephemeral computing, in which local storage data is irrelevant following a power failure. This is the case in public and private clouds, share nothing architectures and many Test&Dev environments. 2. Persistent computing, in which traditional storage semantics are expected. This requires the usage of non volatile memory cards, so be sure to follow Plexistor`s hardware compatibility list (HCL) or purchase a pre installed and tuned server via one of the recommended integrators. Plexistor SDM software offers the fastest storage performance known. The latency of a few microseconds at the application level, while serving millions of file level accesses per second. m1fs performance boost is valid for all file and access types: Random and sequential access Small and large files Byte and page alignment and granularity Write and read intensive Sync (O_SYNC) and Direct I/O (O_DIRECT) as well as async usage The current version is provided with no warranty or support guarantees and should not be used in production or redistributed. Plexistor SDM Installation & Setup An SDM CE install script is available for download from our website: It is intended to be executed on a physical Intel XEON CPU based server, running a recent Linux distribution, such as Ubuntu 15.10, RHEL 7.2 or 7.1, CentOS 7.2 or 7.1, and Fedora 22. Installation Script Running the script: bashsdm_install.sh blk<device> <or> bashsdm_install.sh # ifnoflashtieravailable Replace <device> with the path to your FLASH tier device (for example /dev/nvme0n1) An optional flag perf_comp is available for user who wish to compare performance as an apples to apples comparison, e.g. m1fs versus NFS/ZFS/XFS/EXT4/... using the same hardware and identical memory footprints. This is a suboptimal configuration and should not be used in production A copy of the End User License Agreement (EULA) will remain in /usr/share/doc/plexistor/eula.txt All rights reserved to Plexistor 2

3 See appendix C for additional details regarding the installation script. The Basic configuration The install script aims to automate a basic configuration, which is convenient for preliminary performance tests, e.g. for comparing XFS, NFS and m1fs. The basic configuration includes: A single SDM mount point (/m1fs) At least 32GB of memory, and at least half of it assigned to Plexistor SDM first tier. Refer to tables 1 and 2 for examples. An optional 2nd FLASH based tier, with the following characteristics: Single device (you can use LVM to aggregate multiple devices, see appendix A ) For devices larger than 13.5 times the capacity of the 1st tier, SDM will create a partition. Refer to table 1 for examples. Activated call home agent (CHA) for proactive support. The call home agent collects various OS level logs and counters periodically. It does not collect user data stored in the file system. If your server operates behind a firewall, please enable outgoing port 443. Otherwise, Plexistor will not receive the telemetry and will not be able to provide proactive support. If other configurations are required contact support@plexistor.com. Physical memory OS memory Storage 1st tier Storage 2nd tier (Before installing m1fs, identical to Table 2) (left to OS and applications) (memory based) (FLASH based) 1 32 GB 16 GB 16 GB Up to 216 GB GB 32* GB 96 GB Up to 1296 GB GB 32* GB 224 GB Up to 3,024 GB Table 1 Examples of the default memory and Flash usage for Ephemeral computing ( * ) Traditional file systems and storage clients are typically the heaviest OS memory users. Plexistor`s SDM bypasses the OS page cache because it is not faster than the 1st storage tier. Physical memory OS memory (Available for applications) Storage 1st tier Storage 2nd tier 1 32 GB 16 GB 16 GB Up to 216 GB GB 64 GB 64 GB Up to 864 GB All rights reserved to Plexistor 3

4 3 256 GB 128 GB 128 GB Up to 1,728 GB Table 2 The default configuration presented in Table 1 is tuned for production purposes. This table shows the same machines, but installed with the perf_comp flag. The performance comparison flag is convenient for an apples to apples performance comparison of m1fs to NFS/ZFS/XFS/EXT4 and the likes. Uninstall To uninstall, run: sdm_uninstall.sh System Information & Analytics The current version does not provide a graphical dashboard. Please use the following command line tools for system information and analysis: Installed packages & version information The SDM uses your distribution package manager for installation. To see the SDM packages & version information on your system execute: rpm qa grep we'(m1fs sdm plexistor)' <or> dpkg l grep we'(m1fs sdm plexistor)' # forcentos/rhel # forubuntu Configuration Command mount t m1fs lsblk /dev/pmem* Description Display mounted m1fs file systems and their mount options Display persistent memory devices, their size and relation within m1fs file systems. Capacity Command df t m1fs Description Display m1fs file systems capacity information Note that the number of used blocks includes both data and metadata blocks. All rights reserved to Plexistor 4

5 Also note that the filesystem reserves the 1st tier capacity on the 2nd FLASH based tier when mounted with the t2sync flag. This is the default when using the install script. du s /m1fs Display m1fs file system size (counting only data blocks). Substitute /m1fs with your m1fs mount point. Advanced Insight Plexistor SDM provides several counters that give insight about the tiering mechanism and the system calls it serves. The counters are accessed, per mounted file system (represented by its primary mount device) with the m1fs_counters.py script: For example, if the mount directory is /m1fs, run the following to get the current counters values: m1fs_counters.py/m1fs This will print three counters called T1 hits, Tier ups and Tier downs. Tiering and data locality insight can be gained by sampling two points in time, calculating the delta values and then the 1st tier miss rate according to the following formula: Δ tier ups (Δtier ups + Δt1 hits) Application data access insight can be gained by viewing the system call counters (as seen by the file system). This is also true for the basic access size granularity. To do so, run the following command: m1fs_counters.py mi/m1fs This will show for example: All rights reserved to Plexistor 5

6 Additional help is available via m1fs_counters.py help Performance sanity test In addition to the m1fs counter, we provide a performance script to assess the local m1fs installation. Please run the script with an empty mounted m1fs (i.e., immediately after mounting m1fs for the first time after mkfs) and provide us with the results. To run, assuming m1fs is mounted on /m1fs: /opt/m1fs support/m1fsperf.py/m1fs The script may run for several minutes, depending on the file system s size and hardware configuration. All rights reserved to Plexistor 6

7 Reporting a Bug Please don t hesitate to contact us with any question or issue. support@plexistor.com When contacting about a bug, please attach the output file of the following script: /opt/m1fs support/collect_logs.sh The output file location is /tmp/m1fs support <date>.tar.gz All rights reserved to Plexistor 7

8 Known Issues The grayed out issues can be ignored as they are irrelevant for the installation type described in this document: Issue Description Bypass & Fix estimation PXS When using volatile memory as if it was persistent: The definition of the persistent memory block device (pmem), cannot overlap memory ranges reserved for I/O memory. In virtual machines these ranges span 0 1GB and 3.5 4GB. In physical machines the range is 0 4GB Running tail f " over a file residing on m1fs, will print the following warning: unrecognized file system type 0x d UEFI Secure Boot (Windows Secure Boot) not supported Upstream distribution updates might remove m1fs kernel as default boot kernel Kernel log includes the following message "d_pmem namespace0.0: unable to guarantee persistence of writes" Mount point does not exist, the sdm setup service failed (see with systemctl status sdm setup.service ) A possible packaging conflict with RHEL / CentOS GUI installation (Xorg x11 drv vmmouse) Irrelevant for NVDIMM users. Irrelevant for install script users. When reserving memory by yours, reserve memory after the 4GB mark. A false warning that can be ignored. It will be removed in a future Linux version. Irrelevant for install script users. Disable Secure Boot before installing the m1fs kernel After updating the host, please make sure that the m1fs kernel is still the default boot option. A false warning that can be ignored. It will be removed in a future Linux version. On very slow hosts, the mkfs command may be issued before udev completes. Restart the service by running systemctl restart sdm setup.service The install script offers to remove the conflicting package All rights reserved to Plexistor 8

9 Appendix A: Configuring LVM2 to support multiple FLASH based devices 1. Install LVM2 tools: yuminstall ylvm2 2. Create a volume group: vgcreate s1g<vg name><device1>[<device2>...] For example, to create a volume group called m1fs t2 vg1 with two NVMe devices /dev/nvme0n1 and /dev/nvme1n1, run: vgcreate s1gm1fs t2 vg1/dev/nvme0n1/dev/nvme1n1 3. Create a logical volume: lvcreate i<number of devices> I2M n<lv name> L<size><vg name> To continue our example from the setup chapter, create a logical volume called m1fs t2 lv1 that corresponds to the 128GB 1st tier size (128G*13.5=1.728T) by running: lvcreate i2 I2M nm1fs t2 lv1 L1.728Tm1fs t2 vg1 For volume groups that are smaller than 13.5 times the 1st tier size, it is also possible to use the entire volume group by replacing the L <size> argument with l 100%FREE. 4. Now you can use the created logical volume as the second tier device in mkfs.m1fs and mount commands. The device path is /dev/<vg name>/<lv name> In our example, the device path is /dev/m1fs t2 vg1/m1fs t2 lv1. 5. To verify the correct LVM configuration you may run the following command and review its output. lvs a o+devices,stripes,stripe_size,vg_extent_size All rights reserved to Plexistor 9

10 Appendix B: Upgrading Ubuntu If you re running a version of Ubuntu that isn t supported by us, you can upgrade by following these instructions: If running the latest LTS version (14.04), edit the configuration file /etc/update manager/release upgrades and change Prompt=lts to Prompt=normal. Run the command do release upgrade as user root and follow the on screen instruction. It s possible to run the command multiple times to upgrade several versions forward. Appendix C: Using Plexistor s Automatic Installation Script Notes regarding the automatic install script: The script adds the Plexistor package repository to your package manager. The script installs Plexistor s flavored kernel for latest support of SDM devices. On boot a service called sdm setup will configure m1fs to be mounted on /m1fs with predefined recommended settings. It is possible to change the automatic setting by editing the file /etc/sdm.conf. Run systemctl restart sdm setup to apply the changes afterwards. To see the status of the setup process, run systemctl status sdm setup. Appendix D: Re mount When a 2nd FLASH based tier (t2) device is available to the install script, it mounts the m1fs file system using the t2sync flag. This forward compatibility feature means that upon a proper umount system call all data and metadata will be moved to the 2nd tier. When re mounting the m1fs file system, previously mounted under the above circumstances, be sure to run: mount tm1fs oephemeral,t2sync /dev/sdb /m1fs or mount tm1fs ot2sync /dev/sdb /m1fs Replace /dev/ sdb with the path to your 2nd FLASH based tier # WhenusingrealNVDIMMs All rights reserved to Plexistor 10