This guide provides configuration settings and considerations for SANsymphony Hosts running Oracle's Solaris.

Transcription

1 The Host Server Oracle Solaris Configuration Guide October 2017 This guide provides configuration settings and considerations for SANsymphony Hosts running Oracle's Solaris. Basic Solaris administration skills are assumed including how to connect to iscsi and/or Fibre Channel Storage Array target ports as well as the processes of discovering, mounting and formatting a disk device.

2 Table of contents Changes made to this document 3 Oracle Solaris compatibility list 4 The DataCore Server's settings 6 The Solaris Host's settings 8 Operating system settings 8 Configuring MPxIO support 9 Appendix A 11 Preferred Server & Preferred Path settings 11 Appendix B 12 Configuring Disk Pools 12 Appendix C 13 Reclaiming Storage 13 Previous changes 16 Page 2

3 Changes made to this document The most recent version of this document is available from here: All changes since May 2016 Updated List of qualified Oracle Solaris Versions Solaris 11.3 is now qualified with SANsymphony-V 10.x without ALUA. All previous changes Please see page 16 Page 3

4 Oracle Solaris compatibility list SANsymphony 9.0 PSP4 Update 4 (1) Version 10.0 (all versions) Version With ALUA Without ALUA With ALUA Without ALUA 9 or earlier Not Supported Not Supported Not Supported Not Supported 10.0 Not Supported Not Qualified Not Supported Qualified 11.0 Not Supported Qualified Not Supported Not Qualified 11.1 Not Supported Qualified Not Supported Qualified 11.2 Not Supported Not Qualified Not Supported Qualified 11.3 Not Supported Not Supported Not Qualified [2] Qualified Notes: Qualified vs. Not Qualified vs. Not Supported See page 5 for definitions. DataCore Server Front-End Port connections Fibre Channel connections are supported. ISCSI connections are not supported. Oracle Solaris Cluster Oracle Solaris Cluster 2.x and earlier is not supported. Oracle Solaris Cluster 3.x is supported only supported with Oracle Solaris Version Oracle Solaris Cluster 4.x is supported only supported with Oracle Solaris Version 11.x SCSI UNMAP SCSI UNMAP is not supported. Reclaiming storage from DataCore Disk Pools See Appendix C: 'Reclaiming Storage' on page 13 for version-specific 'how to' instructions. 1 SANsymphony-V 8.x and all versions of SANsymphony 9.x before PSP4 Update 4 are now End of Life. Please see: End of Life Notifications 2 A fix by Oracle for Solaris 11.3 was added in Package Update 3 (11.3.3) that allows ALUA on Solaris to work with DataCore Servers. Users who wish to 'Self Qualify' with ALUA can do so, but must be on at least Oracle Solaris to run the self-qualification tests. Page 4

5 Qualified vs. Not Qualified vs. Not Supported Qualified This combination has been tested by DataCore and all the host-specific settings listed in this document applied using non-mirrored, mirrored and Dual Virtual Disks. Not Qualified This combination has not yet been tested by DataCore using Mirrored or Dual Virtual Disks types. DataCore cannot guarantee 'high availability' (failover/failback, continued access etc.) even if the host-specific settings listed in this document are applied. Self-qualification may be possible please see Technical Support FAQ #1506 Mirrored or Dual Virtual Disks types are configured at the users own risk; however, any problems that are encountered while using Oracle Solaris versions that are 'Not Qualified' will still get root-cause analysis. Non-mirrored Virtual Disks are always considered 'Qualified' - even for 'Not Qualified' combinations of Oracle Solaris/SANsymphony. Not Supported This combination has either failed 'high availability' testing by DataCore using Mirrored or Dual Virtual Disks types; or the operating System's own requirements/limitations (e.g. age, specific hardware requirements) make it impractical to test. DataCore will not guarantee 'high availability' (failover/failback, continued access etc.) if the host-specific settings listed in this document are applied. Mirrored or Dual Virtual Disks types are configured at the users own risk. Self-qualification is not possible. Mirrored or Dual Virtual Disks types are configured at the users own risk; however, any problems that are encountered while using Oracle Solaris versions that are 'Not Supported' will get best-effort Technical Support (e.g. to get access to Virtual Disks) but no root-cause analysis will be done. Non-mirrored Virtual Disks are always considered 'Qualified' even for 'Not Supported' combinations of Oracle Solaris/SANsymphony. Oracle Solaris versions that are End of Life For versions that are listed as 'Not Supported', self-qualification is not possible. For versions that are listed as 'Not Qualified', self-qualification may be possible if there is an agreed support contract with Oracle as well. Please contact DataCore Technical Support before attempting any self-qualification. For any problems that are encountered while using Oracle Solaris versions that are EOL with DataCore Software, only best-effort Technical Support will be performed (e.g. to get access to Virtual Disks). Root-cause analysis will not be done. Non-mirrored Virtual Disks are always considered 'Qualified'. Page 5

6 The DataCore Server's settings Operating System Type See the Registering Hosts section from the SANsymphony Help: When registering the Host for the first time, choose the 'Oracle (SUN) Solaris' menu option. Port roles Ports used for serving Virtual Disks to Hosts should only have the Front End (FE) role enabled. Mixing other Port Role types may cause unexpected results as Ports that only have the FE role enabled will be turned off when the DataCore Server software is stopped (even if the physical server remains running). This helps to guarantee that any Hosts do not still try to access FE Ports, for any reason, once the DataCore Software is stopped but where the DataCore Server remains running. Any Port with the Mirror and/or Back End role enabled do not shut off when the DataCore Server software is stopped but still remain active. Multipathing support The Multipathing Support option should be enabled so that Mirrored Virtual Disks or Dual Virtual Disks can be served to Hosts from all available DataCore FE ports. Also see the Multipathing Support section from the SANsymphony Help: Webhelp/Hosts.htm Non-mirrored Virtual Disks and Multipathing Non-mirrored Virtual Disks can still be served to multiple Hosts and/or multiple Host Ports from one or more DataCore Server FE Ports if required; in this case the Host can use its own multipathing software to manage the multiple Host paths to the Single Virtual Disk as if it was a Mirrored or Dual Virtual Disk. Note: Hosts that have non-mirrored Virtual Disks served to them do not need Multipathing Support enabled unless they have other Mirrored or Dual Virtual Disks served as well. Asymmetrical Logical Unit Access (ALUA) support ALUA is not supported for Oracle Solaris Hosts. Serving Virtual Disks to the Hosts for the first time DataCore recommends that before serving Virtual Disks for the first time to a Host, that all DataCore Front-End ports on all DataCore Servers are correctly discovered by the Host first. Then, from within the SANsymphony Console, the Virtual Disk is marked Online, up to date and that the storage sources have a host access status of Read/Write. Page 6

7 The DataCore Server's settings Virtual Disks LUNs and serving to more than one Host or Port DataCore Virtual Disks always have their own unique Network Address Authority (NAA) identifier that a Host can use to manage the same Virtual Disk being served to multiple Ports on the same Host Server and the same Virtual Disk being served to multiple Hosts. See the SCSI Standard Inquiry Data section from the online Help for more information on this: While DataCore cannot guarantee that a disk device's NAA is used by a Host's operating system to identify a disk device served to it over different paths generally we have found that it is. And while there is sometimes a convention that all paths by the same disk device should always using the same LUN 'number' to guarantees consistency for device identification, this may not be technically true. Always refer to the Host Operating System vendor s own documentation for advice on this. DataCore's Software does, however always try to create mappings between the Host's ports and the DataCore Server's Front-end (FE) ports for a Virtual Disk using the same LUN number (1) where it can. The software will first find the next available (lowest) LUN 'number' for the Host- DataCore FE mapping combination being applied and will then try to apply that same LUN number for all other mappings that are being attempted when the Virtual Disk is being served. If any Host-DataCore FE port combination being requested at that moment is already using that same LUN number (e.g. if a Host has other Virtual Disks served to it from previous) then the software will find the next available LUN number and apply that to those specific Host- DataCore FE mappings only. 1 The software will also try to match a LUN 'number' for all DataCore Server Mirror Port mappings of a Virtual Disk too, although the Host does not 'see' these mirror mappings and so this does not technically need to be the same as the Front End port mappings (or indeed as other Mirror Path mappings for the same Virtual Disk). Having Mirror mappings using different LUNs has no functional impact on the Host or DataCore Server at all. Page 7

8 The Solaris Host's settings Operating system settings Also refer to the Solaris SAN Configuration and Multipathing Guide - including the section Third- Party Device Configuration Considerations - for more information: In the Oracle Solaris 11 release, driver customizations are made in the /etc/driver/drv directory rather than in the /kernel directory as in previous releases. This improvement means that your driver customizations are not overwritten when the system is upgraded and the files in the /etc/driver/drv directory are preserved during an upgrade. See the How to Customize a Driver Configuration section from Oracle s Solaris Administration Guide: Disk Settings Edit either the /kernel/drv/sd.conf or ssd.conf file - depending on your Host Server's architecture - and add the following entry: sd-config-list = DataCoreVirtual Disk, retries-busy:60 ; Note: A reboot of the Host will be needed for the setting to take effect. Disk Timeouts Edit /etc/system and either add or modify (if not present) the 'sd' setting to 60 seconds in Hexadecimal notation: set sd:sd_io_time = 0x3c Note: A reboot of the Host may be needed for the setting to take effect. Please refer to Oracle for advice. Page 8

9 The Solaris Host's settings Configuring MPxIO support Oracle Solaris 11.x Add the following scsi-vhci-failover-override entries to the scsi_vhci.conf file: scsi-vhci-failover-override = "DataCoreVirtual Disk", "f_sym"; scsi-vhci-failover-override = "DataCoreSANsymphony", "f_sym"; scsi-vhci-failover-override = "DataCoreSANmelody", "f_sym"; In the Load balancing global configuration section set the following entry: load-balance= none ; Notes: The 'stmsboot e' and 'stmsboot u' commands will be needed to enable multipathing (if not already done) and to update the vfstab respectively; a reboot will be needed if the vfstab is updated. See Oracle s own documentation on the 'stmsboot' command: If, when attempting the Solaris format command, there are two (or more) Disk Devices listed for the same DataCore Virtual Disk, then check for any mistakes in the scsi_vhci.conf file entries, that multipathing has been enabled using the 'stmsboot e' command and/or the Host has been rebooted after the 'stmsboot u' command. As all paths to the Virtual Disk are considered primary there is no easy way to determine the path that is the active path (e.g. using the 'luxadm' command); in this case either use the Host s system messages which may indicate the current path being used or SANsymphony s own Live Performance UI to determine where IO is being sent - see the Live Performance section from the SANsymphony Help: Page 9

10 The Solaris Host's settings Oracle Solaris 10.x Add the following device-type-scsi-options-list entries to the scsi_vhci.conf file: device-type-scsi-options-list = "DataCoreVirtual Disk", "symmetric-option"; "DataCoreSANsymphony", "symmetric-option"; "DataCoreSANmelody", "symmetric-option"; symmetric-option=0x ; In the Load balancing global configuration section set the following entry: load-balance= none ; Notes: The 'stmsboot e' and 'stmsboot u' commands will be needed to enable multipathing (if not already done) and to update the vfstab respectively; a reboot will be needed if the vfstab is updated. See Oracle s own documentation on the 'stmsboot' command: If, when using the Solaris format command, there are two or more Disk Devices indicated for the same Virtual Disk then this can indicate that either the scsi_vhci.conf file entry is incorrect or that the Host has not been rebooted. As all paths to the Virtual Disk are considered primary there is no easy way to determine the path that is the active path (e.g. using the 'luxadm' command); in this case either use the Host s system messages which may indicate the current path being used or SANsymphony s own Live Performance UI to determine where IO is being sent - see the Live Performance section from the SANsymphony Help: Page 10

11 Appendix A Preferred Server & Preferred Path settings See the Preferred Servers and Preferred Paths sections from the SANsymphony Help: Without ALUA enabled If Hosts are registered without ALUA support, the Preferred Server and Preferred Path settings will serve no function. All DataCore Servers and their respective Front End (FE) paths are considered equal. It is up to the Host s own Operating System or Failover Software to determine which DataCore Server is its preferred server. With ALUA enabled ALUA is not supported for Oracle Solaris Hosts. Page 11

12 Appendix B Configuring Disk Pools See Creating Disk Pools and Adding Physical Disks from the SANsymphony Help: The smaller the SAU size, the larger the number of indexes are required, by the Disk Pool driver, to keep track of the equivalent amount of allocated storage compared to a Disk Pool with a larger SAU size; e.g. there are potentially four times as many indexes required in a Disk Pool using a 32MB SAU size compared to one using 128MB the default SAU size. As SAUs are allocated for the very first time, the Disk Pool needs to update these indexes and this may cause a slight delay for IO completion and might be noticeable on the Host. However this will depend on a number of factors such as the speed of the physical disks, the number of Hosts accessing the Disk Pool and their IO READ/WRITE patterns, the number of Virtual Disks in the Disk Pool and their corresponding Storage Profiles. Therefore, DataCore usually recommend using the default SAU size (128MB) as it is a good compromise between physical storage allocation and IO overhead during the initial SAU allocation index update. Should a smaller SAU size be preferred, the configuration should be tested to make sure that a potential increased number of initial SAU allocations does not impact the overall Host performance. Page 12

13 Appendix C Reclaiming Storage Using SCSI UNMAP commands SCSI UNMAP is not supported for Oracle Solaris Hosts. SANsymphony's Automatic Reclamation feature DataCore Servers keep track of any 'all-zero' write I/O requests sent to Storage Allocation Units (SAU) in all Disk Pools. When enough 'all-zero' writes have been detected to have been passed down to an entire SAUs logical address space, that SAU will be immediately assigned as 'free' (as if it had been manually reclaimed) and made available to the entire Disk Pool for future (re)use. No additional 'zeroing' of the Physical Disk or 'scanning' of the Disk Pool is required. Important technical notes on Automatic Reclamation The Disk Pool driver has a small amount of system memory that it uses keep a list of all address spaces in a Disk Pool that are sent 'all-zero' writes; all other (non-zero) write requests are ignored by the Automatic Reclamation feature and not included in the in-memory list. Any all-zero write addresses that are detected to be physically 'adjacent' to each other from a block address point of view the Disk Pool driver will 'merge' these requests together in the list so as to keep the size of it as small as possible. Also as entire 'all-zeroed' SAUs are re-assigned back to the Disk Pool, the record of all its address spaces is removed from the in-memory list making space available for future all-zero writes to other SAUs that are still allocated. However if write I/O pattern of the Hosts mean that the Disk Pool receives all-zero writes to many, non-adjacent block addresses the list will require more space to keep track of them compared to all-adjacent block addresses. In extreme cases, where the in-memory list can no longer hold any more new all-zero writes (because all the allocated system memory for the Automatic Reclamation feature has been used) the Disk Pool driver will discard the oldest records of the all-zero writes to accommodate newer records of all-zero write I/O. Likewise if a DataCore Server is rebooted for any reason, then the in-memory list is completely lost and any knowledge of SAUs that were already partially detected as having been written with all-zeroes will now no longer be remembered. In both of these cases this can mean that, over time, even though technically an SAU may have been completely overwritten with all-zero writes, the Disk Pool driver does not have a record that cover the entire address space of that SAU in its in-memory list and so the SAU will not be Page 13

14 Appendix C Reclaiming storage made available to the Disk Pool but remain allocated to the Virtual Disk until any future all-zero writes happen to re-write the same address spaces that were forgotten about previously by the Disk Pool driver. In these scenarios, a Manual Reclamation will force the Disk Pool to re-read all SAUs and perhaps detect those now missing all-zero address spaces. See the section 'Manual Reclamation' on the next page for more information. Reclaiming storage by sending all-zero writes to a Hosts's own filesystem For Oracle Solaris Hosts, a suggestion would be to create a sparse file of an appropriate size (If there is enough free space available in the file system) and then zero-fill it using the dd command: dd if=/dev/zero of=my_file bs=1024 count= This I/O will then be detected by SANsymphony's Automatic Reclamation function (see previous page for more details). Also see: Performing Reclamation section from the SANsymphony Help: SANsymphony's Manual Reclamation feature Manual reclamation forces the Disk Pool driver to 'read' all SAUs currently assigned to a Virtual Disk looking for SAUs that contain only all-zero IO data. Once detected, that SAU will be immediately assigned as 'free' and made available to the entire Disk Pool for future (re)use. No additional 'zeroing' of the Physical Disk is required. Note that manual reclamation will create additional 'read' I/O on the Storage Array used by the Disk Pool, as this process runs at 'low priority' it should not interfere with normal I/O operations. However, caution is advised, especially when scripting the manual reclamation process. Manual Reclamation may still be required even when Automatic Reclamation has taken place (see the 'Automatic Reclamation' section on the previous page for more information). How much storage will be reclaimed? It is impossible to predict exactly how many Storage Allocation Units (SAUs) will be reclaimed. For reclamation of an SAU to take place, it must contain only all-zero block data over the Page 14

15 Appendix C Reclaiming storage entire SAU else it will remain allocated and this is entirely dependent on how and where the Host has written its data on the DataCore LUN. For example, if the Host has written the data in such a way that every allocated SAU contains a small amount of non-zero block data then no (or very few) SAUs can be reclaimed, even if the total amount of data is much less than the total amount of assigned SAUs. It may be possible to use the Host operating system s own defragmentation tools to move andy data that is spread out over the DataCore LUN so that it ends up as one or more large areas of contiguous non-zero block addresses. This might then leave the the DataCore LUN with SAUs that now only have all-zero data on them and that can then be reclaimed. However care should be taken that the act of defragmenting the data itself does not cause more SAU allocation as the block data is moved around (i.e. re-written to new areas on the DataCore LUN) during the re-organization. Page 15

16 Previous changes 2017 May Updated This document has been reviewed for SANsymphony 10.0 PSP 6 (including all updates). No additional technical information was required November Updated Appendix C - Reclaiming storage Automatic and Manual reclamation These two sections have been re-written with more detailed explanations and technical notes. July Updated This document has been reviewed for SANsymphony 10.0 PSP 5. No updates were required. March Updated Host Settings - Configuring MPxIO support for DataCore Virtual Disks Oracle Solaris 10.x Two more lines have been added for the 'scsi_vhci.conf 'file. Previously the required setting was documented as this: device-type-scsi-options-list = "DataCoreVirtual Disk", "symmetric-option"; symmetric-option=0x ; The setting is now documented as: device-type-scsi-options-list = "DataCoreVirtual Disk", "symmetric-option"; "DataCoreSANsymphony", "symmetric-option"; "DataCoreSANmelody", "symmetric-option"; symmetric-option=0x ; Oracle Solaris 11.x Two more lines have been added for the scsi-vhci-failover-override file. Previously the required setting was documented as this: scsi-vhci-failover-override = "DataCoreVirtual Disk", "f_sym"; The setting is now documented as: scsi-vhci-failover-override = "DataCoreVirtual Disk", "f_sym"; scsi-vhci-failover-override = "DataCoreSANsymphony", "f_sym"; scsi-vhci-failover-override = "DataCoreSANmelody", "f_sym"; Existing users that have never had to upgrade from either SANsymphony 7 or SANmelody 3 can ignore adding these additional lines. Use Solaris's 'format' command and then the 'inquiry' option from the menu to obtain the DataCore Virtual Disk's SCSI Information (including the Vital Product Data (VPD) that will determine if you need to add these additional lines or not). Page 16

17 Previous changes February Added List of qualified Oracle Solaris Versions Added Oracle Solaris version Updated SANsymphony-V 8.x and all versions of SANsymphony 9.x before PSP4 Update 4 are now End of Life. Please see: End of Life Notifications April 2015 Added List of qualified Oracle Solaris Versions - Notes on qualification This section has been updated and new information added regarding the definitions of all qualified, unqualified and not supported labels. A new section on Oracle Solaris versions that are no longer in development has also been added at the end of this section. March 2015 Added Additional settings are now required for all new and existing Solaris Hosts: Host Settings All Versions of Oracle Solaris SCSI Disk Setting A new entry in the sd.conf/ssd.conf file (depending on your architecture) is required: sd-config-list = DataCoreVirtual Disk, retries-busy:60 ; Host Settings Configuring MPxIO support for DataCore Virtual Disks Oracle Solaris 10.x and 11.x An edit in the scsi_vhci.conf file is required in the Load balancing global configuration section: load-balance= none ; February 2015 Updated List of qualified Oracle Solaris Versions Solaris 10.x is now qualified with SANsymphony-V 10.x (also with Oracle Solaris Cluster 3.x) using non-alua settings. Note: iscsi is still not considered qualified at this time; all qualified versions are with Fibre Channel only December 2014 Updated List of qualified Oracle Solaris Versions Footnotes have been added with additional information about supported Oracle Solaris Cluster configurations with the different versions of the base operating system. November 2014 Updated List of qualified Solaris Versions Updated the table include Solaris Not Supported has now been changed to mean explicitly Not Supported for Mirrored or Dual Virtual Disks. Page 17

18 Previous changes Single Virtual Disks are now always considered supported. Host Settings Oracle Solaris 10.x - Configuring MPxIO support for DataCore Virtual Disks The required edit for the scsi_vhci.conf file was not clear enough. It previously said to edit the defaults section. This was not accurate enough and so the description and example has been updated to say to edit the devicetype-scsi-options-list section. Oracle Solaris 11.x - Configuring MPxIO support for DataCore Virtual Disks The required edit for the scsi_vhci.conf file was not clear enough. It previously said to edit the defaults section. This was not accurate enough and so the description and example has been updated to say to edit the scsi-vhcifailover-override section. Added an additional Note to run the stmsboot e command to enable Multipathing (including a reference to Oracle s own SAN Configuration and Multipathing Guide) August 2014 Updated This document combines all of DataCore s Oracle Solaris-related information from older Technical Bulletins into a single document including: Technical Bulletin 3: Oracle/SUN Hosts with StorEdge SAN Foundation Software with Traffic Manager Technical Bulletin 8: Formatting Host s File Systems on Virtual Disks created from Disk Pools Technical Bulletin 11: Disk Timeout Settings on Hosts Technical Bulletin 16: Reclaiming Space in Disk Pools Added Which Distributions are qualified? Added new tables to show which versions are explicitly qualified, unqualified and not supported with either SANsymphony-V 8.x or 9.x, and if the configuration is with or without ALUA enabled Hosts. Note that the minimum requirement for SANsymphony-V 8.x is now 8.1 PSP1 Update 4 to guarantee expected behavior for qualified versions of Oracle Solaris Oracle Solaris 11 has now been added. Appendix A This section gives more detail on the Preferred Server and Preferred Path settings with regard to how it may affect a Host. Appendix B This section incorporates information regarding Reclaiming Space in Disk Pools (from Technical Bulletin 16) that is specific to Oracle Solaris Hosts. Host Settings - Improved explanations to most of the required Host Settings and DataCore Server Settings generally. July and earlier Technical Bulletin 3: Oracle/SUN Hosts with StorEdge SAN Foundation Software with Traffic Manager Page 18

19 Previous changes Removed all references to SANmelody as this product is now end of life. Updated How to determine the Active IO Path in the troubleshooting section. July 2012 Update steps to allow for the product ID to be used July 2012 Added SANsymphony-V 9.x. No additional Appserver/Host changes required. April 2012 Updated DataCore Server and Host minimum requirements. Removed all references to End of Life SANsymphony and SANmelody versions that are no longer supported as of December Updated copyright. Updated DataCore Server and Host minimum requirements. Updated DataCore Server and Host configuration settings. January 2011 Added note for load-balance=none. June 2010 Not supported for iscsi Failover. Added Oracle Solaris 10. July 2007 Initial publication of Technical Bulletin. Tested on Solaris 9 and STMS version Page 19

20 COPYRIGHT Copyright 2017 by DataCore Software Corporation. All rights reserved. DataCore, the DataCore logo and SANsymphony are trademarks of DataCore Software Corporation. Other DataCore product or service names or logos referenced herein are trademarks of DataCore Software Corporation. All other products, services and company names mentioned herein may be trademarks of their respective owners. ALTHOUGH THE MATERIAL PRESENTED IN THIS DOCUMENT IS BELIEVED TO BE ACCURATE, IT IS PROVIDED AS IS AND USERS MUST TAKE ALL RESPONSIBILITY FOR THE USE OR APPLICATION OF THE PRODUCTS DESCRIBED AND THE INFORMATION CONTAINED IN THIS DOCUMENT. NEITHER DATACORE NOR ITS SUPPLIERS MAKE ANY EXPRESS OR IMPLIED REPRESENTATION, WARRANTY OR ENDORSEMENT REGARDING, AND SHALL HAVE NO LIABILITY FOR, THE USE OR APPLICATION OF ANY DATACORE OR THIRD PARTY PRODUCTS OR THE OTHER INFORMATION REFERRED TO IN THIS DOCUMENT. ALL SUCH WARRANTIES (INCLUDING ANY IMPLIED WARRANTIES OF MERCHANTABILITY, NON- INFRINGEMENT, FITNESS FOR A PARTICULAR PURPOSE AND AGAINST HIDDEN DEFECTS) AND LIABILITY ARE HEREBY DISCLAIMED TO THE FULLEST EXTENT PERMITTED BY LAW. No part of this document may be copied, reproduced, translated or reduced to any electronic medium or machine-readable form without the prior written consent of DataCore Software Corporation