StarWind Virtual SAN 2-Node Stretched Cluster on VMware vsphere 6.5

Transcription

1 One Stop Virtualization Shop 2-Node Stretched Cluster on VMware vsphere 6.5 JULY 2018 TECHNICAL PAPER

2 Trademarks StarWind, StarWind Software and the StarWind and the StarWind Software logos are registered trademarks of StarWind Software. StarWind LSFS is a trademark of StarWind Software which may be registered in some jurisdictions. All other trademarks are owned by their respective owners. Changes The material in this document is for information only and is subject to change without notice. While reasonable efforts have been made in the preparation of this document to assure its accuracy, StarWind Software assumes no liability resulting from errors or omissions in this document, or from the use of the information contained herein. StarWind Software reserves the right to make changes in the product design without reservation and without notification to its users. Technical Support and Services If you have questions about installing or using this software, check this and other documents first - you will find answers to most of your questions on the Technical Papers webpage or in StarWind Forum. If you need further assistance, please contact us. About StarWind StarWind is a pioneer in virtualization and a company that participated in the development of this technology from its earliest days. Now the company is among the leading vendors of software and hardware hyperconverged solutions. The company s core product is the years-proven StarWind Virtual SAN, which allows SMB and ROBO to benefit from cost-efficient HyperConverged IT infrastructure. Having earned a reputation of reliability, StarWind created a hardware product line and is actively tapping into HyperConverged and storage appliances market. In 2016, Gartner named StarWind Cool Vendor for Compute Platforms following the success and popularity of StarWind HyperConverged Appliance. StarWind partners with world-known companies: Microsoft, VMware, Veeam, Intel, Dell, Mellanox, Citrix, Western Digital, etc. Copyright StarWind Software Inc. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written consent of StarWind Software. TECHNICAL PAPER 2

3 Contents Introduction... 4 Prerequisites... 5 StarWind Failover Strategies... 6 Heartbeat Failover Strategy... 6 Node Majority Failover Strategy... 7 Preconfiguring the Servers Enabling Multipath Support Downloading, Installing, and Registering the Software Configuring Shared Storage Heartbeat Failover Strategy Node Majority Failover Strategy Adding Witness Node Discovering Target Portals Connecting Targets Configuring Multipath Connecting Disks to Servers Adding Storage to the Cluster Post-Configuration Tasks Conclusion Contacts TECHNICAL PAPER 3

4 Introduction Stretched clustering technology allows many organizations to secure their constant business continuity by replicating data and distributing workloads not only among different nodes, but also among different remote locations. Mission-critical applications are kept 24/7/365 up and running due to storage mirroring between distant server locations. This technical paper provides detailed instructions on how to set up a 2-node stretched cluster on VMware vsphere 6.5 with StarWind Virtual SAN as a storage provider. is a hardware-agnostic VM storage solution. By mirroring the existing servers storage and RAM between the participating cluster nodes, it allows creating a faulttolerant high-performing storage purpose-built for intensive virtualization workloads. All I/O is processed by local RAM, SSD cache, and disks. Thus, it never gets bottlenecked by the storage fabric. The mirrored storage is utilized by all cluster nodes and is treated by all hypervisors and clustered applications just as one big local storage. StarWind Virtual SAN features stretched clustering to ensure constant applications uptime due to Live Migration between different geographical locations. A full set of up-to-date technical documentation is available here, or by pressing the Help button in the StarWind Management Console. For any technical inquiries, please, visit our online community, Frequently Asked Questions page, or use the support form to contact our technical support department. TECHNICAL PAPER 4

5 Pre-Configuring the Servers The diagram below illustrates the connection scheme of the StarWind stretched cluster configuration described in this guide. Make sure that the prerequisites for deploying StarWind stretched cluster on VMware are met: L2/L3 multisite network applied according to the appropriate StarWind failover strategy. Each iscsi and Synchronization network channel throughput should be at least 1Gbps. The 10Gbps or higher link bandwidth is highly recommended. The maximum supported latency for StarWind synchronous storage replication should be 10ms round-trip time (RTT). The maximum supported latency between the ESXi ethernet networks should be 10ms round-trip time (RTT). vsphere 6.5 or newer installed on the servers to be clustered. StarWind Virtual SAN installed on Windows Server 2016 VMs. TECHNICAL PAPER 5

6 StarWind Failover Strategies Select the appropriate StarWind failover strategy before creating the device. Once selected, the failover strategy applies for the device time being. StarWind Virtual SAN provides two options: Heartbeat failover strategy and Node Majority failover strategy. Heartbeat Failover Strategy Heartbeat is a technology that allows avoiding the so-called split-brain scenario when the HA cluster nodes are unable to synchronize but continue to accept write commands from the initiators independently. It can occur when all synchronization and heartbeat channels disconnect simultaneously, and the other partner nodes do not respond to the node s requests. As a result, StarWind service assumes the partner nodes to be offline and continues operations in a single-node mode using the data written to it. If at least one heartbeat link is online, StarWind services can communicate with each other via this link. The services mark the device with the lowest priority as not-synchronized one. Subsequently it gets blocked for further read and write operations until the synchronization channel resumption. Then, the partner device on the synchronized node flushes data from the cache to the disk to preserve data integrity in case the node goes down unexpectedly. It is recommended to assign more independent heartbeat channels during replica creation to improve system stability and avoid the split-brain issue. With the Heartbeat Failover Strategy, the storage cluster will continue working with only one StarWind node available. Heartbeat Failover Strategy Network Design Management / Heartbeat 100Mbps network or higher. iscsi / Heartbeat 1Gbps network or higher. The 10Gbps or higher bandwidth link is highly recommended. Synchronization 1Gbps network or higher. The 10Gbps or higher bandwidth link is highly recommended. TECHNICAL PAPER 6

7 Node Majority Failover Strategy This strategy ensures synchronization connection without any additional heartbeat links. The failure-handling process occurs when the node has detected the absence of connection with the partner. The main requirement for keeping the node operational is an active connection with more than a half of the HA device s nodes. Calculation of the available partners bases on their "votes". In case of a two-node HA storage, all nodes disconnect if there is a problem with the node itself, or with communication within the cluster. Therefore, the Node Majority failover strategy does not work in case if only two synchronous nodes are available. To apply this strategy, the third entity is required. It can be a Witness node which participates in the nodes count for the majority, but neither contains data nor processes clients requests. Node Majority failover strategy allows tolerating failure of only one node. If two nodes fail, the third one will also become unavailable to clients requests. If replicated between 2 nodes, the Witness node requires additional configuration for an HA device that uses Node Majority failover strategy. Replication of an HA device among 3 nodes requires no Witness nodes. Node Majority Failover Strategy Network Design Management / Heartbeat /Synchronization 1Gbps network or higher. The 10Gbps or higher bandwidth link is highly recommended. TECHNICAL PAPER 7

8 Preparing Hypervisor for StarWind Deployment Configuring Networks Configure network interfaces on each node to make sure that vmotion and StarWind Synchronization interfaces are in different subnets and connected according to one of failover strategies scenarios (see the network diagrams above). In this document, Management/iSCSI traffic uses x subnet while the Synchronization traffic uses x subnet. 1. Create two vswitches, one for the iscsi/ StarWind Heartbeat channel and the other one for the Synchronization channel. NOTE: iscsi/ StarWind Heartbeat and the Synchronization vswitches require creation of Virtual Machine Port Group while iscsi traffic requires creation of VMKernel port only. VMKernel ports require assignment of static IP address to them. TECHNICAL PAPER 8

9 2. Create a VMKernel port for the iscsi/starwind Heartbeat channel. 3. Add Virtual Machine Port Group on the vswitch for iscsi traffic and on the vswitch for Synchronization traffic. NOTE: It is recommended to set jumbo frames to 9000 on vswitches and VMKernel ports for iscsi and Synchronization traffic. Additionally, users can enable vmotion on VMKernel ports. 4. Repeat steps 1 to 3 for any other links intended for Synchronization and iscsi/heartbeat traffic on both ESXi nodes. TECHNICAL PAPER 9

10 Preparing StarWind Virtual Machines 5. Create a Virtual Machine (VM) on each ESXi host with Windows Server 2016 (2012 R2) and StarWind VSAN installed. Configure the following settings for StarWind VMs on ESXi hosts: RAM: at least 4 GB (plus the size of the RAM cache if applicable) CPUs: at least 4 virtual processors with 2 GHz reserved; Hard disk 1: 100 GB for OS (recommended); Hard disk 2: Depends on the intended shared storage volume. NOTE: VMs require Thick Provisioned Eager Zeroed for each hard disk. Networking: Network adapter 1: Management Network adapter 2: iscsi Network adapter 3: Sync NOTE: Select VMXNET3 type for all network adapters. NOTE: If necessary, to make StarWind Virtual Machine (VM) serve as a domain controller add Active Directory Domain Services role on it. NOTE: When using StarWind Virtual Machine with synchronous replication, it is recommended not to make backups and snapshots of the Virtual Machine with StarWind Service. This can pause the StarWind Virtual Machine. Pausing the Virtual Machines while StarWind service under load may lead to split-brain issues in the devices with synchronous replication and data corruption. TECHNICAL PAPER 10

11 Configuring StarWind VMs Startup/Shutdown Setup the VMs startup policy on both ESXi hosts from the Manage -> System tab in ESXi web console. In the appeared window, check Yes to enable the option and select Stop action as Shut down. Click Save to proceed. To configure VM autostart, right-click on it, navigate to Autostart and click Enable. Complete the actions above on StarWind VM located on another host. Start both virtual machines, install OS and StarWind Virtual SAN. TECHNICAL PAPER 11

12 Downloading, Installing, and Registering the Software 6. Download the StarWind setup executable file from the official StarWind website: NOTE: The setup file is the same for x86 and x64 systems, as well as for all Virtual SAN deployment scenarios. 7. Launch the downloaded setup file on the server to install StarWind Virtual SAN or one of its components. The Setup wizard will appear. Read and accept the License Agreement. Click Next to continue. TECHNICAL PAPER 12

13 8. Read the important information about new features and improvements. Text in red indicates warnings for users that are updating their existing software installations. Click Next to continue. 9. Select Browse to modify the installation path if necessary. Click Next to continue. TECHNICAL PAPER 13

14 10. Select the following components for the minimum setup: StarWind Virtual SAN Service. StarWind service is the core of the software. It allows creating iscsi targets as well as share virtual and physical devices. StarWind Management Console allows managing the service from any Windows computer or VSA in the same network. Alternatively, users can manage StarWind Web Console deployed separately. StarWind Management Console. StarWind Management Console is the Graphic User Interface (GUI) part of the software that controls and monitors all storage-related operations (e.g., allows users to create targets and devices on StarWind Virtual SAN servers connected to the network). Click Next to continue. TECHNICAL PAPER 14

15 11. Specify Start Menu Folder. Click Next to continue. 12. Enable the checkbox if a desktop icon is needed. Click Next to continue. TECHNICAL PAPER 15

16 13. When the license key prompt appears, select the appropriate option: Request time-limited fully functional evaluation key Request FREE version key. Thank you, I do have a key already. Click Next to continue. 14. Click Browse to locate the license file. Press Next to continue. TECHNICAL PAPER 16

17 15. Read the licensing information. Click Next to continue. 16. Verify the installation settings. Click Back to make any changes. Press Install to proceed with the installation. TECHNICAL PAPER 17

18 17. Enable the appropriate checkbox to launch StarWind Management Console right after the setup wizard closes. Click Finish to close the wizard. 18. Repeat the installation steps on the partner node. NOTE: Managing StarWind Virtual SAN on a Windows Server Core edition without GUI requires installation of StarWind Management Console on a different computer running a GUIenabled Windows edition. Configuring Automatic Storage Rescan Configure automatic storage rescan for each ESXi host. 19. Log in to StarWind VM and install vsphere PowerCLI on each StarWind virtual machine by adding the PowerShell module (Internet connectivity is required). To do so, run the following command in PowerShell: Install-Module -Name VMware.PowerCLI -AllowClobber NOTE: For Windows Server 2012 R2, online installation of PowerCLI requires Windows Management Framework 5.1 or upper version available on VMs. Download Windows Management Framework 5.1 by the following link: TECHNICAL PAPER 18

19 20. Open PowerShell and change the Execution Policy to Unrestricted by running the following command: Set-ExecutionPolicy Unrestricted 21. Create the PowerShell script which will perform an HBA rescan on the hypervisor host. Import-Module VMware.PowerCLI $counter = 0 if ($counter -eq 0){ Set-PowerCLIConfiguration -InvalidCertificateAction ignore -Confirm:$false Out-Null } $ESXiHost = "IP address" $ESXiUser = "Login" $ESXiPassword = "Password" Connect-VIServer $ESXiHost -User $ESXiUser -Password $ESXiPassword Out-Null Get-VMHostStorage $ESXiHost -RescanAllHba Out-Null Get-ScsiLun -VMHost $ESXiHost -LunType disk Where-Object Vendor -EQ "STARWIND" Where-Object ConsoleDeviceName -NE " " Set-ScsiLun -MultipathPolicy RoundRobin Out-Null $StarwindCN = Get-ScsiLun -VMHost $ESXiHost -LunType disk Where-Object Vendor -EQ "STARWIND" Where-Object ConsoleDeviceName -NE " " TECHNICAL PAPER 19

20 Select-Object CanonicalName $esxcli = Get-EsxCli -VMHost $ESXiHost foreach($cn in $StarwindCN){ $esxcli.storage.nmp.psp.roundrobin.deviceconfig.set(0,$null,$cn.canonicalname,1,` "iops",0) Out-Null } Disconnect-VIServer $ESXiHost -Confirm:$false $file = Get-Content "$PSScriptRoot\rescan_script.ps1" if ($file[1] -ne "`$counter = 1") { $file[1] = "`$counter = 1" $file > "$PSScriptRoot\rescan_script.ps1" } In the appropriate lines, specify the IP address and login credentials of the ESXi host on which the current StarWind VM is stored and running: $ESXiHost1 = IP address $ESXiUser = Login $ESXiPassword = Password Save the script as rescan_script.ps1 to the VM s C:\ drive root. 22. Perform the configuration steps above on the partner node. 23. Go to Control Panel -> Administrative Tools -> Task Scheduler -> Create Basic Task and follow the wizard steps: TECHNICAL PAPER 20

21 24. Specify the task name, select When a specific event is logged, and click Next. 25. Select Application in the Log drop-down list, type StarWindService as the event source and 788 as the event ID. Click the Next button. TECHNICAL PAPER 21

22 26. Select Start a Program as the task action and click Next. 27. Type powershell.exe in the Program/script field. In the Add arguments field, type: -ExecutionPolicy Bypass -NoLogo -NonInteractive -NoProfile -WindowStyle Hidden -File C:\rescan_script.ps1 Click the Next button to continue. TECHNICAL PAPER 22

23 28. Click Finish to exit the Wizard. 29. Configure the task to run with highest privileges by enabling the checkbox at the bottom of the window. Also, make sure that the Run whether user is logged on or not option is enabled. TECHNICAL PAPER 23

24 30. Switch to the Triggers tab. Verify that the trigger on event 788 is set up correctly. 31. Click New and add other triggers by Event ID 782, 257, 773, and 817. TECHNICAL PAPER 24

25 32. The list of triggers should look like in the picture below: 33. Switch to the Actions tab and verify the parameters for the task. Press OK and type the role to execute the command in the user credentials. 34. Perform the same steps on the second StarWind VM, specifying the corresponding settings. TECHNICAL PAPER 25

26 Provisioning Shared Storage with StarWind VSAN 35. Open StarWind Management Console and click the Add Device (advanced) button. 36. Right-click on the Servers field and click on the Add Server button. Add new StarWind Server to use as the second StarWind VSAN node. 37. Select the StarWind Server to create the device on and press the Add Device (advanced) button on the toolbar. 38. Add Device Wizard will appear. Select Hard Disk Device and click Next. TECHNICAL PAPER 26

27 39. Select Virtual Disk and click Next. 40. Specify the Virtual Disk Location, Name, and Size. Click Next. TECHNICAL PAPER 27

28 41. Specify the Virtual Disk options and click Next. NOTE: Sector size should be 512 bytes in case of ESXi. TECHNICAL PAPER 28

29 42. Define the caching policy and specify the cache size (in GB). Click Next to continue. NOTE: It s recommended to assign 1 GB of L1 cache in Write-Back or Write-Through mode per 1 TB of storage capacity. Yet, the cache size should correspond with the storage working set of the servers. TECHNICAL PAPER 29

30 43. Define Flash Cache Parameters and Size if necessary. Specify SSD location in the Wizard. Press Next. NOTE: The recommended size of the L2 cache is 10% of the initial StarWind device capacity. TECHNICAL PAPER 30

31 44. Specify the Target Parameters. Enable the Target Name checkbox to customize the target name. Otherwise, the program generates the name automatically based on the Target Alias. Click Next. 45. Click Create to add new device and attach it to the target. 46. Click Finish to close the Wizard. 47. Right-click on the recently created device and select Replication Manager from the shortcut menu. TECHNICAL PAPER 31

32 48. Then, click Add Replica. 49. Select Synchronous Two-Way Replication. Click Next to proceed. TECHNICAL PAPER 32

33 50. Specify the partner server IP address. The default StarWind management port is For a different port, type it in the Port Number field. Click Next to continue. TECHNICAL PAPER 33

34 Heartbeat Failover Strategy 51. Check Heartbeat Failover Strategy according to the network design. Click Next to continue. NOTE: For setting the Node Majority failover strategy, jump to Step Select Create new Partner Device. Click Next. TECHNICAL PAPER 34

35 53. Specify the partner device location if necessary and/or modify the device Target Name. Click Next. 54. Select the Synchronization and Heartbeat networks for the HA device by clicking Change Network Settings. TECHNICAL PAPER 35

36 55. Specify the interfaces for the Synchronization and Heartbeat. Press OK. Then, click Next. NOTE: It is recommended to configure the Heartbeat and iscsi channels on the same interfaces to avoid the split-brain issue. If the Synchronization and Heartbeat interfaces locate on the same network adapter, it is recommended to assign another Heartbeat interface to a separate adapter. TECHNICAL PAPER 36

37 56. Select Synchronize from existing Device for the partner device initialization mode. Click Next. 57. Press the Create Replica button. Then, click Close. TECHNICAL PAPER 37

38 58. The added device will appear in StarWind Management Console. Repeat the HA device creation steps for any virtual disks to be used as Cluster Shared Volumes. Once created, the devices appear in the left pane of the Management Console as shown in the screenshot below. TECHNICAL PAPER 38

39 Node Majority Failover Strategy 59. Check Node Majority Failover Strategy according to the network design. Click Next to continue. 60. Select Create new Partner Device. Click Next. TECHNICAL PAPER 39

40 61. Specify the partner device location if necessary and/or modify the target name of the device. Click Next. 62. Select the Synchronization and Heartbeat networks for the HA device by clicking Change Network Settings. TECHNICAL PAPER 40

41 63. Specify the interfaces for Synchronization and Heartbeat. Press OK. Then, click Next. 64. Select Synchronize from existing Device for the partner device initialization mode. Click Next. TECHNICAL PAPER 41

42 65. Press the Create Replica button. Then click Close. 66. The added device will appear in StarWind Management Console. Repeat HA device creation steps for any virtual disks that will be further used as a Cluster Shared Volumes. Once created, the devices appear in the left pane of the Management Console as shown in the screenshot below. TECHNICAL PAPER 42

43 Adding Witness Node This section describes adding a Witness node. Witness nodes count for the majority, but neither contain data nor process any clients requests. Configure the Witness node at a separate location. There are two options to do so: it can either be a virtual machine run in the cloud or a host at another site. Witness node should have StarWind Virtual SAN service installed on it. 67. Open the StarWind Management Console, right-click on the Servers field and press the Add Server button. Add new StarWind Server to be used as the Witness node and click OK. 68. Right-click the HA device with the configured Node Majority failover policy and select Replication Manager. The Replication Manager window will appear. Press the Add Replica button. 69. Select Witness Node and click Next TECHNICAL PAPER 43

44 Specify the Witness node name or its IP address. 70. Specify the Witness device location and its target name if necessary. Click Next. TECHNICAL PAPER 44

45 71. For the HA device, select the synchronization channel with the Witness node by clicking on the Change network settings button. 72. Specify Interfaces for Synchronization Channels, confirm, and click Next. TECHNICAL PAPER 45

46 73. Click Create Replica. After the device creation is completed, close the Wizard by pressing the Close button. TECHNICAL PAPER 46

47 74. Repeat the steps above to create other virtual disks. 75. The added device will appear in StarWind Management Console. The list of HA devices should look as follows: TECHNICAL PAPER 47

48 Preparing Datastores Adding Discover Portals 76. To connect the previously created devices to the ESXi host, click the Storage -> Adapters -> Configure iscsi and select the Enabled option to enable Software iscsi storage adapter. 77. In the Configure iscsi window, under Dynamic Targets, click on the Add dynamic target button to specify iscsi interfaces. TECHNICAL PAPER 48

49 78. Enter the iscsi IP address of the first StarWind node from the virtual local network (e.g x). 79. Add the IP address of the second StarWind node (e.g x). Confirm your actions by pressing Save configuration. 80. The list of targets should look like in the image below: TECHNICAL PAPER 49

50 81. Click on the Rescan button to rescan the storage. Now, the previously created StarWind devices are visible. 82. Repeat all the steps from this section on the other ESXi node, specifying corresponding IP addresses for the iscsi subnet. TECHNICAL PAPER 50

51 Creating Datastores 83. Open the Storage tab on one of your hosts and click New Datastore. 84. Specify the Datastore name, select the previously discovered StarWind device, and click Next. TECHNICAL PAPER 51

52 85. Enter datastore size. Click Next. 86. Verify the settings. Click Finish. TECHNICAL PAPER 52

53 87. Add another Datastore (DS2) in the same way but select the second device for the second datastore. 88. Verify that your storages (DS1, DS2) are connected to both hosts. Otherwise, rescan the storage adapter. 89. The Rescan Script already contains the Path Selection Policy for Datastores from Most Recently Used (VMware) to Round Robin (VMware), and the program performs this action automatically. For checking and changing this parameter manually, connect the hosts to vcenter. TECHNICAL PAPER 53

54 90. vcenter only allows checking the multipathing configuration. To check it, click the Configure tab, open Storage Devices, select the proper storage device, and click the Edit Multipathing button. Additional Tweaks 91. From the ESXi host, click on the Manage tab on one of the hosts and proceed to Services. Select TSW SSH and right-click on Start, then select Start and Stop with host. TECHNICAL PAPER 54

55 92. Connect to the host using the SSH client (e.g. Putty). 93. Check the device list using the following command: esxcli storage nmp device list 94. For devices, adjust Round Robin size from 1000 to 1 using the following command: esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 device NOTE: Paste UID of the StarWind device in the end of the command. Also, Rescan Script already contains this parameter, and the program performs this action automatically. 95. Repeat the steps above on each host for each datastore. TECHNICAL PAPER 55

56 96. Click the Manage tab on one of the hosts and open Advanced Settings. 97. Select Disk and change the Disk.DiskMaxIOSize parameter to 512. TECHNICAL PAPER 56

57 Creating Datacenter NOTE: Creating Datacenter requires deployment of the vcenter server prior to creation. 98. Connect to vcenter, select the Getting Started tab, click Create Datacenter. 99. Enter the Datacenter name and click OK. TECHNICAL PAPER 57

58 Creating Cluster 100. Click the Datacenter s Getting Started tab and click Create a cluster. Enter the cluster name and click Next. Adding Hosts to Cluster 101. Open the Cluster tab and click Add a host. TECHNICAL PAPER 58

59 102. Enter the ESXi host name or IP address and enter the administrative account Lockdown mode is disabled by default. TECHNICAL PAPER 59

60 104. Assign the License from the appropriate tab Click Cluster -> Configure -> Edit and turn on vsphere HA. TECHNICAL PAPER 60

61 Conclusion This technical paper has explained how to set up a stretched cluster on VMware vsphere 6.5 with StarWind VSAN as a storage provider. A simple 2-nodes on ESXi scenario was taken as a base for this technical paper. Using StarWind VSAN, the distributed storage of both ESXi hosts is converted into a fault-tolerant shared storage synchronously ''mirrored'' between the nodes, whatever the location. The technical paper has also covered the so-called split-brain problem and ways to avoid it by properly organizing failover strategies. TECHNICAL PAPER 61

62 Contacts US Headquarters EMEA and APAC (UK) (Spain and Portugal) Customer Support Portal: Support Forum: Sales: General Information: StarWind Software, Inc. 35 Village Rd., Suite 100, Middleton, MA USA , StarWind Software Inc. All rights reserved. TECHNICAL PAPER 62