Interpolation calculation using cloud computing

Transcription

1 Interpolation calculation using cloud computing MPI in Amazon Web Services Eduard Vesel Faculty of Natural Sciences Matej Bel University Banská Bystrica, Slovakia At present time, when we are increasing our demands on computer resources and time complexity, cloud computing has become more popular. Using cloud computing is from business solutions services to the needs of individual and research projects. So in this work we decided to demonstrate the interpolation calculations using cloud computing, specifically in the service from Amazon, Amazon Web Services (Elastic Compute Cloud). Keywords: cloud computing, parallel programs, Amazon Web Services Interpolačné Metódy I. INTERPOLATION METHODS Our program is calculating the unknown values of the phenomenon by interpolation because the interpolation methods are objective to geoinformation systems. The result of the used interpolation methods is calculated value the phenomenon at the place where data were not available or were not measured at all. In practice, it is impossible to measure value with measuring stations in the selected area at any given point, so we need to get this value with a suitably selected algorithm. This algorithm calculate unknown values by using weighted average, method Inverse distance weighting (IDW). IDW is a method for multidimensional interpolation. The process of assigning values to unknown places is usually by using distributed range of known places. The value over an unknown place is the weighted sum of N known places. [1] II. AMAZON ELASTIC COMPUTE CLOUD First, we have to choose the right cloud service which best suits for our program. The requirements include availability, reliability, configurability, offered hardware solutions and price. Of the total services offered on the market, we chose a solution from Amazon, Amazon Web Services and service Amazon Elastic Compute Cloud (EC2). EC2 is a web service interface that provides computing and its change in size of the cloud. It is designed to configure compute capacity for developers as simple as possible. This gives us complete control over our computing resources and lets us run programs on the best computing environments in Amazon. Amazon EC2 reduces the time needed to acquire and run a new instance of the server in seconds, allowing us to quickly adapt capacity to maximum or minimum performance. Amazon EC2 also introduced an interesting new pricing policy. Users for computing capacity pay no monthly or activation fees. Price is for real time use, when an instance is running, therefore we use it (pay-as-you-go). This allows us to parallel computing meets because we do not have to pay a fixed monthly fee, even if we do not use an instance. [2] III. PREPARING INSTANCES Before we start use the program for parallel computing, we have to choose the right operating system for instances and configure systems resources for our grid. Instances will be located in the area of West EU (Ireland). A. Operating System There is a solution for each operating system which makes it possible to use the MPI program. We have chosen Linux distribution, specifically Ubuntu. Ubuntu was chosen mainly for our previous experience of clusters solutions. In the EC2 are available many different versions of Ubuntu, we use ami-0e0f3d7a image (ubuntuhardy amd64-server ). B. Creation And Configuration Instances After selecting the operating system we choose offered configurations for instance from the menu. Figure 1. Offered configuration

2 In this work we describe how to configure Micro configuration (Intel Xeon Processor E5507 (4M Cache, 2.26 GHz, 4.80 GT / s Intel QPI), 613 MB RAM, 8 GB HDD), but the procedure is the same for all other configurations. The final evaluation of measured results is also mentioned the Large configuration. The number of instances to be created is 4, because the correct running need one master and three worker nodes. Of course, the price per use depend on the selected configuration The Micro is the lowest configuration and price is $ for each hour when the instance is running. The next step is to create our own private key which is used to login into instances. This key must be downloaded and securely backup, because without key you cannot login. In the last step we have to create instance s security group. It involves us opening all ports for TCP, UDP and ICMP protocols. Open ports can be continuously changed as needed. After summing up properties of the instances and confirm the correctness are instances created. Figure 2. Created instances in EC2 C. Connect And Login to Instance Connection to the instance works via ssh. Because we are working in the Windows environment, we will use the program PuTTY. PuTTY works with private key in its PuTTY format. We have to convert our private key from EC2 to this format. Therefore, we use the program PuTTYgen. Log on to the instance is by using Public DNS, which we find out in EC2. We enter following Public DNS address, converted key to the PuTTY and login. After the connection is established we are prompted to entry a user name, we enter ubuntu. We are now successfully logged into the instance. IV. SETUP THE SYSTEM After successful login to the instance (master) we begin with system settings for correct running the MPI program. In the first step we have to on each of the four instances set the hosts file. Log on to other instances is the same as we have described here to the master. In this hosts file, we enter IP addresses allocated to instances and for easier work with them also their names and we verify the connections. For the proper functioning commands in the system, we need to first update the list of packages via apt-get update. Without this step, we could not further install additional required packages. After updating the package list, we install step by step the apt-get install build-essential and apt-get install dkms-blcr. Package buildessential is needed for compiling C language in which we have written a program and blcr-dkms package is needed for properly installing OpenMPI. Communication between the master and nodes is through ssh connection. Therefore, for the proper functioning programs we have to stored private key to our instances. We get the key to instances by using program which supports SFTP. In our case, we use FileZilla. We change file permissions for the private key with command chmod 400 <name of the key> and then we enter command `eval` ssh-agent and ssh-add <name of the key>. This guarantee us a direct connection between the instances. A. Torque Installation The program Torque is a resource manager which provides control over dosing tasks and distributed computing resources. At the present is considered Torque as one of the best in scalability, reliability and functionality. [3] After changing directory (cd /usr/local/src) we are downloading (wget-c) Torque in version We extracting Torque (tar xvf), change the directory to folder which was created after unpacking (cd torque-2.3.7) and run configure (. /Configure-with-defaultserver=<name of master stored in hosts file> --with-server-home=/var/spool/pbswith-rcp-scp). After completing the configuration we are finishing installation with the commands make and make install. When installation is done, we write to the file /etc/ld.so.conf.d/local.conf note /usr/local /lib and also execute the command ldconfig. At this point, we can create a local server on the master by command pbs_server-t create. Now we are setting up pbs server: qmgr c ser server scheduling=true qmgr c create queue batch queue_type=execution qmgr c set queue batch started=true qmgr c set queue batch enabled=true qmgr c set queue batch resources_default.nodes=1 qmgr c set queue batch resources_default.walltime=3600 qmgr c set server default_queue=batch qmgr c set server operators = root@localhost Figure 3. Commands for setting up pbs-server

3 After setting up the pbs server we are writing to the file /var/spool/pbs/server_priv/nodes name of nodes and number of processors which contains for example. wn1 np = 1. Packages for clients are prepared by command make packages and we are transferring these packages on each node (scp torque-package-*-linux-x86_64.sh ubuntu@<name of node>:/home/ubuntu for * we are entering mom, devel and clients). We moved installation packages for Torque to the nodes and now we are modifying the hosts file, realizing commands to update the package list and installing packages which we also did on master. We are installing Torque on the nodes by command sh torque-package-*- linux-x86_64.sh-install (for * we are entering mom, devel and clients). Check file server_name (/var/spool/pbs/server_name) if we have correct name for the master. We are launching pbs by command pbs_mom and for automatic run we are writing to the file /etc/rc.local note /usr/local/sbin/pbs_mom. When settings for Torque on nodes are complete, we pause and start queues on master by the commands qterm and pbs_server. If we have done it all correct, after typing command pbsnodes a we see a list of all nodes and their current status which is free. With this we have completed the installation and set-up Torque. B. Maui Installation Maui is one of the most used open source scheduler. We use it to improve the management and efficiency of instances consisting of several multiprocessor clusters. It is highly optimized and supports a wide range of planning policies. We use Maui in version [4] We have to download a package of Maui to this folder (/usr/local/src) and unpack this package (tar xvf maui tar.gz). Change folder to Maui folder (cd maui-3.3.1) and run configure (./Configurewith-pbs-with-spooldir=/var/spool/maui/ --with-key=770929). We continue in installation with command make and make install, which will finish the installation. Start Maui with export PATH = $PATH:/usr/local/maui/bin:/usr/local/mau i/sbin, next command is maui and check run with showq. Now we have installed in the master Torque and Maui and only OpenMPI is missing. C. OpenMPI Installation OpenMPI is an open source MPI-2 implementation that is developed and maintained current by the scientific, research and industrial institutes. Thus OpenMPI contains resources from professional community of high-performance computers, which guarantees us the best MPI library. [5] We are installing OpenMPI on the master and all nodes, so the process of installation is the same. First, install the supporting package openmpi-bin (apt-get install openmpi-bin). We change the folder to /usr/local/src/ and download the program OpenMPI in version Unpack the downloaded package and change the folder to openmpi Launch configuration:./configure --enable-languages=c,c++,fortran,objc,objc++,treelang --prefix=/usr/local --enable-shared --withsystem-zlib --libexecdir=/usr/local/lib --without-includedgettext --enable-threads=posix --enable-nls --enable- cxa_atexit --enable-clocale=gnu --enable-libstdcxxdebug --enable-mpfr --enable-checking=release x86_64- linux-gnu --with-ts=/var/spool/pbs Figure 4. OpenMPI configuration After completing the configuration in all instances run commands make and make install. First on the Node1 to Node3 and on master at last. In each instance open a file (/etc/ld.so.conf) and type into it a note include/usr/local/lib. Now run the command ldconfig. In this, we have created a functional grid and we have an instances ready to run MPI programs. To verify functionality we create a file with name list, in which we type the names of the master and nodes (as we have defined them at the beginning in the file hosts). As first in the list must be name of master and then nodes. We prepare the test program, which will return the order number and name of the responding node. Compile a written program with the command mpicc-o heelo.out hello.c and run mpirun-np 4-hostfile list hello.out. The output shows us the proper functioning of grid: Figure 5. Command for run test program and the result V. RUN THE PROGRAM SNOW Through the program FileZilla we have transfered files for a program of parallel computing - snow. The program works with input data from the file input.txt, where are stored input data from several measurement stations (data include the amount of snow cover and location coordinates). The size of the raster (matrix), we determine in the file main.c. After each change in the file main.c we have to make a new report with the command make.

4 The output of the program are two files: time.txt and output.txt. Output.txt file contains the calculated values (coordinates and height of snow cover). The file we can use for the graphical representation in the right program, such as GRASS GISS. Figure 6. Source code of the program In order our program works properly we send the current version of the program snow to all nodes. When we have sent the program to all nodes we can run the program with command mpirun-np 4-hostfile list snow. List informs us about the current size of the matrix, the number of worker, input data and the status of the program: Figure 8. File output.txt File tme.txt contains the value of the measured time. Time is measured from assignment the task to nodes until the calculated results is sent to master. VI. MEASURED TIMES We have received the values with successive measurements on Micro configuration and in the same way on Large configuration. Because it is a grid service, it should be possible to predict delays of the service provider. Thus we perform three measurements and selects the smallest measured time (differences in times were in hundredths of seconds). We start with matrix size 1000 x 1000 and successively increase it for each measurement by 1000, till Output file with measured values in our matrix size x is sufficient and has no great practical importance to continue to higher matrix. We test larger matrix on the Large configuration just for our curiosity what load can be calculated on our cluster. At x matrix, we find out that the output text file containing the measured times have the size of 1 GB. On Micro configuration, we could not measure matrix 5000 x 5000 because there was not enough computing power. Table 1. Measured times Matrix size Micro configuration Measured times (s) Large configuration Large vs. Micro (faster) 1000 x ,720 0,160 78% 2000 x ,530 0,610 59% Figure 7. Program snow

5 Matrix size Micro configuration Measured times (s) Large configuration Large vs. Micro (faster) 3000 x ,570 1,350 51% 4000 x ,380 2,250 49% 5000 x , x , x , x , x , x , x ,770 - Micro configuration, 4 instances, 1 CPU per instance: Intel Xeon Processor E5507 (4M Cache, 2.26 GHz, 4.80 GT/s Intel QPI), 613 MB RAM, 8 GB HDD Large Configuration, 4 instances, 4 CPUs per instance: Intel Xeon Processor E5645 (12M Cache, 2.40 GHz, 5.86 GT/s Intel QPI), 7,5 GB RAM, 8 GB HDD Graph of measured times: VII. SUMMARY With our experiment we were able to run program with parallel computations in the cloud. This finding we may meet in the future when we will need immediate great computing power. A great plus point of this solution is that in a few hours we can create and get the results obtained from the grid. Significant advantages are also costs. With a price $ per hour, 4 Micro configuration instances cost around $. This configuration, as we found by testing, provide average computing power (grid consists of 4 CPUs). An important part of running time of instances took their configuration and setup (3 hours), so the cost for time of computational power is around $. Cons for this solution are possible delays caused by server load. We can avoid to this problem calculating tasks in the evening or at night. At the time of testing we had this problem only once. Despite the potential disadvantages, we believe that if you are not able to use a local cluster, this is the best solution. Price per time/performance are very acceptable. [1] HURAJ, L., SILÁDI, V Efektívnosť a bezpečnosť gridových prostredí. Trnava : Univerzita sv. Cyrila a Metoda v Trnave, 2011, ISBN [2] Amazon Elastic Compute Cloud (Amazon EC2) [online]. Available on the internet: [ ] [3] TORQUE Resource Manager [online]. Available on the internet: [ ] [4] Maui Cluster Scheduler [online]. Available on internet: aui/index.php [ ] [5] A High Performance Message Passing Library [online]. Available on internet: [ ] Figure 9. Graph of measured times