This document is downloaded from DR-NTU, Nanyang Technological University Library, Singapore.

Transcription

1 This document is downloaded from DR-NTU, Nanyang Technological University Library, Singapore. Title Web access to cloud system Author(s) Tao, Qingyi Citation Tao, Q. (2014). Web access to cloud system. Student research paper, Nanyang Technological University. Date 2014 URL Rights 2014 The Author(s).

2 Web Access to Cloud System Tao Qingyi School of Computer Engineering Abstract - This URECA project Web Access to Cloud System is an attempt to enable the Hadoop file system accessible from mobile platforms. Particularly, the project is conducted based on the Android platform. Given the existing campus network and the established Hadoop servers, the major objective is to develop a mobile application to access the Hadoop File System (HDFS) implemented on the Hadoop servers from an Android device. The application is supposed to support basic functions of file management, including view, selection, upload, download, deletion of the files. The following tasks are achieved: the Google Android development tools are installed; the prototype of the user interface is designed and implemented; the file list from the cloud is shown with file information; single files could be deleted. This is the first attempt of accessing to the HDFS on mobile platform and a few limitations are discovered for further researches. Keywords Hadoop, Android, HDFS 1 INTRODUCTION The Hadoop Distributed File System (HDFS) allows users to access to the cloud for file storage and retrieving. However, to access to HDFS, the user is required to perform Unix commands or use the browser to view the file list. Moreover, as the only existing graphical user interface, the web interface only support list operation. It is inconvenient for users to manage the files on the cloud. Therefore, the objective of this project is to develop a mobile tool to access the HDFS from an Android device. Android application is mainly implemented with Java programming language. This report will provide the details of the project including the software setup, prototyping, and application implementation. Although the final outcome is not a fully functioning Android application, this report will help the future discovery on this topic as it provides the basic knowledge of Android programming for network connection purpose. 2 PRE-IMPLEMENTATION 2.1 ENVIRONMENT SETUP PC Setup Google Android is an open source platform mainly resides on mobile devices such as Android smart phones or tablets. Java is the recommended Assoc Prof Cai Jianfei Mr Lu Haifeng School of Computer Engineering programming language. To start the project, the following software needs to be installed on the PC. Install Java Development Kit (JDK) to enable Java runtime environment and compiler Install Eclipse for Java programming Install the running envirment for Android development: Android SKD or ADT with Eclipse Mobile Device Setup Although the Android develop tool provides the Android emulator from the PC to emulate the Android device, it could be more convenient and efficient to have an real Android mobile device for testing purpose. For this report, an HTC One with Android version 4.3 is used to run the application. The Android version is since with Android version lower than 4.1, the device will not support some necessary libraries used in this project Hadoop File System The Linux server has been setup for Hadoop services. The Hadoop service is available and functioning across the campus network NTUWL. The client device needs to access to the server IP address. For this project, the server side is the Hadoop server set up in the campus ROSE data center with IP address BASIC KNOWLEDGE OF ANDROID DEVELOPMENT As the beginner of Android programming, it is fundamental to know the basic structure of the application in terms of its programming components. The two key components of an Android application are 1) Activity (Java Class) and 2) View (xml layout). Android programming is based on Java programming language which is objective oriented. In particular, the Activity class is necessary for window creation. The activity is the foreground mode of an application. There are also background services that could run without the window changes. When the whole window is replaced by another activity, the current activity will be stopped or even killed. View is the displayable UI of the application. It is realized by xml layout configuration. In the activity class, the event can look up the corresponding view component by functions such as findviewbyid() could be used to update the activity actions in the View. In each view, a event listener could be registered and when 1

3 the user touches the view component, the event can be triggered by View.OnClickListener. 2.3 REQUIREMENT ELICITATION AND PROTOTYPING The main task of this project is to refine the GUI to access to the Hadoop File System on mobile device. Therefore, the application needs to have a clear and concise GUI to present the items in the file system, i.e. the directories and files with details such as the file name, size, and the last modification time. It should also have the menu for action selection. The actions for touch screen such as short click and long click effects can be utilized to reflect the advantage of using touch screen devices. To make the task clearer, the low fidelity mock-up of the UI is shown as below Start Page This is a relative layout with simple image view and a start button. The reason to have this page is to allow further improvement when the system needs log in control. The text areas could be easily added to retrieve the username and IP address File List View The file list view is the page where all files under the current directory are listed with file details. Each item in the list view carries the file name, file size, last modification time and the file icon (directory or file). The text view and the image view is identified by id as shown below. <TextView android:text="@+id/textviewname" android:id="@+id/textviewname" android:layout_width="wrap_content" android:layout_height="wrap_content" android:singleline="true" android:textstyle="bold" android:layout_torightof="@+id/fd_icon1" android:layout_margintop="5dip" android:layout_marginleft="5dip"> </TextView> Action Bar The action bar is the menu at the top area with the action icons home, go back, upload and create directory. The action bar can be created by adding Menu Element to the xml file in the res menu folder. In this application, the following three elements are added: Figure 1Application Low-Fi Mock Up The UI can be divided in to three parts: the top menu of actions, the main content of the file list and popup dialogs when any action is triggered. 3 IMPLEMENTATION 3.1 GUI IMPLEMENTATION The first step to build up this application is to draw the GUI by creating new xml file under res layout folder. There are several different layout for this application: a start page, a file view page, the top menu view, and some popup windows. Each view is an xml file in the res folder. Figure 2 Menu Element Capture Popup Dialogs When the user clicks the icons in the action bar or triggers special events, the prompts or confirmation dialogs will be triggered. In this application, the following dialogs are implemented: 1) The confirmation dialog for uploading files 2) The confirmation dialog for deleting files 3) The text dialog to enter directory name for directory creation 2

4 Similar as the main layout, the dialogs are consist of the text view, buttons and image view if needed. The figures below are some captures of the GUI. Figure 5 Upload Confirmation Figure 3 Start Page Figure 4 File List View 3.2 KEY ENTITIES Figure 6 Action Prompt Hadoop File Enity public class HDFile HDFile is designed to hold the information of the files in Hadoop file system at the server side. It has two main attributes: name and absolute path. The attributes identify a file in HDFS. Apart from the getters of the attributes, getparentname () and getparentpath() are 3

5 implemented since they are useful to go back to upper directory Displayable Item Entity public class Item implements Comparable<Item> This class is used for the file list items in the view. It holds the item s name, data, date, image, i.e. anything will be presented in the list view Array Adapter public class FileArrayAdapter extends ArrayAdapter<Item> The array adapter can accept the list data structure, and then construct the list view for display purpose. 3.3 MAIN ACTIVITIES The first activity is the file explorer activity: public class FileexplorerActivity extends Activity. This content of this main activity is set as the start page (Home). When the start button is pressed, the new intent is created. Intent is used when application switches from one activity to another. Intent intent1 = new Intent(this, FileChooser.class); The next activity is: public class FileChooser extends ListActivity. The is the key activity of this program. This file chooser activity extends list activity and it displays the file list of the Hadoop server. All server connection actions are implemented in the file chooser class to retrieve the server output and extract the details. 3.4 SERVER CONNECTION With the GUI and the main activities created, the most essential task is to connect to the Hadoop server. The below permission should be added in AndroidManifest.xml file to allow network access. <uses-permission android:name="android.permission.access_netwo RK_STATE" /> The server connection functionality uses WebHDFS REST API [1] of Hadoop file system. From Android side, the methods provided Client library can access to the url with various methods such as get, put and delete. Function List Files Make Director y Delete Open Table 1 Request Type Operation GET PUT DELETE GET URL " RT>/webhdfs/v1/<PA TH>?op=LISTSTAT US" " RT>/<PATH>?op=M KDIRS[&permission =<OCTAL>]" " webhdfs/v1/<path>?o p=delete [&recursive=<true fal se>]" " RT>/webhdfs/v1/<PA TH>?op=OPEN[&off set=<long>][&leng th=<long>][&buffe rsize=<int>]" Client Reuqest Type HttpGet HttpPut HttpDelete HttpGet Since the server connection needs to do in the background without changing the current window display in the application, these tasks are defined as AsyncTask. For example, the list file task uses private class HttpGetTask extends AsyncTask<Void, Void, Void> in which http get method will do in background. The source code section is provided below. protected Void doinbackground(void... params) { if (!iscancelled()){ HttpClient httpclient = new DefaultHttpClient(); // Prepare a request object HttpGet httpget = new HttpGet(url); // Execute the request HttpResponse response; try { response = httpclient.execute(httpget); HttpEntity entity = response.getentity(); if (entity!= null) { InputStream instream = entity.getcontent(); jsonstr= StreamConverter.convertStreamToString(instream); instream.close(); try { jsonresponse = new JSONObject(jsonStr); catch (JSONException e) { 4

6 e.printstacktrace(); Similarly, for PUT and DELETE operations, separate AsyncTask classes are created and responses could be retrived. By giving correct url, the http response could be retrieved. After the stream is converted, the response from the server could be reprensented in JSON format. Figure 7 File Statuses JSON Reponse 3.5 INFORMATION EXTRACTION FOR FILE LIST STATUS In Java, there is a data type called JSONArray. The JSON string could be converted into an array of JSON object. filestatusesresponse = new JSONObject (jsonresponse.getstring("filestatuses")); filestatusarray = new JSONArray (filestatusesresponse.getstring("filestatus")); The JSONArray needs to be converted into a list of items for display, so the JSON object also needs to be parsed. The item name is defined by the value of pathsuffix. The item type is determined by the value of type which could be a directory or a file. The 13-bit modification time is represented in format of UNIX machine time stamp. Therefore it should be parsed into simple date format. Finally the item list will be sorted by name and the directories will be shown on the top followed by the files. public List<Item> getfilelistitem (JSONArray filestatusarray){ int lengthjsonarr = filestatusarray.length(); List<Item>dir = new ArrayList<Item>(); List<Item>fls = new ArrayList<Item>(); try{ int i = 0; for (i=0;i<lengthjsonarr; i++) { JSONObject filestatus = filestatusarray.getjsonobject(i); String filename = filestatus.getstring("pathsuffix"); String timestampstr = filestatus.getstring("modificationtime"); long timestamp = Long.parseLong(timeStampStr); java.util.date time=new java.util.date((long)timestamp); SimpleDateFormat df = new SimpleDateFormat("dd-MM-yyyy HH:mm:ss"); String formatedtime = df.format(time); String size =filestatus.getstring ("blocksize"); boolean isdirectory = filestatus.getstring("type").equals("directory"); String absolutepath = currentdir.getabsolutepath() + "/"+filename; if(isdirectory){ dir.add(new Item(fileName,"Directory",formatedTime,absolutePath,"directory_ico n")); else{ fls.add(new Item(fileName,size+ " Byte", formatedtime, absolutepath,"file_icon")); catch(exception e){ e.printstacktrace(); Collections.sort(dir); Collections.sort(fls); dir.addall(fls); return dir; 4 LIMITATION AND FUTURE RESEARCH TASKS The implemented application is able to perform the GET, PUT and DELETE request. However, the open and upload functions are not fully implemented at the back end. Due to the setup at the server side, the request with operation = open will be performed with two steps. Firstly, a GET request is submitted with automatically following redirects. Then the request is redirected to a datanode where the file data can be read. Finally The client follows the redirect to the datanode and receives the file data. The datanode address is in a server local format: Location: RS4:50075/webhdfs/v1/user/flyingkid/setup.txt?op=OP EN&user.name=flyingkid&offset=0 5

7 Therefore the host name CeMNet-RS4 is not recognizable by the android device. After research, it is found that the host name should be added in the host file of the android device. The host file is not accessible in a normal condition and the editing action needs the super user permission. Therefore, the Android device needs to be rooted. With many trails, the redirection of http request is not successful and no response could be get from the server side. Therefore the open and download functions are not successful implemented. In the future researches, it is also possible to setup an agent server to coordinate the communication between the Hadoop server and the mobile client. 5 CONCLUSION In this project, an Android application is implemented to access to the Hadoop server with the campus network. The application support the listing, making directories and deleting functions, whereas the open and upload functions are not implemented. With basic knowledge of the Hadoop file system and the Android programming, the functionality is relative simple but the UI is quite clear and neat. However, there are still a couple of unsolved problems including how to get the response after the request is redirected and how to manage the background tasks if multiple background tasks are in execution. This is a very first attempt of implementing Hadoop file system into the mobile platform, more effort should be put into this topic to eliminate the limitations. ACKNOWLEDGMENT I wish to acknowledge the support for this project from Nanyang Technological University under the Undergraduate Research Experience on CAmpus (URECA) programme and the resources provided by the ROSE data centre. REFERENCES [1] Unknown,(2013, Feb13) WebHDFS REST API, Apache Hadoop Project, Retrived from l 6