BIRLA INSTITUTE OF TECHNOLOGY & SCIENCE, PILANI July, 2006

Transcription

1 BACKUP PLANNING AND IMPLEMENTATION FOR ANUPAM SUPERCOMPUTER USING ROBOTIC AUTOLOADERS BY Aalap Tripathy 2004P3PS208 B.E. (Hons) Electrical & Electronics Prepared in partial fulfillment of the Practice School I Course AT Supercomputing Research Facility, Computer Division Bhabha Atomic Research Centre, Trombay A Practice School I Station of BIRLA INSTITUTE OF TECHNOLOGY & SCIENCE, PILANI July, 2006

2 ACKNOWLEDGEMENT It s a moment of intense pleasure to express my gratitude towards everyone who directly or indirectly helped me during this project work. I am thankful to Sh. A G Apte, Head, Computer Division, for providing me an opportunity to work on this project. My sincere gratitude to Mr. K Rajesh, SO(F), SRF, Computer Division to have assigned & assisted me in the course of his project. Despite his busy schedule, he always found time out for his sharp review of the development I made during the day. It is truly an honour for me to have been associated with such a co-operative and extraordinarily brilliant scientist. But, I am absolutely sure that this project work couldn t be better as it is now without timely trouble shooting by Mr. Kislay Bhatt, SO(E), SRF, Computer Division who has rescued me from numerous weird error messages. Thanks to Mr. Phool Chand, SO(E), Computer Division for having helped in design of the Backup Management Suite especially in critical PHP code segments. I must specially thank Mr Rohitashva Sharma, SO(E), Computer Division for his timely help and support in Mr Phool Chand s absence. Thank you, Sirs. I must mention here that none of this would have been possible but for the confidence of Dr. Himanshu Agarwal, PS-faculty-in-charge in my abilities. He made it a point to have me assigned to this Division of my choice for my project. Aalap Tripathy aalap@bits-goa.ac.in 2

3 BIRLA INSTITUTE OF TECHNOLOGY AND SCIENCE PILANI (RAJASTHAN) Practice School Division Station : Computer Division, BARC Centre : Mumbai, Maharashtra Title of the Project : BACKUP PLANNING AND IMPLEMENTATION FOR ANUPAM SUPERCOMPUTER USING ROBOTIC AUTOLOADERS Duration : 24 th May, th July, 2006 Date of Start : 2 nd, June, 2006 Date of Submission : 13 th, July, 2006 Prepared By : ID Number Name Discipline 2004P3PS208 Aalap Tripathy B.E. Hons. Electrical & Electronics Experts Involved : Name Designation Division Mr K Rajesh Scientific Officer (F) Supercomputing Research Facility, Computer Division Mr Kislay Bhatt Scientific Officer (E) Supercomputing Research Facility, Computer Division Mr Phool Chand Scientific Officer (E) Supercomputing Research Facility, Computer Division Name of the PS Faculty : Dr. Himanshu Agarwal, Asst. Professor, Mechanical Engineering Group Key Words : Automation, Backup, Supercomputers, High Performance Clusters, Autoloaders, Tape Drives, High Capacity Tapes, Backup management, Shell Scripting Project Areas : Backup Automation for user data to the tune of 3.2 Terabytes. Customized scripts to simplify administration of the Autoloader by robotics control commands Abstract : This project aims to automate backup from 12 fileservers to tapes mounted on Tandberg Tape autoloader - LTO Ultrium ( Ultrium 2 ) by the use of shell scripts. A complete package including web interface has been developed and customized for use for ANUPAM supercomputer at BARC BMS (Backup Management Suite) Signature of Student Date Signature of PS Faculty Date 3

4 CONTENTS Sl. No. Title Page Numbers 1 Introduction Shell Scripting, PHP, Javascript 6 3 Commercial/Open Source Implementations 7 4 Resources Used 8 5 Overview 9 6 Implementation Details 10 7 Code Functionality Code Explanation Future Enhancements References 25 Appendix A : Technical Specifications ANUPAM Supercomputer Appendix B : Technical data - Tandberg SuperLoader LTO2 Appendix C : Technical data Backup Server Appendix D : Technical data File Server Appendix E : Study of User Space Utilisation (21st June, 2006) Appendix F : Code View of the scripts written Appendix G : Code View of the PHP Scripts & HTML Pages 4

5 1. Introduction 1.1 Objective The major driving force behind this work was to automate the back up procedure for the fileservers (sagar01-sagar12) from a full backup taken on the 1 st, 8 th, 15 th, 23 rd days of a month to a combination of full and incremental types. Failsafe procedures were to be developed with appropriate checks for extraordinary circumstances like power failure / hardware error. Logging was to be improved. Index files of the files being created were generated which were searchable. The aim is to provide a complete backup management suite similar to what major commercial/open source packages provide. Some additional features like holding disks etc were also to be implemented. 1.2 System Analysis and Definition ANUPAM is a series of parallel computers developed at Computer Division, BARC to cater to the need of high computing power for various high performance applications. This system is used by BARC scientists in a wide range of fields like Reactor Physics, Reactor Engineering, Crystallography, Molecular Dynamics and Computational Fluid Dynamics etc. The cluster is spread across 14 racks. o 10 racks(racks 1-5, 10-14) house the compute nodes o 2 racks (racks 6,9)house the storage and backup servers o 2 racks (racks 7,8) house the service nodes and networking equipment. o Racks numbered N1 to N4 are campus networking racks Fig : Plan Layout of Ameya Supercomputer Fig : Perspective View of Ameya Supercomputer 5

6 Fig : Perspective View of Aruna Cluster System/Network Diagram of AMEYA Fig : Perspective View of Ashva Cluster All of the user s files are stored onto partitions located on any one of the 12 Storage Servers. These are accessible from the backup servers (kosh01 & kosh02) which are themselves interfaced to tape backup devices which in turn perform the backup operation as explained further. All of the development and control is done from the User Terminals shown above. 6

7 1.2.2 Submission of jobs A job is typically a shell script, FORTRAN programme, C language program etc with a set of attributes which provide resource and control information about the job. A batch means that the job will be scheduled for execution at a time chosen by the subsystem (defined below) according to a defined policy and the availability of resources. Portable Batch System (PBS) is a networked subsystem for submitting, monitoring and controlling a workload of batch jobs on one or more systems. In PBS, a queue is just a collection point for jobs and does not imply execution ordering. This is done by a scheduling policy decided by the system administrator. The above details are only for academic interest and are not required for an understanding of the backup mechanism described later. 1.3 Analysis of Organizational Data For the execution of a job, users of ANUPAM have to submit valid program(s) with associated data set onto their user space. The results of their computation are also stored onto it. There is no restriction on the number/size of jobs which can be submitted to the system. System Administrators are also flexible with respect to the amount of space allocated to a user i.e. it is often changed on request. However the size of filesystems is fixed and their entire content is to be backed up to the tape drives. User Data for Anupam (Ashva and Ameya) consists of programs, scripts applications. These are present on the Fileservers which are mounted by an auto mount script by the Network File System onto mount points accessible as /home/usr010, /home/usr011, /home/usr020, /home/usr021, /home/usr030, /home/usr040, /home/usr050, /home/usr060, /home/usr070, /home/usr080, /home/usr090, /home/usr110, /home/usr120 Besides there are other partitions which are not backed up. They are in single digits viz /home/usr7. Each of the above mentioned partitions contain home directories of various users. Each one of the above mentioned partitions are of size 200 GB each. A study of each partition as detailed in Appendix C shown showed that in most cases the entire space is not being used currently. 1.4 Choosing the right backup type requirements v/s issues involved Backup Scenarios Server Backup Large Network Backup Personal Backup 7

8 The system at ANUPAM corresponds to a Large Network Type where the file systems are connected by Fast Ethernet Switches ( 1 Gbps) to each other and the controller. When to backup o It is a resource hungry process o It stresses both the systems and the network o Full backup takes 4 hours on an average o It should therefore be done when the system is not in active use o Full Backup is may be done on weekends where the backup window is hours o Incremental Backup to be done on weekends after Office hours with a backup window of 3-12 hours Volume Configuration o The data on each file server is in terabytes, so high efficiency is to be achieved by efficiently transferring only essential data minimum number of times. o The life of the tapes is also an issue. Supplied tapes have a proclaimed efficiency of 2000 rewrite cycles. Backup Policy o The backup policy to be such that, system administrators can assure users that their files will be safe upto the maximum possible extent. o Also it should be able to restore a version of a file that the user requests say the version on <date>. o All possible user files should be backed up onto tapes with minimum effort. 1.5 Possibility of growth There are currently 12 fileservers. Each of them have averaged GB hard disks. This computes to = GB approx 14 Terrabytes. Further, there is a proposal to increase the current configuration from 256 nodes (256 2 = 512 CPUs) to an aggregate 1000 CPUs. There is also a proposal to migrate to higher capacity tape drives with autoloader capacity. The current system is to be built to survive this expansion successfully. Also Autoloader s with 100 tapes is proposed to be bought. The backup scripts should be able to run on the advanced autoloaders easily. 1.6 Tape Loader Interfacing The Tandberg Autoloader was equipped with 16 removable tapes each having a recording capacity of 200 GB. This adds up to available storing capacity of 3200 GB i.e. 3.2 TB This Autoloader was interfaced to a server running Scientific Linux. This server was kosh02. 8

9 2. Shell Scripting, PHP, Javascript The shell, in UNIX and Linux is the equivalent of a command interpreter in Windows. Its job is to accept commands, as typed by the user on the command line, and interpret each of those commands, acting on it however necessary. A shell script is in essence, a whole bunch of shell commands entered together in a file. A little like the DOS batch file, where many shell commands are grouped together to perform some function(s). PHP is a widely used open source, general-purpose server side scripting language. It was originally designed for use in Web site development. In fact, PHP started life as Personal Home Page tools, developed by Rasmus Lerdorf to assist users with Web page tasks. PHP proved so useful and popular, it rapidly grew to become the full-featured language that it is today, acquiring the name PHP Hypertext Preprocessor along the way to represent its expanded abilities processing Web pages before they re displayed. The popularity of PHP continues to grow rapidly because of its many advantages: o It s fast: On Web sites, because it is embedded in HTML code, the time to process and load a Web page is short. o It s free: PHP o It s easy to use: The syntax is simple and easy to understand and use, even for non-programmers. For use in Web sites, PHP code is designed to be included easily in an HTML file. o It s versatile: PHP runs on a wide variety of operating systems Windows, Linux, Mac OS, and most varieties of Unix. o Technical support is widely available: One can join one of several discussion lists offered on the PHP Web site ( which cover topics such as general PHP, PHP on Windows, or databases and PHP. In addition, a Web interface to the discussion lists is available at news.php.net, where one can browse or search the messages. o It s secure: As long as the scripts are designed correctly, the user does not see the PHP code. o It s customizable: The open source license allows programmers to modify the PHP software, adding or modifying features as needed to fit their own environments. PHP provides significant control over the environment, reducing chances of failure. PHP has many features designed specifically for use in Web sites, including the following: o Interact with HTML forms: PHP can display an HTML form and process the information that the user types in. o Communicate with databases: PHP can interact with databases to store information from the user or retrieve information that is displayed to the user. o Generate secure Web pages: PHP allows the developer to create secure Web pages that require users to enter a valid username and password before seeing the Web page content. PHP is only server-side, meaning it can t interact directly with the user s computer. That means PHP can t initiate actions based on the status of the user s computer, such as mouse actions or screen size. Therefore, PHP alone can t produce some popular effects, such as navigation menus that drop down or change color. On the other hand, JavaScript, a client-side scripting language, can t access the server, limiting its possibilities. For example, one can t use JavaScript to store data on the server or retrieve data from the server. One can use JavaScript and PHP together to produce Web pages that neither can produce alone. Therefore, I have used PHP embedded in HTML pages together with javascript in my final application to produce the desired output. 9

10 3. Some Commercial/Open Source Implementations Backup is like the weather, everyone talks about it but no one can do anything about it Many open source and commercial packages which readily perform the operations that are the aim and objective of this project are already available. However the commercial packages are prohibitively costly and often come bundled with specific hardware. Most exist only for the Windows Platform. The common open source packages like AMANDA (Advanced Maryland Automatic Network Disk Archiver), BACULA, lonetar, Omnibackup, Net Backup, BRU (Backup & Restore Utility), Arkeia and Brightstor were tried out. Most of them were difficult to configure and did not support all hardware configurations. Also they could not be readily customized. One had to depend on the original creators to perform any changes. However, I tried out AMANDA and BACULA before setting down to develop my own package. The contributing factor was that AMANDA which is the current industry leader was developed as a generalization of a package developed at the University of Maryland for their HPC. I have tried implementing all the features that these packages offer in my Backup Management Suite. If you are currently using a program such as tar, dump, or bru to backup your computer data, and you would like a network solution, more flexibility, or catalog services, Bacula will most likely provide the additional features you want. BACULA essentially answered the problem I was facing. The system here was being used with tar and a network based solution was required with catalog services. I gave up on it primarily because, it was a network based structure. Here independent packages such as Admin Workstation, a Database Server, a Specialised Backup Server had to installed on independent machines. They could however theoretically all be installed on a single machine as the manual said. Besides this unnecessary complication, I still lacked the ability to customize this software. After a few trial runs, I gave up on this and decided to build my own software base using existing and tested tools. However I decided to necessarily implement all features that such packages promised such as catalogue services, indexing facilities, tape scheduling facility, search utility etc. 10

11 4. Resources Used 4.1 Hardware Used ANUPAM P-IV System User enters their data/programs by logging on to ashva01 system. This is linked to their home directories placed on one of the sagar fileservers. These details change dynamically as determined by the system administrator and hence are not presented in this report. KOSH Hardware Controller Kosh02 was the computer system used to give commands to the tape autoloader. It was interfaced to this system by SCSI cabling. Also the fileservers were connected by Ethernet to the two supercomputers. 4.2 Software Used Development Platform : Scientific Linux Development Tools : 1. Vi Editor 2. Shell Scripts 3. Linux shell commands 4. HTML 5. PHP 6. Javascript Libraries : 1. mtx tape library 2. Standard Linux/Unix components 3. Standard PHP String/Array Functions 11

12 5. Overview To gain better understanding of this work, let us first see the block diagram of the overall system as shown in the figure Job Submitted by the user Ameya Nodes Gigabit Switch Ashva Nodes Gigabit Switch Gigabit Switch Kosh02 PBS User data placed in his home directory on one of the sagar s º º Individual Tapes on storage magazines (8 in number) /dev/st2 Tandberg Autoloader accessible on kosh02 as /dev/sg5 1. For Full backup, we copy one fileserver onto one tape. Although, in some cases, the disk usage was much less, this provision was made keeping in view possible future expansion wherein the entire fileserver capacity might be utilized. 2. For incremental backup s, I used tapes numbered 14,15,16 to store files/directories whose inode times have changed beyond the last full backup time. Inode value changes when the file is modified/created/permissions changed etc. A list of all such files is generated by the find command. This is passed onto the tar utility which compresses and writes onto the tape drive /dev/nst2 i.e. currently loaded tape. I also calculate time taken and number of bytes written onto files. This data is checked. If one tape s limit is exceeded, the system progresses to the next tape until the 16 th tape is full. After this a full backup is forced or happens by virtue of backup policy and the tape status is reset to 14 where a new incremental cycle begins. 3. A backup find (bfind) utility was developed to pass through the index files and report which tape and no. of space count s ahead, the required file is in. 12

13 6. Implementation Details 6.1 List of files which are part of package A complete listing of all files part of this package is given in Appendix C. However, since the package is still under testing, some file contents may not match with those loaded on the system. The basic concept remains the same. change used to load desired tape on tape drive. Many error check conditions implemented full main script to perform automated biweekly backups to 13 tapes from 13 fileservers incremental performs daily incremental backups basing on current utilization of loaded tape bfind It is a utility script to search for file/username etc on the tapes. Parses index files for results 6.2 File Recovery The full backup routine places tar s of entire file partitions onto tape. Either the entire filesystem or particular file can be extracted using tar xvf /dev/nst2 <filename> For incremental recovery however, the procedure is slightly complex. Tandberg drives automatically rewind tapes after use. When a tape holding incremental data is accessed, we can view the first tar only. Subsequently on the 2 nd tar xvf after this, the second valid tar is accessed. This is because the tape hits a EOF marker of the first tar, proceeds to the next one. Alternately, as the space count is known from the bfind utility and the tape can be made to progress by the command mt f /dev/nst2 fsf <space count from bfind> -1-1 (minus 1) is required as fsf places tape head at beginning of the next archive 6.3 Proposed Backup Schedule Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 or or 7 Incremental 1 Incremental 2 Incremental 3 FULL Incremental 5 Incremental 6 FULL This symbol indicates crossover time i.e. the beginning of the next day ( currently chosen as 1 AM) 6.4 Observation & Tweaking 1. It was observed that some files in the user space were forward dated. So they appeared regularly in the incremental backup s performed. This issue was reported to Mr K Rajesh, who immediately took corrective action. 2. It was observed that some tar s took exceptionally large amount of time to backup, even though they were simply daily incremental dosages. This was because a user on the particular file system was using 13

14 7. Code Functionality 7.1 Requirement The Shell scripts created previously required a privileged user (root) to execute or schedule the scripts. This was an obvious disadvantage. To avoid such complication a web based user interface was developed. The Coding was done on PHP as it is natively supported by Apache Web Server installed on the linux machines here. Number of configuration steps were taken which I am not mentioning here. Also SELinux Protection had to be disabled for apache to run. Some of the scripts require high memory usage. Therefore appropriate settings in the /etc/php.ini file were also modified accordingly. 7.2 Application Interface This is the primary interface to manage operations on the Autoloader. The user has options to View File Indexes, set restore folder options for files in Full Backup or Incremental Backup Logs. Also if particular file/directory is known, the user can directly search for them by selecting appropriate option. 14

15 Here the user has chosen to search only the full backup logs. These are sorted date-wise and further alphabetically. Here the user is searching the incremental logs only where he views all versions of a file available on the various tapes. Also all details about the file as recorded in the logs are available. 15

16 This is a individual file search ( bash ). The above figure shows all backed up files containing the word bash. Also in case any increments are avaliable, they are shown on the next level. For convenience details like date of backup, permissions, owner, filesize,date of creation/modification time of creation/modification is also shown. Also there is an entry like This means this particular file is avaliable in the 14 th tape at a space count interval of 40. To extract this however, one has to use fsf Logs & Records Essential Data Logs/Records are maintained in the /usr/local/backup/log folder as shown below. However, File Log Records are maintained in /usr/local/backup/status/ folder Left : Contents of the Log folder. Shows contents of full and incremental folder. Top : Contents of incr/data folder 16

17 Clockwise(L-R) : 1. Contents of full log folder, 2. Sample entries in one of full logs, 3. Contents of incr log folder, 4. Sample entries in one of incremental logs

18 8. Code Explanation This section is written specifically for someone who is trying to modify the code written to add more functionality to the existing package. I assume the reader to be familiar with shell scripting, PHP, HTML and javascript which is used extensively in this project. Maximum understanding can be obtained by uncommenting the existing echo/display/print statements which I have purposely left in the final code. In some cases I have included small comment segments explaining what I am doing. Here I present a brief explanation of some of the critical code segments 8.1 PHP Pages I started this work primarily because I was asked to develop a file explorer kind of listing available commonly, however for existing filesystems. Here I had recursive listing of the files backed up onto tapes indexed by partition-date-tape number-space count. index.php This is the basic navigation page and was actually created last. However for consistent reading, I am including this at the beginning. Basic HTML and include statements are not covered in this explanation. Only most critical code is covered. <form method="post" name=send_full action="display_array.php"> Full Backup Files :<select name="which_index"> <?php system("cd /usr/local/backup/status/full_status/ ; ls i* > /tmp/file_list"); $data=file("/tmp/file_list") or die('</select> </form> Cannor=t find index listing'); for($ctr=0;$ctr<=count($data);$ctr++) print("<option value=\"$data[$ctr]\"> $data[$ctr]</option>\n"); system("rm -rf /tmp/file_list");?> This form sends the restore folder and the full backup log to be accessed to the display_array.php (and to create_array.php ) Here I am accessing all my full index logs ( I list only files starting with i to ensure only index-full/incremental-* are listed. I read each line of output into the select s value tags. I observed that the file function is the best to handle large input. Using fread, fgets etc gave me size restrictions which I could not rectify despite making changes in the /etc/php.ini file. A similar form sends parameters for the incremental listing. <H2><LEFT>Search Particular File or Directory </LEFT></H2> <form method="post" name = send_param action="search.php"> <font color="#ffffff"> Search Parameter : <input type='text' name='search' value=''><br> <input type="radio" name="search_type" value=1><font color="#ffffff"> Full Backup<br></font> <input type="radio" name="search_type" value=2><font color="#ffffff"> Incremental Backup<br></font> <input type="radio" name="search_type" value=3><font color="#ffffff"> All Backup Logs<br></font> <input type='submit' value='search'> </form> </font> <BR><BR> 18

19 later Here I send the search parameter and which search_type to search for use in the search.php explained <?php system("./auth -l > /tmp/crondump"); $filearray=file("/tmp/crondump"); $i=0; while($filearray[$i]) echo "$filearray[$i]<br>"; $individual_words = explode(" ",$filearray[$i]); //echo "$individual_words[2]<br>"; $i++;?> I tried to develop a feature for operator controlled modification of the backup schedule. However initially the apache user (under which all this runs) was unable to access/modify the cron schedule without superuser access. Rajesh Sir suggested a program appended below #include <unistd.h> #include <stdio.h> char *str="crontab"; main ( int argc, char *argv[]) argv[0]=str; execv("/usr/bin/crontab",argv); This is a special case C program which I use in the line system("./auth -l > /tmp/crondump"); instead of saying crontab l. This is to make the system believe that the program is being run by root and so it displays the roots crontab. However, I needed to write separate programs like this to enable all required root privileged commands/scripts to be executed. However sudo is a tool which is available. This has not yet been implemented successfully during the writing of this report and must be tried out in future. Also I wanted to parse the crontab results and graphically show the backup schedule in a format as below which I haven t done yet. Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 or or 7 Incremental 1 Incremental 2 Incremental 3 FULL Incremental 5 Incremental 6 FULL This symbol indicates crossover time i.e. the beginning of the next day ( currently chosen as 1 AM) create_array.php So I first build a recursive array building function which could convert any text file containing input as x/y/z/.. into an infinite array with keys being the content of each level. This is built using the create_list function. function create_list($file_comp) global $map; 19

20 $var = & $map; for ($i =0; $i < count($file_comp); $i++) $var[$file_comp[$i]][] = array(); $var = & $var[$file_comp[$i]]; Here $file_comp is a component of one line (say x) it recursively checks all entries at its level and assigns its children to it else is creates a new entry at this level. Here, however I am generating some bogus elements such as some keys pointing to null arrays etc. This code can definitely be improved. But, I think the logic would remain essentially the same. $which_file=$_post['which_index']; $file_param=explode("-",$which_file); which_index is a hidden entry passed from the index.php page containing the name of the log file to operate on. if ($file_param[1] == 'full') system("cat /usr/local/backup/status/full_status/$which_file egrep -v '/$' > /tmp/bms_file "); elseif ($file_param[1] == 'incremental') system("cat /usr/local/backup/status/incremental_status/$which_file egrep -v '/$' > /tmp/bms_file "); In the above code, I use egrep v /$ This is required to access only the lines which do not end with a / This is required because the recursive array forming function requires at most a single line containing the entire file structure. In case I allow recursive listing, the program functions, however inefficiently requiring longer time and generates some lines of gibberish output. Besides, the array size increases exponentially. display_array.php <script language="javascript"> function add_to_selected(key) document.send.keys.value += "/" + key ; document.send.submit(); </script> This script when called adds the selected key to the keys object in the send form. I also add a / because this essentially gives me a path listing. This is subsequently resent to this page again to regenerate the tree display. Here I want to add features which recognize clicking of a node other than the currently opened node and separate them with another unique identifier such as ; to show another node. This can be subsequently exploded on ; to obtain separate lists. Currently I don t operate this as the way I search a node doesn t need this. However if we select a node independedent of the one selected, it opens normally but appends its name to the current key list only separated by a / This is wrong but is solving my requirement. <form method=post name=send_to_create action="create_array.php"> <input type=hidden name=which_index value='<?=$_post['which_index'];?>'> </form> <?php include("create_array.php");// This includes the created array $map 20

21 This passes the file_name selected in index.php and sent to display_array.php again to create_array.php where it is actually required. Cookies could have been also used for this task. However for generality and ease of use, I chose this method. function displ($map,$level,$limit,$keyword) foreach ($map as $key => $val) check ($key,$val,$level,$map,$limit,$keyword); This function uses the array generated ($map), the levels it is at ($level), the level upto which to go to ($level) and the keyword to search keys for ($keyword) as entries. Since displaying a file listing is essentially a recursive process, every parameter excluding $map and $level remains constant throughout the function call and are checked at every level. To check I use another function called check. function check($key,$val,$level,$map,$limit,$keyword) //Display variables $tab_space = 3; $line_width = 1; $line_length =$tab_space*10-5; //echo strpos($key,"full"); //if (strpos($key,"full")==0) // echo "Got it"; if (is_array($val) &&!is_numeric($key) &&!is_null($key) && $key!="\n" && $key!=$keyword[$limit])?> If a key points to another array as value, and is not numeric and the key doesn t equal the keyword at limit level it is a valid entity and must be displayed. The following lines just perform the formatting like assigning the required spaces and displaying the tree structure images <tr> <td align =left class=fixed_width> <? for ($i=0; $i<=$level;$i=$i+0.3) for ( $j=0; $j < $tab_space; $j++)?>&nbsp<??> <img src=images/red.bmp width=<?=$line_width?> height=30 align = middle hspace=0 vspace=0 border=0> <img src=images/red.bmp width=<?=$line_length?> height=<?=$line_width?> hspace=0 space=0 border=0 > <? if($key!= $keyword[$limit])?> <a href="javascript:add_to_selected('<?=$key?>')" > <img src="./images/plus.gif" width = 15 height = 15 hspace=2 vspace=0 border=0 align=middle></a> <? Here I call the javascript to add the selected entity to the keylist because this has to now be expanded. else?> <img src="./images/folder.jpg" width = 15 height = 15 hspace=2 vspace=0 border=0 align=middle> <??> <?php echo "<b> $key</b>";?></a><?php echo "<br>"; $level++; 21

22 if ($level <= $limit+1 && in_array($key,$keyword)) //if ($key == $keyword)echo "I am breaking";break; displ($val,$level,$limit,$keyword); elseif (!is_numeric($key) ) echo "$key";?> In this program, I recursively search each level (level 0 onwards) until the key+level match the user s selection. In each step I pass, the reduced array recursively to the display function. In the first case, when the user hasn t selected anything, I obtain only the top level entries. Actually this code segement can be used to display a recursive tree structure for an input file having a tree level hierarchy stored in x/y/z/a/b/c. format search.php This is by far the most complicated program I wrote essentially because I did not use any functions. Therefore many lines get repeated over and over again. This happened because I developed this page after numerous trial and error attempts and actually didn t expect to be successful. When I realized the need for having functions for each and every operation, the program had become extremely complicated and implementing functions would have required me revisiting each line of code for consistency and generality $search_option=$_post['search_type']; $search_string=$_post['search']; $date_previous=''; $disp=array(); $tab_space = 3; $line_width = 1; $line_length =$tab_space*10-5; Basically, from the index.php page, the user has selected the search_type he wants to perform and on which_file, these are recovered and stored. Now I will be performing three separate operations using the search_type parameter to process the data. Also I assign some variables here which I would use for the full backup search. Ideally they should have been inside the search_type==1 condition, but there is no loss of functionality. Case 1 : Full Backup Search system("/usr/local/backup/bfind_full $search_string > /tmp/searchlist_full"); $file_full=file("/tmp/searchlist_full"); if (is_null($file_full[0])) echo "<Blink><font color=yellow> <b>no Matching Records Found: Please try again</b></font></blink><br>"; break; else 22

23 I now recover the search results using the bfind_full shell script and store the results to a temporary file which is subsequently recovered. If the file contents are null, this meants no search results are obtained and I display suitable error message. $i=0; while($file_full[$i]) $date=substr($file_full[$i],0,11); if ($date!= $date_previous) $date_previous=$date; //echo "<b>$date_previous</b><br>"; $disp[$date_previous][]=strchr($file_full[$i],"usr"); //echo "/home/$disp<br>"; else $disp[$date_previous][]=strchr($file_full[$i],"usr"); //$disp=strchr($file_full[$i],"usr"); //echo "/home/$disp<br>"; $i++; //echo "<pre>"; //print_r($disp); //echo "</pre>"; Now that I am sure that some search results have been obtained, I know have to parse them to display them in a format acceptable to me. So, I extract each line component and extract date. Since, I decided to sort on the basis of date, I form a two dimensional array whose first component is the date and the second onwards are the various files backed up on that date. This I will display in a appropriate manner using the following lines of code. Also I have commented the display lines above, on uncommenting them, one can see the array listing. The display is to be within the <pre> and </pre> tags. This ensures that the var_dump is displayed as is i.e. without the blank spaces being ignored. This gives a humanly understandable result. foreach ($disp as $date => $val) echo "<Blink><font color=yellow> <b>$date</b></font></blink><br>"; sort($val); foreach ($val as $key => $display)?><img <img src=images/red.bmp width=<?=$line_width?> height=30 align = middle hspace=0 vspace=0 border=0> <img src=images/red.bmp width=<?=$line_length?> height=<?=$line_width?> hspace=0 space=0 border=0 > <? echo "$display<br>"; system("rm -rf /tmp/searchlist_full"); Now I access this two dimensional array using two foreach statements. I also sort the entries (one a particular date) by their file names. This will make it easier to view. Also some <img> tags to make the result 23

24 look visually appealing. Finally, I discard the temporary file created (actually this could have been done once the array had formed) Case 2 :Incremental Backup Search $file_prev=""; system("/usr/local/backup/bfind_incr $search_string > /tmp/searchlist_incr"); system("./scripts/space_remove /tmp/searchlist_incr"); $file_incr=file("/tmp/searchlist_incr"); if (is_null($file_incr[0])) echo "<Blink><font color=yellow> <b>no Matching Records Found: Please try again</b></font></blink><br>"; I search the incremental logs using my bfind_incr shell script. The results if any are stored into an array and I now check whether some results have been obtained else display an error message. $i=0; while($file_incr[$i]) //echo "$file_incr[$i]<br>"; $date=substr($file_incr[$i],0,11); //echo "$date<br>"; $rest=strchr($file_incr[$i],"."); //echo "$rest<br>"; $tape_count=substr($rest,5,5); //echo "$tape_count<br>"; $rest2=strchr($rest,":-"); //print_r($rest2); $line_arr=explode(" ",$rest2); //print_r($line_array); // Since some entries in this array have null values we have to remove them if ($line_arr[5]!= $file_prv) $file_prv=$line_arr[5]; $disp[$file_prv][$date][]=$line_arr[0]; $disp[$file_prv][$date][]=$line_arr[1]; $disp[$file_prv][$date][]=$line_arr[2]; $disp[$file_prv][$date][]=$line_arr[3]; $disp[$file_prv][$date][]=$line_arr[4]; $disp[$file_prv][$date][]=$tape_count; else $disp[$line_arr[5]][$date][]=$line_arr[0]; $disp[$line_arr[5]][$date][]=$line_arr[1]; $disp[$line_arr[5]][$date][]=$line_arr[2]; $disp[$line_arr[5]][$date][]=$line_arr[3]; $disp[$line_arr[5]][$date][]=$line_arr[4]; $disp[$line_arr[5]][$date][]=$tape_count; $i++; //echo "<pre>"; //print_r($disp); //echo "</pre>"; 24

25 For incremental backup results, however my motivation is different. I am interested in both the date and filename. So I have formed a three dimensional array where the third dimension stores information about each entry viz permission [0], owner [1], size [2], date created [3], time created[4] and most significantly the tape+space_count [5]. The tape+space_count information is what would be required in a restore operation (not included till the time of writing of this report) besides the complete path of the file being recovered. Also I recover all this information on top using a rather complicated extraction scheme. This was because simple extraction didn t always give me the correct results. This type of extraction although complicated always gives me the right results. // Now displaying the array created foreach ($disp as $file_name => $in_arr) echo "<Blink><font color=yellow> <b>$file_name</b></font></blink><br>"; ksort($in_arr); foreach ($in_arr as $date => $components)?> <img src=images/red.bmp width=<?=$line_width?> height=30 align = middle hspace=0 vspace=0 border=0> <img src=images/red.bmp width=<?=$line_length?> height=<?=$line_width?> hspace=0 space=0 border=0 > <? echo "$date"; foreach ($components as $index => $value_of_component) for ($spc=0;$spc<=5;$spc++)?> <? echo "$value_of_component"; echo "<br>"; echo "<br>"; system("rm -rf /tmp/searchlist_incr"); Now that a three dimensional array has been created, we have to access its entries by atleast 3 for loops. This is what I have done above. Some lines in between are required to give it some visual appeal by introducing images and spacing. Case 3 : All Backup Search (Full + Incremental) By far this is the most complicated operation. Since I not only want the file name but also various versions of the same file (if they exist) in the incremental backups. Also, if a file doesn t exist in the full logs, it might exist in the incrementals. Else it might not exist I both. The code for this section is basically the same as in the above Cases 1 & 2. Ideally a function call should have been used. Since, I was comfortable with the code sections as above, I simply copied them again. However there is a slight change in accessing the incremental logs once a file search is successful in the full backups which I am quoting below. // Now that I have found out the version in full if any i will see if it is there in incrementals $out=array(); exec("/usr/local/backup/bfind_full_incr $display ",$out ); foreach ($out as $each_result) //$each_result = trim($each_result); $words = explode(" ",$each_result); //echo "<pre>"; //var_dump($words); 25

26 //echo "</pre>"; Here I have used the exec command to store the incremental search results directly onto an array $out. for ($spacing=0;$spacing<=3;$spacing++)?> <??> <img src=images/green.bmp width=<?=$line_width?> height=30 align = middle hspace=0 vspace=0 border=0> <img src=images/green.bmp width=<?=$line_length?> height=<?=$line_width?> hspace=0 space=0 border=0 > <? for ($s=0;$s<=5;$s++) echo $words[$s]; for ($spacing=0;$spacing<=5;$spacing++)?> <? echo "<br>"; Now after extracting the required parametere, I simply display it. I have not formed an array storing each parameter into an array because of additional complications involved. However the result displays the tape and space count in the first result. So to extract a certain version of the file, the user can simply load the right tape, use mt fsf <count mentioned -1 > to reach the right file. Now he has to untar that file from that backup component using tar xvf. 26

27 9. Future Enhancements This section is written specifically for someone who would be modifying my scripts in future. I am mentioning below the possible enhancements with the reasons why I chose not to implement them now. 1. File System Date/Time Check : If permission is granted, shell scripts could be written to automatically detect and change, dates/times of files in user file space to be coherent. That is forward and backward dated files should not be allowed 2. Exception : The --exclude option in the tar command was not used as there was no specific requirement. 3. Incremental using tar itself: The present script uses the find command to build a list of files to needed be backed up on incremental basis. The tar provides inbuilt facility to perform incremental s using self generated index files. tar G Old GNU format tar g New GNU format I did not perform this because of the following lacunae i) We are performing tape backups, essentially linear devices. However this seems to be essentially designed for Hard Disks ii) The documentation itself is very sketchy. Not many resources/clarifications on this were readily available. Mostly commercial solutions were used to implement such incremental s and very little information appeared available for autoloader s. In case clear documentation is available, this option can easily be tried. iii) This would generate its own index files which we cannot easily view or manipulate. Of course, a tar tvf after the backup is done, can be used to generate useful index files. But this would increase runtime critical for large backup volumes. 4. Data Log Simplification : I have used separate files where I store status of which_tape_loaded, current_no_of_bytes in loaded tape and which_space_count. All this information could be put into a single file to make it more professional. However, this would make data extraction slightly complicated. We can simply grep and cut the data from file based on space delimiters. However writing the data to file would be complicated. 27

28 10. References [1] Shell Scripting Hamish Whittal Published :36:19 Copyright 2004 The Shuttleworth Foundation [2] Implementing Network Backup Sanjay Majumdar PCQuest Magazine April 2006 [3] KnowledgeStorm - Protecting your Business with HP Proliant Data Protection Storage Servers powered by Microsoft System Center Data Protection Manager 2006 Whitepaper - Hewlett Packard Data Storage Published on : February 2006 [4] Backup Strategies & Implementation Debian GNU / Linux Bible Extract in Linux For You Volume: 04. Issue:03 May, 2006 [5] Project Report on Anupam Accounting Package BARC Neeraj Malhotra OCES 46 th Batch, Computer Science July, 2003 [6] PHP 5 For Dummies Janet Valade Wiley Publishing Inc 111 River Street Hoboken NJ [7] HTML & JavaScript for Visual Learners Visibooks, LLC, Richmond, VA [8] The Official AMANDA Documentation Stefan G. Weichinger and AMANDA Core Team [9] Bacula Manual Version (02 May 2006) Copyright , Kern Sibbald 28

29 APPENDIX A Technical data ANUPAM Specifications ANUPAM Series Technology INTEL i860 INTEL Pentium PCs ALPHA Interconnection Scheme Multibus-II ATM Switch Fast Ethernet 100 Mpbs (switch) Gigabit 1000 Mbps (switch) Scalable coherent Interface ( 2D/3D Torus) Components and Specifications 1. Nodes: 275 nos. of 1 U servers, each dual processor Xeons a. Compute nodes : 256 b. Service nodes : 7 c. Spare nodes : File servers: 12 nos. of 2 U servers, 3. Backup servers: 2 nos. of 5 U backup servers 4. Tape Libraries: 2 nos. of 3.2 TB tape libraries 5. Ethernet switch:. One 312 port Gigabit Ethernet switch a. One 48 port Gigabit Ethernet switch b. One 24 port Gigabit Ethernet switch c. Six 24 port Fast Ethernet switches 6. Terminal Servers: 22 nos. of 16 port terminal servers 7. PDUs: 46 nos. of 8 port switched rack PDUs 8. Racks: 14 nos of 42 U racks 9. UPS: One 7.5 KVA UPS, 7 nos. of 3 KVA UPS 10. KVM: Two nos. of 8 port KVM switches 11. Central Console comprising of: a) One Pentium-4 based PC b) 2x2 LCD Tiled Display 12. Terminals: 6 nos. of Pentium 4 based PCs Location The Ameya cluster is located in the New Computer Centre (Supercomputing Facility), first floor computer hall. The user terminals are located in the first floor user gallery (right side after entering). The user gallery is where we worked 29

30 Software OS : AT&T UNIX/LINUX/True UNIX Compilers : ABSOFT/INTEL/COMPAQ FORTRAN Parallel Library : PVM, MPI,ANULIB ANULIB Tools : PSIM, FFLOW, SYN, PRE, S_TRACE, ANU2MPI Other Tools: PBS Job Scheduler, Control Panel Benchmarked Sustained Performance: 72GLOPS for PIV based ANUPAM Supercomputer connected by SCI 30

31 Technical data - Tandberg SuperLoader LTO2 General Device Type Tape autoloader - LTO Ultrium ( Ultrium 2 ) Enclosure Type Built-in Devices Width Depth Height Weight Compatibility Rack-mountable - 2 U Status LCD, keypad 45 cm 75.5 cm 8.9 cm 24 kg PC APPENDIX B Storage Removable Capacity 200 GB (native) / 400 GB (compressed) Total Storage Capacity (Native) 3.2 TB Total Storage Capacity (Compressed) 6.4 TB Interface Type SCSI Data Transfer Rate 30 MBps (native) / 60 MBps (compressed) Average Seek / Access Time 52 sec Recording Standard Ultrium 2 Media Type LTO Ultrium Removable Media Capacity 16 Miscellaneous MTBF 250,000 hour(s) Power Type Power supply Voltage Required AC 120/230 V ( 50/60 Hz ) Manufacturer Warranty Service & Support Service & Support Details 3 years warranty Limited warranty - 3 years Limited warranty - 1 year - on-site Environmental Parameters Min Operating Temperature 10 C Max Operating Temperature 35 C The SuperLoader is scalable with an optional second tape cartridge module doubling the compressed storage capacity to 6.4 TeraByte. With only 2 height units in a standard 19-inch rack, users achieve an extremely high storage density, which helps them to reduce the cost of valuable space in the rack. 31

32 Backup Servers APPENDIX C Processor Dual Intel Xeon 3 GHz with 1 MB cache with 800 MHz FSB. Chipset Intel 7520 Motherboard Asus NCL-DS Memory 4 GB Registered ECC 333 MHz DDR SDRAM Hard Disk 2*250 GB SATA hard disks Network One PXE boot compliant on-board 1000BaseT Gigabit Ethernet Ports Display Adapter One on-board PCI display adapter DVD One bootable DVD RW drive DAT One 20/40 GB internal DAT drive DLT One 40/80 GB internal DLT drive Ports 1 Serial Port, 2 USB ports, 1 Ultra 160 SCSI LVD port Form Factor 5U Rack Mountable Chassis 5U Rack Mountable Asus chassis with rail kit and triple redundant AC power supplies Operating System RedHat Enterprise Linux 4.2 File Servers APPENDIX D Processor Dual Intel Xeon 3 GHz with 1 MB cache with 800 MHz FSB. Chipset Intel 7520 Motherboard Memory Hard Disk Network Display Adapter CDROM Ports Form Factor Chassis Asus NCL-DS 4 GB Registered ECC 333 MHz DDR SDRAM 6*300 GB Ultra320 SCSI hot pluggable hard disks in RAID5 configuration Three PXE boot compliant 1000BaseT Gigabit Ethernet Ports (2 on-board) One on-board PCI display adapter One bootable slim line CDROM drive 1 Serial Port, 2 USB ports 2U Rack Mountable 2U Rack Mountable Asus chassis with rail kit and dual redundant AC power supplies Operating System Scientific Linux 4.1 Disk Space There are 12 file servers named sagar01 to sagar12. Out of these, the first 9 servers have GB hard disks whereas the last three have 5 hard disks. The 6th hard disk of these three machines are 32

33 plugged into the server but are not used. They are kept as spares. Consequently, the total hard RAID space on the first 9 machines is 1.5 TB whereas on the last 3 machines it is 1.2 TB. In sagar01 to sagar09 machines, 1.2 TB is available to users and the rest are for the system and the administrators. This space is divided into GB partitions. On the last three machines, 1 TB is available to users (200*5) and the rest to the system. Partitions The following are the partitions on the hard disk of each server / 4000 MB /boot /tmp /usr /usr/local /usr/spool /var swap /padmin Volume Group 'users' 500 MB 4000 MB MB 8000 MB 4000 MB 4000 MB 8000 MB 60 or 150 GB 1 or 1.2 TB The Volume Group 'users' are further divided into logical volumes of 200 GB each. These volumes are named as 'usrxxy' where XX is the server number (01 to 12) and Y ranges from 0 to 4 or 5 depending on the server. For example, sagar04 contains LVs from usr040 to usr045, whereas sagar12 contains LVs from usr120 to usr