GriPhyN-LIGO Prototype Draft Please send comments to Leila Meshkat

Transcription

1 Technical Report GriPhyN GriPhyN-LIGO Prototype Draft Please send comments to Leila Meshkat Kent Blackburn, Phil Ehrens, Albert Lazzarini, Roy Williams Caltech Ewa Deelman, Carl Kesselman, Gaurang Mehta, Leila Meshkat, Laura Pearlman USC/ISI Bruce Allen, Scott Koranda UWM 1

2 1 Prototype Overview Goal Resources used in Prototype Prototype Logic Replica Catalog Replica Catalog Structure Supported Virtual Data Requests User Interface LIGO-LW Syntax Data and metadata flow Virtual Data Request (VDR) Syntax and Semantics Time interval Channels Response Request Interpretation Security Explanation of Myproxy Planner Planner Overview User request Replica Catalog Query Planning Scenarios Scenario 1: Data is Available at User-requested location Scenario 2: Data is Available but not at the User-requested location Scenario 3: User-requested data is unavailable Scenario 3.1: Full frame is located at UWM Scenario 3.2: Full frame is located at ISI: Scenario 3.3: The full frame is available at both ISI and UWM Scenario 4: Full frame does not exist Executor Conclusions and Future Directions References

3 1 Prototype Overview 1.1 Goal The goal of this prototype is two-fold: first, to develop initial prototypes for the Virtual Data Toolkit in support of the Virtual Data concepts, and second to provide a scientifically meaningful, grid-enabled environment for the LIGO collaboration. This prototype was demonstrated at the SC 2001 in November. MyProxy server xml Cgi interface HTTP frontend Frame P. Charlton Desired Result : Transformation Catalog Power spectrum of seismic channels Planner Monitoring Replica Catalog GridCVS Replica Selection G-DAG (DAGMan) Prototype exclusive Executor CondorG/ DAGMan Logs In design Globus component In integration GridFTP GRAM/LDAS GridFTP GridFTP GRAM GridFTP GRAM/LDAS LDAS at UWM UWM SC floor Compute Resource LDAS at Caltech Figure 1: Prototype Overview Figure 1 depicts the overall logic and design of the prototype. The user makes a request for data via a GUI, this request is translated into an XML syntax and is used as input for the Request Manager (RM). The RM surveys the available data and compute resources and makes a plan to satisfy the request. The plan is specified in the DAGMan format and presented to the Condor-G system [1] for execution. Condor-G executes the plan by 3

4 accessing various storage and compute resources. The resulting data is moved to the desired, user-specified location and registered in the replica catalog. 1.2 Resources used in Prototype The LIGO prototype resources which are shown at the bottom of figure 1 and in figure 2 include: Storage Resources: workstation at Caltech, 21 terabytes of commodity disk on the UWM Linux cluster, cluster storage at ISI. Compute Resources: LDAS at Caltech, LDAS at UWM, Planner and job submission server using Condor-G at ISI Replica Catalog: An LDAP (Light Weight Directory Access Protocol) on cluster front end node (dc-user@isi.edu) at ISI: ldap://dc-user.isi.edu:9129 Web Server: apache on dc-user@isi.edu at ISI UW M ISI Replica Catalog Caltech LDAS LDAS Storage Resource Storage Resource Storage Resource Figure 2: Existing Resources for the prototype 2 Prototype Logic The user makes a request for virtual data via a web portal. The user is able to specify the virtual channel name, the start time, where the resulting data should be placed and the format of the resulting data. The format of the data can be XML (specifically XSIL[2]), or Frame, which is a standard data format for the gravitational wave community. The request is translated into XML and is passed to the Request Manager (RM). 4

5 The Request Manager interprets the requests and constructs a DAG that fully represents the computations and data movements necessary to satisfy the request. Some of the subtasks of the RM are: Authenticate user. Make an optimal plan for satisfying the user s request using the available resources. (The planner is described in section 6) Map request into a sequence of grid operations that need to be performed. Submit the plan to Condor-G. Return the status of the request and the location of the data to the user. The RM sends the DAG to Condor-G. The DAG can contain LDAS commands, gridftp[11] commands, arbitrary analysis pipeline commands, updates to the Replica Catalog (RC) and other local commands such as FrCopy (not implemented yet) Condor-G executes the specified data movements, submits jobs to LDAS via a GRAM (Globus Resource Allocation Manager) [5] interface, and updates the replica catalog. Details about the DAG can be found in section 7. LDAS or another analysis system perform computation. Data is moved to the user-specified location and the user is then notified of the request s completion and the updates performed to the catalog. 3 Replica Catalog 3.1 Replica Catalog Structure Ou=isi.edu, o=netscaperoot Rc=Ligo_Replica_Catalog lc= lc= lc= lc= Caltech UWM Caltech UWM Caltech UWM Caltech UWM ISI ISI ISI ISI :Raw and materialized data :Materialized data only Figure 3: Replica Catalog Structure 5

6 The Replica Catalog maintains mappings from logical file names to physical file locations. The logical file names are grouped into logical collections. When trying to find the physical location of a file, the collection name and logical file name need to be provided. Following is the structure for the catalog: 1. Collections are based on the instrument name and time interval, the instruments are: LO and HO and H1. 2. The time interval for a collection is 200 seconds. For instance a collection might contain files [ ]. This is shown in figure Each collection will contain frame files of 1 second in duration, both full channel and single channel frames. The data is located at: workstation at Caltech, storage cluster at ISI, storage cluster at UWM. We have four collections, each containing a 200 second time intervals of data. The distribution of the raw and materialized data for each of these collections is shown in figure 2. The acronym lc is used to denote the logical collections. 4 Supported Virtual Data Requests For the purpose of the prototype, we decided to focus on a raw data set that represents the Hanford system in lock. We identified the locked interferometer (IFO) data as being present in run E2 and being composed of 1800 to 2000 seconds of full frame (raw) data. Several raw channels have been identified as "interesting": H2:LSC-AS_Q H2:LSC-AS_DC H2:PSL-FSS_FAST_F H0:PEM-LVEA_SEISX H0:PEM-LVEA_SEISY H0:PEM-LVEA_SEISZ H0:ALL H2:ALL The names are determined based on the instrument used for collecting the data (H0, H2), and the channel names (LSC-AS_Q, LSC-AS_DC, ) ALL is a valid channel name used to specify the entire frame file. For the purposes of this prototype, we have used the data collected in the time interval [ ] and classified it into four collections, each consisting of 200 seconds of data. The user uses a web based portal to make a request. The web page where the user makes the request is shown in figure 3. The input parameters include the channel name, the time in GPS, the output data location and the return data type. 6

7 Figure 4: User request web page 5 User Interface 5.1 LIGO-LW Syntax We used the LIGO-LW as the syntax for formatting the request that the user makes to the system and the response that the user receives from the system. The XML is built from the LIGO-LW dialect, which is essentially the same as the XSIL dialect [2][4]. LIGO-LW has a small number of elements, and each of them can have a "Name" and a "Type" attribute. The ones we use are mostly these: 1. <Param> has the semantics of a keyword-value pair. To express "x=3" we would say <Param Name="x">3</Param>. 2. <LIGO-LW> makes a collection. By nesting these elements, we can make a hierarchy. There is a special meaning to the "Type" attribute of this element, as this is used to extend the language. We could build a date extension to LIGO-LW as follows: 7

8 <LIGO-LW Type="Date"> <Param Name="Day">11</Param> <Param Name="Month">8</Param> <Param Name="Year">2001</Param> </LIGO-LW> When the parsing software gets the Type of a LIGO-LW element, it expects to build an object of that type from the enclosed elements Data and metadata flow Initially, a JobID number is allocated to the user s request. As the request is being processed, the user receives feedback as to the location of the requested data files and the various transfer and transforms that are being conducted on it. The various scenarios that can occur are discussed in section 7. When processing is complete, the status request gives a final status response (FSR) indicating whether or not the user s request has been satisfied and the output location of the desired data files. The JobID allocated to a request becomes invalid after the request has been processed completely. 5.2 Virtual Data Request (VDR) Syntax and Semantics The request is in XML 1.0 format, with the standard LIGO-LW DTD (Document Type Definition). Thus the first two lines are: <?xml version="1.0"?> <!DOCTYPE LIGO-LW SYSTEM "LIGO- LW.dtd"> A VDR begins with a LIGO-LW element of type VirtualDataRequest, like this: <LIGO_LW Name="banana" Type="VirtualDataRequest"> There may be other LIGO-LW elements of different types in the document, which will be ignored by the VDR parser. There may be multiple VDR's in the XML document, which are parsed in an arbitrary order. The system connects the job-id s with the name attributes of the VDR to provide a clear mapping between the ID s and the VDR s. 8

9 Semantically, there are four attributes to the VDR: channel name, timestamp, output location (including output filename) and output format. Each of these attributes is unique. The following sections list the element types that are relevant to the VDR Time interval This is part of the specification of what data is to be retrieved from the data system. It is expressed by two of the following three elements being present. Note that <Time> is a base type in LIGO-LW <Time Name="StartTime" Type="GPS"> </Time> The default type of the Time element is GPS (seconds since January ), but other formats are possible, as detailed in the LIGO-LW specifications cited above Channels The list of channels to be requested from the data archive comprises both channel names and the interferometers at which the channels were recorderd. This is specified by a LIGO-LW extension of type "ChannelSpecification". Currently we will assume that each interferometer records the same set of channels, each with the same name, frequency, etc. The data to be obtained is a Cartesian product of a list of interferometers with a list of channels. The list of interferometers is specified by a comma-separated list of names, and the channels are fully specified. If multiple channels are requested, each channel name has to be listed For example: <LIGO_LW Type="ChannelSpecification"> <Param Name="Detector">LHO,LLO</Param> <Param Name="RegEx">H2%3ALSC- AS_Q</Param> </LIGO_LW> Response The semantic content of this section of the VDR is currently: What format the response data should be, The location in the grid where the data should be put, Hints about caching, expiration, etc.(to be included in the future) This information is specified in a LIGO-LW extension of type "ResponseSpecification": <LIGO-LW Type="ResponseSpecification"> 9

10 This may contain a Param of type "ResponseFormat", and currently it can have the values "LIGO-LW" and "Frame", for example <Param Name="ResponseFormat">Frame</Param> If the requested response format is "LIGO-LW", then data may be returned with the TimeSeries extension of LIGO-LW, as specified in the papers cited above. This response can be seen using the Xlook viewer [1]. If the requested format is "Frame", then the FSR will still be a LIGO-LW formatted document, which includes a citation to the location of the requested Frame file on the grid. The response location, if present, may be specified as a Param, for example: <Param Name="ResponseLocation"> gridftp://dataserver.phys.uwm.edu/albert/myfile.fra me </Param> <Param Name= ResponseDomain > uwm.edu</param> This location is specified in URL format, but the resources must be writeable by the user, by using gridftp. 6 Request Interpretation Initially, the Request Manager provides a job ID to the user and logs the full XML request in its local log file. Following is an example XML file of a user s request: <!DOCTYPE LIGO-LW SYSTEM "LIGO-LW.dtd"> <LIGO_LW Name="banana" Type="VirtualDataRequest"> <Time Name="StartTime" Type="GPS"> </Time> <Time Name="EndTime" Type="GPS"> </Time> <LIGO_LW Type="ChannelSpecification"> <Param Name="Detector">LHO,LLO</Param> <Param Name="RegEx">H2%3ALSC-AS_Q</Param> </LIGO_LW> <Param Name="ResponseDomain"> isi.edu </Param> <Param Name="ResponseFormat"> xml 10

11 The user request is then mapped into a logical file name. This is done by concatenating the channel name and the timestamp. For example the logical file name corresponding to the request for the channel H2: LSC-AS-Q and the timestamp would be H2:LSC-AS-Q F. Next, the Request Manager will ask the user (via an ssl connection) for the user s name and password. This information will be used to access the MyProxy server where the user s certificate resides. The steps used to setup the MyProxy server and use it are described in the next section. 7 Security The security in the prototype is implemented using Globus-GSI [3] (Grid Security Infrastructure), which is a means of authenticating using X509_certificates and a MyProxy [4] Server. The MyProxy [4] server takes a standard Globus [3] proxy certificate and stores it in its cache with added credentials to it. The server can then at runtime upon request and by providing the correct credentials create a delegated proxy certificate, which can give limited access to the Grid resources. 7.1 Explanation of Myproxy The Myproxy package provides a secure method for portal users to access resources using a limited proxy using the GSI. As part of the package, a myproxy-server is set up on a trusted host for a site or application specific portal in order to maintain delegated credentials for users that are valid for a chosen duration. After delegating a credential to the myproxy-server, users can retrieve a delegated credential from the myproxy-server via the web server that can be used for job submission, file transfer, etc. In the event that the web server is broken into, only the time limited delegated credentials are compromised so the hacker would potentially have access to other resources for a short amount of time. The GSI also includes provisions for Certificate Revocation Lists (CRL) as a means to deny users access. The following diagram helps to illustrate how the Myproxy package works: Figure 5: Myproxy package 11

12 The user runs myproxy-init and provides their GSI passphrase to create a proxy. The user enters an additional password (that will later be used in the browser) of their own choosing to delegate a proxy to the myproxy-server. Path A uses the GSI to perform mutual authentication between the client and the myproxy-server. Using any web browser capable of HTTPS, the user enters the password in (1) to get a proxy from the myproxy-server. The myproxy-server does further delegation making the proxy that's used on the web server valid for less than the original lifetime on the myproxy-server. Path B uses the GSI to perform mutual authentication between the web server and the myproxy-server using either the web server's credentials (typically a Verisign or Thawte certificate) or a Globus certificate 8 Planner The Planner determines where on the Grid the relevant data is stored, where the best place is to do the computing and how to route the output to the user requested location. The planner may not necessarily start with raw data: the requested product may be already partially or completely computed. The data movement and computing jobs are controlled by the Condor software, a fault tolerant execution environment for collections of tasks with specified dependencies. Connection to the LDAS systems and data fetching is controlled by the Grid security. 8.1 Planner Overview The function of the planner is to find an optimal plan for satisfying the user s request using the available resources. The plan would depend on the location of the requested data files and the operations that need to be conducted on them. For the purpose of this prototype, we assume that the possible locations are ISI, UWM and Caltech. The user requested location can be any of the above locations and the possible operations include copying files, extracting channels from full frames and updating the replica catalog. 8.2 User request The user of the LIGO-GriPhyN prototype makes a request for virtual data via a web portal. The virtual channel name, the start time and the location where the resulting data should be placed is also specified by the user. Following is an example user request that the prototype uses for demonstration purposes. Channel Name: H2:LSC-AS_Q Timestamp: Output Location: (Y) ISI: dc-n1.isi.edu/output/ligo Data is copied in the collection ligo Output File name: H-H2:LSC-AS_Q F (internal to catalog) 12

13 8.3 Replica Catalog Query Upon the receipt of the user request, the planner needs to query the replica catalog to find the location of the data. In order to identify the collection that needs to be searched, we can use the following algorithm: If we want the data for time instant TIC Let X=TIC Mod 200; Y=TIC X The data is in the collection with starting point Y. Therefore the logical collection in question has the data for the times [Y-Y+199] 8.4 Planning An overview of the plan used for satisfying the users request is shown in figure 5. The information contained in the user request includes the collection name, the channel name, the full frame file name and the time interval. The replica catalog is queried for the file identified with this information. If the file is available at the user specified location, the user will be pointed to the exact physical location of the file. If the file is available but at a different location, a copy of it will be transferred to the user-specified location and the user will be given a pointer to it. If, on the other hand, the file is unavailable, the replica catalog needs to be queried for the full frame file, and the user-requested channel needs to be extracted from the full frame file and a sent to the user-requested location. The replica catalog is updated every time a file is copied to a new location. We shall elaborate on the plan for satisfying the user s request under each of these conditions in section

14 Collection name Channel name Filename.F Time interval Query Replica Catalog File not found File found at location X Query Replica Catalog for Filename.f File found at location X Error: File Does not exist X= Y* X!= Y* X= ISI X= UWM X=ISI & UWM Scenario 1 Scenario 2 Y=ISI Y=UWM * Y=User requested location Scenario 3.1 Scenario 3.2 Scenario 3.1 Scenario 3.2 Figure 6: Plan Overview 14

15 8.5 Scenarios Scenario 1: Data is Available at User-requested location In this case we proceed as follow: Plan for Scenario 1 Query Replica Catalog X=Y (output location is same as user requested - location) Point user to channel ( at the location) Example request is available at ISI. In this case, the user is given following pointers: Scenario 2: Data is Available but not at the User-requested location In this case, the data needs to be copied to the user-specified location and the replica catalog needs to be updated with the new location. After querying the replica catalog for the data and learning about the location of the existing data (Y), the plan is specified in a DAG and submitted to DAGMan for further execution. The plan for scenario 2 would be as follow: 15

16 Plan for scenario 2 Query Replica Catalog Data is available at location Y!=X Make a plan and submit to DAGMAN: Copy from X to Y Update Replica Catalog With location Y Point user to channel In this case, the requested data is available at the storage facility at UWM and therefore needs to be copied to the desired directory at ISI. The replica catalog then needs to be updated with the information about the new storage location path at ISI. The concrete form of the DAG would be as follow Scenario 2 DAGMan(Concrete form) globus_url_copy H-H2:LSC-AS_Q F From dataserver.phys.uwm.edu/ To dc-n1.isi.edu/output/ligo globus-replica -catalog -host dc-n1.isi.edu/output/ ligo logicalfile H-H2:LSC-AS_Q F -location ISI 16

17 The globus-url-copy command indicates that the data in the source file should be copied to the destination file. The globus-url-copy command in DAGman format would have the following information in it: #Command: executable = path/globus-url-copy arguments = #Source: gsiftp:// dataserver.phys.uwm.edu/grid_outgoing/h- H2:LSC- AS_Q F #Destination: gsiftp://dc-n1.isi.edu/output/h-h2:lsc-as_q f The globus-replica-catalog command indicates that the host computer has obtained a copy of the logical file name specified at the given location. It would have the following format: #Command: executable = path/globus-replica-catalog arguments = -host dc-n1.isi.edu/output/ligo logicalfile H-H2:LSC-AS_Q F -location ISI Scenario 3: User-requested data is unavailable Scenario 3.1: Full frame is located at UWM In this case, the initial query to the Replica Catalog does not result in a match. This indicates that the requested channel has not been extracted from the original full frame file. Therefore, there should be a new query to identify the location of the full frame file. Once the location of the full frame file is identified, it should be sent to the nearest LDAS system for extraction of the user specified file. We assume that geographic proximity is the criteria for selecting the LDAS to which the full frame must be sent for extraction purposes. The plan for scenario 3.1 is as follow: 17

18 Plan for scenario 3.1 Query Replica Catalog Data is unavailable Look into Replica Catalog to locate full fram at X. If X== UWM then Send frame file to the LDAS system at UWM. Extract the requested channel Send the channel to the storage system Update replica catalog Point user to channel Following is the abstract DAGMan that is designed for the execution of the plan: Send file to LDAS At UWM Abstract DAGMan Extract requested channel Send Channel to storage At Y The concrete form of this DAGMan is as follow: Update Replica Catalog With Y 18

19 globus_url_copy H: F From dataserver.phys.uwm.edu/ To dataserver.phys.uwm.edu/ LDAS_extract H-H2:LSC-AS_Q F From dataserver.phys.uwm.edu/ To dataserver.phys.uwm.edu/ Concrete DAGman globus_url_copy H-H2:LSC-AS_Q F From dataserver.phys.uwm.edu/ To dc-n1.isi.edu/output/ligo globus -replica -catalog -host dc-n1.isi.edu/output/ ligo logicalfile H -H2:LSC-AS_Q F -location ISI The globus-url-copy command indicates that the full frame file needs to be copied from the storage at UWM to the LDAS system there. #Command: executable = path/globus-url-copy arguments = #Source: gsiftp:// dataserver.phys.uwm.edu/griphyn_test/grid_outgoing/h: F #Destination: gsiftp://dataserver.phys.uwm.edu/griphyn_test/ldas/ldas-outgoing/jobs/grid_incoming/h: F The LDAS-extract command indicates that the user-requested channel must be extracted from the full frame file. #Command: globusrun -o r #From: dataserver.phys.uwm.edu/jobmanager-ldas 19

20 executable=/opt/gldas/bin/extract arguments= --start_time: end_time: channel H2:LSC-AS_Q --input_frame: H F -output_frame:h-h2:lsc-as_q f -frameformat= frame or xml The next globus-url-copy command indicates that the extracted channel must be sent to the user requested location #Command: executable = path/globus-url-copy arguments = #Source: Gsiftp://dataserver.phys.uwm.edu/griphyn_test/LDAS/ldas-outgoing/H-H2:LSC-AS_Q F #Destination: gsiftp://dc-n1.isi.edu/output/ H-H2:LSC-AS_Q F The replica catalog then needs to be updated with the information about the extracted file and it s location.: Command: executable = path/globus-replica-catalog arguments = -host dc-n1.isi.edu/output/ ligo logicalfile H-H2:LSC-AS_Q F -location ISI Scenario 3.2: Full frame is located at ISI: In this case, because of the proximity of Caltech to ISI, the LDAS at Caltech must be used for extraction purposes. The plan for satisfying the user request would be as follow: 20

21 Plan for Scenario 3.2 If X == ISI then Send X to the LDAS system at Caltech. Extract the requested channel Send channel to the storage system at u requested location Y Update replica catalog at user requested location Y Point user to channel. The abstract form of the DAGMan would look like this: Copy frame file to LDAS At Caltech Extract requested channel Abstract DAGMan Copy Channel to storage At Y Update Replica Catalog The concrete DAGMan would be as follow: 21

22 globus_url_copy H:Q F From dataserver.phys.caltech.edu/ To dataserver.phys.caltech.edu/ LDAS_extract H-H2:LSC-AS_Q F From dataserver.phys.caltech.edu/ To dataserver.phys.caltech.edu/ Concrete DAGman globus_url_copy H-H2:LSC-AS_Q F From dataserver.phys.caltech.edu/ To dc-n1.isi.edu/output/ligo globus -replica -catalog -host dc-n1.isi.edu/output/ ligo logicalfile H -H2:LSC-AS_Q F -location ISI The details of the commands specified in the concrete DAGMan are similar to those of scenario 3.1. The only difference is that the extraction process takes place at the LDAS system at Caltech instead of UWM Scenario 3.3: The full frame is available at both ISI and UWM In this case we consider the user-requested output location. If the user wants the data at ISI, then we send the full frame file to Caltech for extraction and the extracted data from Caltech to ISI (as in scenario 3.1). If, however, the user wants the data at UWM, then the data is sent to UWM for extraction (as in scenario 3.2). The plan for satisfying the user s request in this case is as follow: 22

23 Scenario 3.3 X==UWM and X==ISI If Y=ISI Go to Scenario 3.1 If Y=UWM Go to Scenario Scenario 4: Full frame does not exist In this case the user should be issued a warning message. The plan would be as follow: Scenario 4 Frame file does not exist Send following message: DATA DOES NOT EXIST Note that the current intelligence as well as the geographical cost model in the planner is hardwired. 9 Executor Once the submit files and the DAG are generated they are submitted to a Condor-G server which starts a DAGMan [5] which analyses the DAG and then submits jobs in the manner in which they are defined in the DAG. These jobs may run locally or on a condor pool (e.g. the third party data transfers) or via a GRAM interface to LDAS ( the transform jobs, which materialize the data). As the jobs in the plan finish they write the exit status of the job to a log file, which is parsed continuously by the planner. If any of the jobs in the plan fail then the execution of the job halts and the error is reported to the user via the web page. Condor-G has a restart facility available by which a job can be resubmitted and it can continue from where it stopped but this feature is currently not being used. If all the jobs in a plan succeed then the output file is generated and the replica catalog is updated to reflect the availability of the data at the new location. 23

24 The user then has the option to view the data in the browser or use the saved data. Following is an example of the output the user sees on the computer screen while the request is being processed: ************* Welcome leila *************** *************The JOB ID = 15557************ *******output type is = LIGO-LWD******** The desired channel name is H2:LSC-AS_Q with timestamp globus-replica-catalog -host "ldap://dc-user.isi.edu:9129/lc=ligo_ _ , rc=ligo_replica_catalog, ou=isi.edu, o=netscaperoot" -password iplanet.passwd -collection -findlocations./ligodemo/replica/search_list attributes uc >./LIGODEMO/replica/loc_list Searching the Replica Catalog for an existing file...completed The Requested file doesnt exist. Will materialize it. globus-replica-catalog -host "ldap://dc-user.isi.edu:9129/lc=ligo_ _ , rc=ligo_replica_catalog, ou=isi.edu, o=netscaperoot" -password iplanet.passwd -collection -findlocations./ligodemo/replica/search_list attributes uc >./LIGODEMO/replica/loc_list Searching the Replica Catalog for an existing file...completed Raw data is available at ISI Scenario 3.1 The Collection name is ligo_ _ The LDAS jobmanager is ldas-grid.ligo.caltech.edu:2119/jobmanager-ldas Writing sub file transfer_a2b_ sub...successful ************************WILL TRANSFER H F****************** from gsiftp://birdie.isi.edu/~/griphyn/ligodata/frames/h f to gsiftp://ldasgrid.ligo.caltech.edu/~/../../opt/gsiftp_root/gsiftp_root/griphyn_test/ldas/ldas_outgoing/jobs/grid_in coming/h f 24

25 My infile is H F, my outfile is H-H2:LSC-AS_Q xml15557 Writing Sub File transform sub...successful ****************Will apply Transformation ******************* Writing sub file transfer_a2b_ sub...successful **********************WILL GET RESULT H-H2:LSC-AS_Q xml *********************** from gsiftp://ldasgrid.ligo.caltech.edu/~/../../opt/gsiftp_root/gsiftp_root/griphyn_test/ldas/ldas_outgoing/jobs/grid_ou tgoing/h-h2:lsc-as_q xml15557 to gsiftp://birdie.isi.edu/~/griphyn/ligodata/frames/h-h2:lsc-as_q xml Writing Dag File ldas dag...successful Writing CHECK TRANSFER File CheckTransfer dag...Successful Writing Transfer File CheckExtract dag...Successful Checking your DAG input file and all submit files it references. This might take a while... Done File for submitting this DAG to Condor :./LIGODEMO/ldas dag.condor.sub Log of DAGMan debugging messages :./LIGODEMO/ldas dag.dagman.out Log of Condor library debug messages :./LIGODEMO/ldas dag.lib.out Log of the life of condor_dagman itself :./LIGODEMO/ldas dag.dagman.log Condor Log file for all jobs of this DAG :./LIGODEMO/logs/15557.log Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster Submitting Dag to Condor...Successful ************JOB SUBMITTED************* globus-replica-catalog -host "ldap://dc-user.isi.edu:9129/lc=ligo_ _ , rc=ligo_replica_catalog, ou=isi.edu, o=netscaperoot" -manager "cn=directory Manager, ou = isi.edu, o = NetscapeRoot" -password iplanet.passwd -collection -add-filenames./ligodemo/replica/update_list add-anyway 2>error1 Updating Collection ligo_ _ for file H-H2:LSC-AS_Q xml...SUCCESS globus-replica-catalog -host "ldap://dc-user.isi.edu:9129/lc=ligo_ _ , rc=ligo_replica_catalog, ou=isi.edu, o=netscaperoot" -manager "cn=directory Manager, ou = isi.edu, o = NetscapeRoot" -password iplanet.passwd -location ISI -add-filenames./ligodemo/replica/update_list add-anyway 2>error2 Updating Location for ISI file H-H2:LSC-AS_Q xml...SUCCESS *********The data is available at ************ 25

26 gsiftp://birdie.isi.edu/~/griphyn/ligodata/frames/h-h2:lsc-as_q xml To view the data through the browser click on the link VIEW OUTPUT 10 Conclusions and Future Directions The main objective of the prototype described in this paper is to demonstrate the feasibility of integrating the various LIGO data and computing resources via an underlying Globus infrastructure. Moreover, the security is provided based on the Grid Security Infrastructure (GSI). This prototype is based on the Virtual Data concept: the idea that users need not know where their data is stored, combined with the idea that users need not know whether data has been computed yet. There are services to discover what it will cost to obtain data and the quality of derived data. The Virtual Data Grid (VDG) will serve requests efficiently, subject to global and policy constraints, and with strong security. We envision increasing the sophistication of the planning mechanism, the amount of data available, and the number of channels in the future. In addition, more transformations will be supported and eventually the prototype will be extended into a tool. 11 References [1] Condor-G: A Computation Management Agent for Multi-Institutional Grids. J. Frey, T. Tannenbaum, M. Livny, I. Foster, S. Tuecke, Proceedings of the Tenth International Symposium on High Performance Distributed Computing (HPDC-10), IEEE Press, August 2001 [2] XSIL: Java/XML for Scientists by Roy Williams, California Institute of Technology; [3] [4] [5] [6] [7] [8] 26

27 [9] K. Blackburn, A. Lazzarini, P. Ehrens, P. Shawhan, R. Williams, E. Deelman, C. Kesselman, G. Mehta, L. Meshkat, L. Pearlman, B. Allen, S. Koranda. GriPhyN and LIGO: Realizing Virtual Data Concepts- SC2001 Denver Nov 2001 [10] [11] Ewa Deelman, Carl Kesselman, Gaurang Mehta. Transformation Catalog Design for GriPhyN Oct