MetaData Management Control of Distributed Digital Objects using irods. Venkata Raviteja Vutukuri

Transcription

1 Abstract: MetaData Management Control of Distributed Digital Objects using irods Venkata Raviteja Vutukuri irods is a middleware mechanism which accomplishes high level control on diverse distributed digital objects, facilitates user-defined management policies across various data storage locations. It is a generic software application which can develop any data management application going from a data grid which is used for sharing data across collaborations, to a digital library for publishing data. irods provides an adaptable data management technique which allows administrators and users to customize their data based on different guidelines and micro-services. Introduction: irods (Integrated Rule Oriented Data Systems) is a data grid software system developed by the Data Intensive Cyber Environments research group (developers of the Storage Resource Broker (SRB)), and collaborators. The technology was based on knowledge gained through a decade of applying the SRB (Storage Resource Broker) technology in support of Data Grids, Digital Libraries, Persistent Archives, and Real-time Data Systems. The primary idea behind irods mechanism is to deliver a system that enables flexible, adaptive, customizable data management architecture. The ideas for the irods project have existed for quite some time, but became more concrete through an NSF-funded project titled Constraint-based Knowledge Systems for Grids, Digital Libraries, and Persistent Archives which started in fall of The development of irods is driven by the lessons learned from the deployment and use in production of the DICE Storage Resource Broker data grid technology SRB and through applications of theories and concepts from a wide range of well-known paradigms from other fields such as active databases, program verification, transactional systems, logic programming, business rule systems, constraintmanagement systems, workflows and service-oriented architecture. irods software belongs to a class of middleware which is termed as adaptive middleware. The adaptive middleware architecture (AMA for short) provides a means for adapting the middleware to meet the needs of the end users without making any programming changes. One can view the AMA middleware as a glass box where users can see how the system works and can tweak the controls to meet their demands. Normal middleware can be viewed as black box where no changes are programmatically possible to adjust the flow of the operations, except configuration changes that may allow one to set the starting conditions of the middleware. There are different ways for achieving adaptive middleware architecture. The irods architecture provides a means for encoding customization of data management functionalities in an easy and declarative fashion using the ROP paradigm. This is accomplished by coding the processes that are being performed in the irods data grid system as rules that explicitly control the operations that are being performed when a rule is invoked by a particular task. These operations are called as micro services in irods and C- functions that are called when executing the rule body. One can modify the flow of tasks when executing the rules, by intervening new micro-services (or rule-calls) in a given

2 rule or by changing and recompiling the micro-service code. Moreover, one can add another rule in the rule base for the same task, but with a higher priority so that it gets chosen before an existing rule. This preemptive rule will be executed before the original rule. If there is a failure in the execution of any part of this new rule then the original rule gets executed. 3) A Rule System for enforcing and executing adaptive rules. This is the following diagram for irods architecture. Architecture: irods architecture does provide a means of changing the data management functions with the help of ROP prototype in a declarative manner. This mechanism is achieved by writing code for the processes that are being developed in the irods Data Grid System which are called as Rules. These Rules explicitly control the operations which will be performed when any action is invoked by some job. Such operations are called as Micro-services which are C-functions that will be called when executing the rule-body. One can modify the flow of tasks when executing the Rules, by interposing new Micro-services (or Rule invocations) in a given Rule or by changing and recompiling the Micro-service code. Moreover, one can add another Rule in the Rule Base for the same task, but with a higher priority so that it is chosen before an existing Rule. This pre-emptive Rule will be executed before the original Rule. If there is a failure in the execution of any part of this new Rule then the original Rule is executed. irods has some of the major features which include the following. 1) Data grid architecture based on a client/server model and distributed storage and compute resources. 2) A Metadata Catalog for maintaining the attributes and states of data and operations. The irods server system is installed on each storage system where data will be stored. The remote location of the storage system is normally defined by an IP (Internet Protocol) network addresss. The irods Server translates operations into the protocol required by the remote storage system. In addition, a Rule Engine is also installed at each storage location. The Rule Engine controls operations performed at that site. The below figure illustrates the components of the irods system, including a Client for accessing the Data Grid, Data Grid Servers installed at each storage system, a Rule Engine installed at each storage location, the icat Metadata Catalog that stores the persistent state information, and a Rule Base that holds the Rules. The irods Data Grid uses persistent state information to record all attributes that are needed about a file, including the name of the file, the location of the file, the owner of the file, a file checksum, a data expiration date, and many others. More than 100 attributes are used by irods to manage information about each

3 file. The irods Data Grid Servers constitute a peer-to-peer server architecture, in which each server can exchange information with any other irods Server. When a user accesses an irods Server, information is exchanged between the irods Server and the server that hosts the metadata catalog (icat-enabled server). The user is authenticated, and the physical location where the user commands will be executed is identified. The users request is forwarded to the appropriate irods Server. The Rule Engine at that location then verifies whether the desired operations can be executed, translates the operations into the protocol required by that type of storage system, and passes the result of the operations back to the irods client. Any state information that is generated is registered into the icat metadata catalog. i-commands: UNIX like Commands: iinit :- This command will keep your password in a scrambled form for automatic use by other icommands. iput: - This command will store your file. iget :-Get a file imkdir: - This command will make an irods collection which is similar to a directory or Windows folder. ichmod:- This command is similar to chmod, which allows access to data objects by other users. icp:- This command is similar to cp or rcp, which copies an irods data object. irm:-this command is similar to rm which removes an irods data object. ils:- This command is similar to ls, which lists irods data objects (files) and collections(directories). ipwd:- This command is similar to pwd, which print the irods current working directory. icd:- This command is similar to cd, which changes the irods current working directory. irepl:- This command replicates the existing data objects. iexit:- This command helps the user to logout (use 'iexit full' to remove your scrambled password from the disk). ipasswd:- This command will help in changing the irods password. ichksum:- This command is used to calculate checksum for one or more data-object or collection from irods space. imv:-this commands moves/renames an irods data-object or collection. iphymv:-this command physically moves the files from irods space to another storage resource. ireg:- This command will register a file or a directory of files and subdirectory into irods. irmtrash:- This command will remove one or more data-object or collection from a RODS trash bin. irsync:- This command will synchronize the data between a local copy and the copy stored in irods or between two irods copies. itrim:- This command will trim down the number of replica of a file in irods by deleting some replicas. iexecmd:- This command will help in remotely executing (fork and exec) commands on the server. imcoll:- This command will manage (mount, unmount, synchronize and purge of cache)

4 mounted irods collections and the associated cache. ibun:- This command will upload and download structured (e.g. tar) files. iphybun:- This command helps the system admin to physically bundle files in a collection into a number of tar files. MetaData Commands: imeta:- This command will help the user to add, remove, list, or query user-defined Attribute-Value-Unit triplets metadata. isysmeta:-this command helps the user how to show or modify system metadata. iquest: This command helps the user to query (pose a question to) the ICAT, via a SQL-like interface. idbo:- This command helps the user to execute a Database Object (DBO) on a Database Resource (DBR), etc. All these commands are useful in interacting with the Data Grid Servers for various Information Retrieval operations. Clubbing Metadata Services Architecture into irods architecture: Metadata Database architecture should be generic and it should be accessible to all types of Metadata tools present in the World. With the help of introducing the generic server mechanism the latency values can be minimized at the time of Information retrieval and performing other i-command operations like iput,icp etc. Generic Metadata Services Architecture In Meta Data Services architecture, the tools and applications connect to the core engine and storage components through open standards. While the Information models define type information that determines the structure and behavior of metadata that is exposed by tools and applications at the top layer. There are several tools like Adaptive, Rochade which offer Metadata services to the Clients who require Data Services. By introducing the generic mechanism of Metadata services into irods system there can be an easy and efficient way of providing duties to the Clients when the big companies are dealing with magnanimous amounts of data. Summary: Data grids which are present at various locations can be accessed efficiently with the help of irods architecture provided with rules, micro-services and execution of work-flows on distributed databases. To minimize the latency values a generic metadata architecture can be introduced into the irods architecture for high performance and minimizing the efforts and time of Clients who utilize the facilities of irods.

5 References: 1.) Extracting and Ingesting DDI Metadata and Digital Objects from a Data Archive into the irods extension of the NARA TPAP using the OAI-PMH 2.) NITRD irods Demonstration Reagan W. Moore, Richard Marciano, Arcot Rajasekar, Antoine de Torcy, Chien-Yi Hou, Leesa Brieger, Jon Crabtree, Jewel Ward, Mason Chua, UNC Chapel Hill; Wayne Schroeder, Michael Wan, Sheau- Yen Chen, UCSD, sponsored by NARA at NSF, ) lications 4.) umentation#irods_clients_for_ma naging_data 5.) mmands#user_environment_for_ico mmands 6.) ata-tools/repository-architecturefeatures.php