EMC InfoArchive. Configuration & Administration User Guide. Version 4.1

Transcription

1 EMC InfoArchive Version 4.1 Configuration & Administration User Guide EMC Corporation Corporate Headquarters Hopkinton, MA

2 Legal Notice Copyright 2016 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. Adobe and Adobe PDF Library are trademarks or registered trademarks of Adobe Systems Inc. in the U.S. and other countries. All other trademarks used herein are the property of their respective owners. Documentation Feedback Your opinion matters. We want to hear from you regarding our product documentation. If you have feedback about how we can make our documentation better or easier to use, please send us your feedback directly at ECD.Documentation.Feedback@emc.com

3 Table of Contents Revision History... 7 Chapter 1 Overview... 9 Backup Versus Archive Modes of Archiving: When to Use Table Archiving or SIP Archiving Step 1: Setting Archiving Goals Step 2: Understanding the Source Application Step 3: Selecting the Appropriate Archiving Method Table Archiving Data Archiving File Archiving Compound-Record Archiving A Solution for Structured and Unstructured Content Submission Information Packages InfoArchive Information Model Generating SIPs Use Cases Cost Take-Out: Application Decommissioning Optimize: Active Application Archiving Information Transformation and Reuse User Roles Inline Users Troubleshooting Branding Customization Configuration to Add Sample Branding Customization Configuration to Add New Branding Customization Setting the Customization Location when Deploying to Tomcat or Other Containers Verifying and Viewing the Branding Customization Troubleshooting Chapter 2 Connectors Application and Platform Examples Chapter 3 Architecture Ingestion What is ETL? How to Extract Data Storage How Data is Searched Forms Search Results Chapter 4 Core Configuration System and Audit Database

4 Table of Contents xdb Federations and xdb Databases File System Root Space Root xdb Library and Space Root Folder Language Support Adding New Language Support Backing Up and Restoring of the Managed Item Database Configuring the Managed Items Database Backing Up the Managed Items Database Setting the Managed Item Store XdbLibrary Restoring the Managed Database Using the xdblibrary Selected Managed Items Using the Managed Items Chapter 5 Creating a Search Creating a Search for a SIP Archive Application Updating a Search Template Status to Ready Creating a Search for a Table Archive Application Editing a Search Deleting a Search Configuring Search Form Fields Configuring a Result Column Creating a Duplicate Search Table-Based Search XForms, XQueries and Query Results Example: First Name and Last Name with External Variables Incomplete Forms Nested Searches Authorization Multiple XQueries/XForms per Search Using ANT Tasks to View, Create, Delete or Update XQuery Modules Background Searches Order Using the Background Results Tab Deleting a Background Task Exporting Search Results Configuring Export Functionality in the InfoArchive Web Application Composition (Edit) Mode Installing the Export Functionality Backward Compatibility Adding the Ability to Export Search Results Configuring Export Objects Configuring Application-Specific Export Objects using ANT Scripts Configuring EP, EC and ET Objects using IAShell Configuration Export Through ANT Chapter 6 Compliance Terminology Retention Policies Retention Policy Types

5 Table of Contents Determining the Best Method to Apply Retention Applying Multiple Retention Policies Record-Based Retention Hold Management The Purge Process What Happens if a Purge Candidate List is Rejected? Disposition of a SIP-Based Application Disposition of a Table-Based Application Retention Rules on the Disposition of AIPs, AIUs, Applications and Rows Table Archiving Application SIP Archiving Using the Retention Sets Tab Viewing Items in a Retained Set Using the Hold Sets Tab Viewing Items in a Hold Set Removing an Item from a Hold Set Troubleshooting Using the Purge Lists Tab Performing Actions to a Purge List Using the Application Info Tab Applying a Retention Policy to an Application Removing Retention from an Application Applying a Hold to an Application Removing a Hold Set from an Application Using the Packages Tab Retention and ECS Storage Applying Actions to a Package Applying a Retention Policy to an AIP Applying a Hold to an AIP Rejecting or Invalidating an AIP Using the Retention Policies Tab Creating a Retention Policy Editing a Retention Policy Deleting a Retention Policy Event-Based Retention Using the Holds Tab Creating a Hold Editing a Hold Deleting a Hold Chapter 7 Authentication and Authorization Authentication Active Directory Integration User Roles and User Groups Authorization Using the Groups Tab Using the Permissions Tab Chapter 8 Administration Using the Application Page Creating an Application Editing an Application

6 Table of Contents Deleting an Application Deleting Data from an Application Working with Jobs: List of Available Jobs Using the Apply Retention Policy to Records Job Using the Archive Audits Job Using the Clean Up Purge Candidate List Job Using the Requalification Job Using the Refresh Metrics Job Using the Remove Policy Job Using the Trigger Event Policy Job Populating Event Dates for the Trigger Event Policy Job Close Clean Table Data Volume Update Using the Jobs Tab Viewing a Job s Run History Creating a Job Editing a Job Running a Job Suspending a Job Using the Storage Tab Federations Registering a Federation Databases Creating a Database Adding a Storage System Using the Spaces Tab Creating a Space Using the Stores Tab Editing Stores Configuration Configuring ECS Storage Installing Centera SDK on Linux Installing Centera SDK on Windows Configuring Centera Storage Using the Audit Tab Audit Application Load Balancing Testing of InfoArchive Servers Load Balancing with Apache Parallel SIP ingestion Parallel Table Ingestion Load Balancing Testing of the InfoArchive Web Application HTTP with Sticky Sessions How to Run Gateway/InfoArchive Web Application InfoArchive Server and Gateway/InfoArchive Web Application Communication Setup Self-Signed Certificates-Based Setup

7 Revision History The following changes have been made to this document. Revision Date October 2016 Description The following sections were added: Installing Centera SDK on Linux Installing Centera SDK on Windows September 2016 June 2016 Load Balancing Testing of InfoArchive Servers 4.1 Release. InfoArchive version 4.0 initial publication 7

8 Revision History 8

9 Overview Chapter 1 InfoArchive is an integrated product suite designed for managing and archiving application information. It is important to understand fundamental InfoArchive concepts before performing configurative and administrative tasks. This chapter grounds you in the essentials of InfoArchive terminology and concepts that underlie the complete data archival and retrieval processes. InfoArchive is designed using a unified OAIS-compliant data model that dictates the format information is ingested into, stored and archived in, and retrieved from InfoArchive throughout its lifecycle. InfoArchive allows the customer to: Ingest and store data, Protect ingested data, Execute queries against ingested data, and Dispose of obsolete data in a controlled fashion. InfoArchive allows the ingestion of data from: Live systems that currently generate data. The customers plans to continue supporting and maintain the system until the end of its active service. Retired systems that no longer generate data. The customer has been maintaining and supporting the system for compliance reasons, for instance. InfoArchive allows the customer to load the retired application s data into a cheaper system, thereby decreasing the total cost of ownership. The customer can then execute any required queries against the ingested data. Customers are able to ingest tabled-based data as well as OAIS-based data, which is also referred to as SIP archiving. The customer uses a command line interface (CLI) to ingest table and SIP data. Structured and unstructured data can be ingested into InfoArchive. 9

10 Overview Backup Versus Archive Backup Protection Frequent snapshots of data to protect against data loss because of system failure Copies data for protection Supports operations and recovery Supports availability Point-in-time only Poor solution for regulatory compliance Not easily searched Often, old backups cannot be restored Archive Preservation Movement of data to a low-cost platform to ensure compliance and reduce costs Moves data off production disk Supports business and compliance Supports operational efficiencies Comprehensive in nature Ideal solution for regulatory compliance Easily searched Provides historic reference Modes of Archiving: When to Use Table Archiving or SIP Archiving It is important to determine the best method to archive structured and unstructured data. InfoArchive is an application-agnostic solution for information management and archiving that supports different enterprise needs for ingesting different types of application data. It can help reduce the cost of managing application assets, improve information governance, and add value to business processes through the reuse of information. Four methods of ingestion are provided to meet the needs of competing project requirements and optimize the environment to the source application. With InfoArchive, there is no need to take a one-size-fits-all approach. This section provides a step-by step guide to help you choose which of the four InfoArchive methods for preserving and reusing information is the right method to use for each of your applications. Step 1: Setting Archiving Goals It supports three core use cases based on your short and long-term archiving goals: Cost Take-Out. InfoArchive can provide a repository for data from legacy and redundant applications that might have been superseded by an ERP system, replicated during an acquisition, or must be decommissioned as part of a business sale, closure, or industry mandate. Data from applications that a company migrates to InfoArchive will remain accessible for business reporting, audits, or compliance with data-retention regulations. Meanwhile, the organization can shut 10

11 Overview down the applications and save all the costs associated with supporting and maintaining the applications. Optimize. Companies can also use InfoArchive for periodic archiving of data and content from live business applications to reduce costs for production environments, enable compliant data retention, and optimize application and infrastructure performance. In addition to savings on storage costs, companies can reduce costs for backups, system administration, servers, and database licensing costs. IT Transformation & Reuse. Increasingly, organizations require access to information for new and innovative uses. Advanced and predictive analytics as well as application modernization programs top the list of major programs that demand fresh approaches to information access. InfoArchive supports these requirements by serving as a platform for data aggregation and management that offers access to business records in bulk via the Hadoop Distributed File System (HDFS) or individually to other business applications via web services. InfoArchive uses extensible mark-up language (XML) as the format for preserving data and metadata for long-term, platform-independent retention. Data/metadata from multiple sources can be aggregated and represented as XML files. This representation can actually translate to a business object. The technology archives the native XML data and structure, allowing users to query content efficiently and accurately, at any level of detail. Users can also transform that data into views formatted for print, web sites, mobile devices, and other channels. In addition, user interface development tools use declarative XML syntax. This greatly reduces the need for and cost of custom programming to deliver interactive content. With InfoArchive, companies can manage an unlimited number of different data structures in a single repository. They can store both structured data and unstructured content in a single record and access all the information they need for their business processes and reporting from a single query. InfoArchive is the only solution that delivers all of the following methods to ingest information for archiving: Table archiving Data record archiving File archiving Compound record archiving Organizations can use any of these options to decommission applications and archive active applications. Such flexibility is particularly valuable in decommissioning programs that involve many different applications and information formats. Active archiving may use all of the options except table archiving. When information aggregation and reuse is key, Data Record, File and Compound Record archiving options should be considered. 11

12 Overview Step 2: Understanding the Source Application When choosing the best archiving method for a particular application, a business must consider two key questions: What type of information is being archived? How users will access that information going forward? Most information is managed in one of the following systems: Transaction systems Print stream systems Content and image repositories Interaction systems Collaborative systems Transaction systems have databases that hold details of past business events, such as those related to processes for accounting, ERP, enterprise asset management, or supply chain management. They are used to maintain reference data in master files, record activities in transaction files, and store old records in transaction-history tables. They may include cloud-based systems and allow many people to add small bits of detail over time. Traditionally referred to as COLD (computer output to laser disk) systems, these systems store print-stream information for long-term preservation. Most of this information involves customer communications. Other examples include green-bar reporting systems. Content and image repositories store unstructured information and metadata typically in their native formats. Examples include the traditional enterprise content management systems, as well as storage-based systems. Interaction systems connect users with an organization for quick access to complete information. Examples include systems that support customer relationship management (CRM) and collaborative tasks. These systems include data as well as transaction, grouping, and unstructured files. Collaborative systems address the needs of groups of individuals to share information and communicate with each other around specific topics. These systems have all the characteristics of interaction systems but generally cater to a less-structured approach. Notable examples include eroom, Microsoft SharePoint, and Lotus Notes. Step 3: Selecting the Appropriate Archiving Method The four archiving methods offered by InfoArchive are optimized based on the format of data or content that is being archived, the ease of extraction and up-front analysis, and how the information is to be used after it is moved to the InfoArchive repository. Having this choice is a critical success factor for large-scale information management programs that involve a wide range of applications. 12

13 Overview Table Archiving Table archiving is a method that models structured information in the source application as XML in InfoArchive table for table, column for column, and row for row. This method lets organizations quickly decommission applications, providing a fast time to value, while preserving all data relationships for future queries and reports. Organizations can use table archiving to migrate structured data in application tables and linked files from transaction systems to InfoArchive with few, if any transformations. Table archiving can reduce the up-front analysis involved with decommissioning an application and virtually eliminate the data-integrity risks associated with other archiving methods. Due to the fact information is stored in an aggregated manner, access is less flexible than the record-based methods described below. Access is traditionally limited to query-based (list) reports. Table archiving is used primarily with transaction systems that contain structured data and linked files as well as with some interaction systems for the purpose of application decommissioning. Information preserved using the table-archiving method may be reprocessed at a later date into record formats for reuse scenarios. Data Archiving Data archiving involves identifying entities within the target application and extracting and aggregating the associated data into a single XML-based record. Archiving structured data in XML files provides a future-proof format that is ideally suited to long-term retention, access, and comprehension. Any data structure can be modelled as XML and note that InfoArchive does not impose its schema. As such, one record may contain multiple XML packets, information from multiple systems may be drawn together according to the requirements of the project all while preserving a complex multi-system chain-of-custody. Data archiving is especially useful when companies want to reuse the information, while reducing costs, in a context that is different from the source application. Additionally, data archiving is well suited for active archiving of live systems. It typically requires additional and more business-oriented analysis of application data than table archiving does. Examples of appropriate data archiving use are SWIFT transactions, sales histories, and patient histories. There are two important advantages to this method. First, the complex data model of the source application is transformed into a simple data model in the archive. This can reduce costs and simplify future access. Second, because there is no direct link between the source application and data in the archive, any change in the source application does not force a change in the archive. When a change in the source application results in an update to the archive data model, InfoArchive ensures that results for searches of data sets include all records across all the changed data. Data archiving also helps organizations create new value for their data. By extending or recording metadata, they can harmonize records and support searching and filtering across data sets. Data archiving is used with transaction systems for active archiving of individual structured data records (for example, transaction history tables). It is also used with interaction systems (for decommissioning data and queries, optimizing searches, and advanced analytics), and with content systems. Because it presents data as single records, it is ideal for archiving information according to government requirements and legal mandates. 13

14 Overview File Archiving File archiving involves archiving unstructured data and its associated metadata into a single record. The information can be preserved in original format and/or transformed into a more future-proof format, such as PDF-A. One record may contain multiple files to create sets of related information. Metadata is particularly important when working with the file-archiving method. Attributes may be derived from the content itself, or associated with other systems. File archiving is especially useful when organizations want to reuse the information in a context that is different from the source application. Print streams and media archives are an excellent example of file archiving. Transforming very large print streams (such as customer statements, or explanations of benefits) from print-oriented formats (such as AFP or metacode) into a PDF for presentation on company web platforms can improve ROI and customer satisfaction. Image archives (typically multi-page TIF with very limited attributes) can be reprocessed to add document type and even full-text OCR (optical character recognition). File archiving can be used with transaction, content, or reporting systems to decommission content, optimize data search and retrieval, and simplify user access to information across systems. Businesses can use this method to archive any kind of unstructured content files and metadata, including print streams. It extracts value from archived content by addressing files via the associated application metadata, rather than directly from the infrastructure. Pushing files into InfoArchive the moment they become inactive can reduce costs and increase system performance. Having all of an organization s inactive files in the same archiving platform, can also improve data discovery and enhance the options for reusing content and data through new applications or web browsers. Please note the information can still be accessed via end-user portals with high availability. Compound-Record Archiving As the name implies, compound-record archiving preserves structured and unstructured data into a single record. The structured elements are modelled as XML, and the unstructured elements may be preserved in the original format or transformed into a more future-proof format, such as PDF-A. One record will contain multiple files of related information. Compound-record archiving provides the only mechanism that can compliantly archive systems with a blend of structured information (such as wikis, blogs, and comments) with unstructured information (primarily attachments). It serves as a proper format for decommissioning as well as active archiving interactive systems that involve such a mix of structured data and unstructured content. Microsoft SharePoint and Lotus Notes applications are notable examples. As business processes become more complex and regulations more demanding, there is a growing need to archive complex business records that may contain multiple structured data and unstructured content elements and must be brought together to create the final business record. Examples include financial trades, cases, and laboratory reports. Organizations can retain business events as single records that they can reuse for analytics or regulatory audits. Users can search the records using pure business logic, without switching from one application to another. 14

15 Overview A Solution for Structured and Unstructured Content Any ERP implementation involves the creation and management of various records or business objects. Some of these records are composed of structured data that users typically enter and access through form fields. As much as 85% of managed information can be unstructured content, according to some estimates. This content may include digital images, video, digitally rendered faxes, messages, and text documents. While both structured and unstructured information are usually needed to drive efficient, ERP-enabled business processes, most ERP applications do not have the robust functionality required for handling the indexing, searching, storage, and security of huge volumes of unstructured information in multiple formats. A content management solution, such as Documentum, is often needed to provide such support. Submission Information Packages With InfoArchive, information for data record, file, and compound archiving is extracted by an appropriate connector and pushed to the solution in a component known in the terminology of the Open Archival Information System (OAIS) as a submission information package (SIP). The SIP is compressed as a.zip file and can be transported using any file transfer technology. The SIP is ingested to the archive and becomes an archive information package (AIP). Based on the classification, the content is stored in an archive holding. When an end user requests data from the archive, the data is delivered as a dissemination information package (DIP). SIPs are compressed for transfer between the source application and InfoArchive to reduce network traffic. Each SIP includes: 1. The SIP Metadata File: A small XML file containing data that describes the SIP and provides data that InfoArchive uses to set retention dates, access rights, and other archive policies 2. Archive Content Metadata: Another XML file that holds the metadata associated with the content to be archived 3. Content File: The unstructured content that is to be archived When structured data is archived to InfoArchive, the SIP does not contain any content object. The structured data that is to be archived is held as XML in the archive content metadata file and InfoArchive processes the SIP in the same way it would ingest a SIP with unstructured content. This standard process for archiving both types of content enhances efficiency and total cost of ownership for InfoArchive. InfoArchive Information Model AIPs are discrete packages of information that may contain none-to- many structured data elements represented as XML and/or none-to- many unstructured data elements. The ability to layer metadata over an AIP adds additional power, especially with reuse scenarios. The metadata and data elements may be extracted directly from the source application or derived 15

16 Overview from other systems or programmatically constructed. The ability to maintain separate data elements in an AIP allows InfoArchive to balance the requirements to maintain exacting standards around chain-of-custody with the desire to build richer data sets than existed in the original applications. For example, the raw transaction history data may be extracted, modeled as XML, and verified as 100% accurate and complete for chain-of-custody purposes, while additional information from extended systems can be made part of the AIU in another data element making the AIU more usable without compromise. Generating SIPs To generate a SIP, information is extracted from the source application and written to the standard SIP format for InfoArchive as an XML file. A SIP can contain one or more information records, called archive information units (AIUs). The AIUs that are extracted to the SIP are defined by rules that are part of the SIP-generation program. When a SIP is ingested to the repository, it becomes an AIP. The AIP is stored in a logical archive-holding folder. The folder has a number of configuration options for setting management parameters that are applied to AIPs including data retention, access control, and search SLAs. AIPs extracted from different systems can be stored in the same archive holding. Use Cases This section illustrates how InfoArchive can help reduce costs: Cost Take-Out: Drive cost of your existing IT environment Optiomize: Make production application and infrastructure more efficient Information Transformation: Remove information silos and put your information to work in new ways Cost Take-Out: Application Decommissioning When you use InfoArchive to decommission an application, you: Reduce IT costs and complexity of running legacy- or duplicate systems. Reduce the risk of failure of existing systems. Retain information for future business mining or usage. Retain information to ensure compliance requirements are fulfilled. Support e-discovery initiatives. 16

17 Overview With application decommissioning: The Developer can control all aspects of an xquery, nothing is pre-defined. The search form designer must understand XQuery and how it is related to search design. The designer of a search needs to ensure the query is valid. Form widgets, result column and data binding require careful consideration. The Developer has to handle the values and results in the xquery. Developers has to test the query in a tool like xdb Admin, Eclipse or any other similar setup. When you are using table archiving, upfront analysis is not required, as you simply have to ingest the source application s data. Typically, then the Developer has to create search screens similar to the source application s search screens. Table archiving also requires an understanding of SQL queries that are required to create specific reports. Content can be stored as blob (in xdb) or in the storage system, which is the recommended alternative in production. Retention of the source application s data can be applied at the application, table or record level. Table archiving handles one table, including all of the table s row, as one or more XML files. Therefore, the schema of the XML is the schema of the table. InfoArchive is schema-agnostic, meaning that it can work with any valid XML file. InfoArchive requires a file, metadata.xml, per database that describes the extracted XML files. Data can be extracted into multiple files. An ANT-based ingestion script requires all files to be ingested in one session. Indexing needs to be enabled at the column level in the metadata.xml file. Multi-path indexes are created at the end of the ingestion process. The task can be configured in the tools\build-table.xml file. Path Value Indexing It can be used for indexing multiple elements, but it requires every single element to be explicitly listed in the index definition. Smaller size means that it is faster to ingest Better performance if you know the query ahead of time along with the number of predicates B-tree index Multi-Path Indexing Multi-path indexes allow you to specify sub-paths with wildcards that will match more than one element path, so not every element has to be explicitly listed. Making multi-path indexes much more flexible and easy to use. Large index size Only option for table archiving Lucene Inverted index 17

18 Overview Optimize: Active Application Archiving Active archiving allows an organization to: Manage data growth and reduce costs Ensure that compliance requirements are fulfilled Ensure long term accessibility of information Support e-discovery initiatives Optimize production systems Improve application performance Reduce upgrade downtime Improve overall manageability Active archiving allows you to archive and retain large volumes of transaction records, fixed content (i.e., checks, statements) and inactive application data. In regulated industries, information from completed projects such as pharmaceutical studies, Cases and Construction Projects can be archived together, which ensures compliance, long term accessibility and optimization of production systems. Information Transformation and Reuse InfoArchive allows an organization to: Extend the value of information Remove application information silos Provide new context for users User Roles For more information about administering user roles, see Using the Groups Tab and Using the Permissions Tab. Role Business Owner Responsibilities Drives business decisions to archive legacy applications and application data into lnfoarchive. Monitors high-level application archival statuses. IT Owner Defines corporate information preservation requirements. Reviews and responds to archive requests. Plans for data archiving and application archiving. Reviews both high-level and detailed application archival statuses. 18

19 Overview Role IT Administrator Responsibilities Sets up the InfoArchive system. Performs day-to-day administrative tasks on InfoArchive. Retention Manager Implements information security policies. Collects corporate, regulatory, and legal requirements for information preservation. Establishes and implements retention policies and prevents inadvertent disposal of data. Promptly responds to new requests for holds and retentions. Developer End User Keeps data custodians informed about upcoming purges or changes to retention policies. Designs the InfoArchive Web Application UI based on business requirements. Accesses archived application data. Inline Users For easy demo-ability and for getting up and running for POC work, there is a need for set of users with pre-configured roles (server running in infoarchive.ias.profile.http_basic profile): User Authorization Role adam@iacustomer.com Authorization: Basic YWRhbUBpYWN1c3RvbWVyLmNvbTpw- YXNzd29yZA== bob@iacustomer.com Authorization: Basic Ym9iQGlhY3VzdG9tZXIuY29tOnBhc3N3b3Jk connie@iacustomer.com Authorization: Basic Y29ubmllQGlhY3VzdG9tZXIuY29tOnBhc3- N3b3Jk emma@iacustomer.com Authorization: Basic ZW1tYUBpYWN1c3RvbWVyLmNvbTpw- YXNzd29yZA== imran@iacustomer.com Authorization: Basic aw1yyw5aawfjdxn0b21lci5jb206cgfzc3dvcmq= Administrator Business Owner Developer End User IT Owner 19

20 Overview User Authorization Role Authorization: Basic cml0yubpywn1c3rvbwvylmnvbtpw- YXNzd29yZA== Authorization: Basic c3vlqglhy3vzdg9tzxiuy29tonbhc3n3b3jk Retention Manager Administrator Business Owner Developer End User IT Owner Retention Manager The password for each inline user is password. Troubleshooting If you experience difficulty logging in or encounter weird behavior, refresh the browser cache. Branding Customization Infoarchive supports limited, drop-in branding customization. Customers are able to define the view and display of the font, color, images and styling of the InfoArchive web application. Configuration to Add Sample Branding Customization A sample branding customization is provided to the customer in the following directory: <install_directory>\tools\tenants\infoarchive\customization. To review the sample branding, copy the customization folder and paste it into the following directory: <install_directory>\config\webapp. 20

21 Overview Configuration to Add New Branding Customization To replace the image file of the InfoArchive web application, complete the following steps: 1. Open the following directory in Windows Explorer: <install_directory>\config\webapp \customization\branding\images. 2. Review the image requirements and directions in the README file. 3. Copy and replace the image file to the folder. Do not change the name of the image file. To replace the styling and CSS rule of the InfoArchive web application, complete the following steps: 1. Open the following directory in Windows Explorer: <install_directory>\config\webapp \customization\branding\css. 2. Review the information about current CSS rules in the README file. 3. Open the custom.css file with a text editor and edit the styling rules using CSS syntax. Setting the Customization Location when Deploying to Tomcat or Other Containers The previously described steps work when the InfoArchive web application is run as a standalone Springboot application using the infoarchive\bin\infoarchive-webapp command. When it is deployed to an external Tomcat container, however, the situation is different. For example, assuming the external Tomcat container is installed at: C:\apache-tomcat And the InfoArchive web application.war file is deployed at: C:\apache-tomcat \webapps\infoarchive-webapp.war When deployed, the Tomcat expands the.war file to the following folder: C:\apache-tomcat \webapps\infoarchive-webapp Now, assume that you copied the customization folder to the following directory: C:\apache-tomcat \webapps\infoarchive-webapp\config\webapp\customization Then, one way to set the customization location is to edit the application.yml file located at: C: \apache-tomcat \webapps\infoarchive-webapp\web-inf\classes\application.yml and change the key: : infoarchive: gateway: : 21

22 Overview : customization: location: file:///c:/apache-tomcat /webapps/infoarchive-webapp/ config/webapp/customization Alternately, another method to set the customization location is by creating a web.xml file at: C:\apache-tomcat \webapps\infoarchive-webapp\WEB-INF\web.xml Set the contents to: <web-app version="3.0" xmlns=" xmlns:xsi=" xsi:schemalocation=" <context-param> <description>infoarchive Customization location</description> <param-name>infoarchive.webapp.customization.location</param-name> <param-value>file:///c:/apache-tomcat /webapps/infoarchive-webapp/config/webapp/customiza </context-param> </web-app> Verifying and Viewing the Branding Customization To verify if the web customization, complete the following steps: 1. Open the InfoArchive web application in an Internet browser. 2. Refresh and reload the web page. 3. The customized branding style should be displayed. Troubleshooting If the new customization is not displayed, clean up the browser cache. 22

23 Connectors Chapter 2 InfoArchive s simple XML-based ingestion template and integration framework offers a simple, open integration interface. Table Data Record File Compound Any ETL tool* Any ETL tool* Any ETL tool* Any ETL tool* Oracle, BD2, SQL Asset Suite for InfoArchive InfoArchive Documentum Connector InfoArchive SharePoint Connector EMC InfoArchive Connectors EMC Partner Connectors Oracle, SQL EMC Kazeon File shares Crawford Print Stream, reports FME Migration Centre Filenet, Notes, SharePoint, Opentext, Alfresco, File shares * Includes Talend, Powercenter, Datastage (Infosphere Information Server), Pentaho, AB Initio, Clover, Data Integrator and BO Data Integrator 23

24 Connectors Application and Platform Examples InfoArchive preserves application data. Data is typically extracted via the application API so the underlying database and hardware platform is not a limiting factor. Application Platforms Databases Lotus Notes Mainframe Oracle SharePoint AS400 BD2 Documentum Windows SQL PeopleSoft Unix XML databases Baan Solaris ADABAS BASE T24 ASG Mobius ERPs Financial Applications HR Systems (multiple) Core Banking Applications Customer Statement Applications Healthcare Applications Life Sciences Applications VMS LINUX 24

25 Architecture Chapter 3 Table archiving focuses on the power of xdb as an XML database. SIP archiving focuses on using the library functionality of xdb and the power of XML as standard and perennial format. InfoArchive uses a Gateway pattern as a single point of access and authentication, which offers: An architecture that is highly scaleable. InfoArchive servers can be scaled vertically but can also be specialized on the different functionality that InfoArchive REST services offer (i.e., Ingestion, Search, JDBC, Administration, etc.). This enables scalability on any level and can grow with the SLA and ROI needs of the business. Modern pattern for emerging microservice-based architecture. A cloud-friendly environment. OAuth2 based stateless authentication uses a modern JSON Web Token JWT. This allows for self-contained, signed, secure tokens for clustering support. InfoArchive s role sensitive user interface relies on an AngularJS and Bootstrap framework, along with Spring Boot tier. The InfoArchive server uses Spring Data, which relies on REST services authentication (OAuth2/JWT). Distribution is performed with HTTP load balancing. xdb allows the customer to: Ingest data Manage federations, databases and detachable libraries Manage transactions Perform searches Perform transformations and serialization The InfoArchive database can also be scaled vertically and separated into: The system repository that maintains the consistency of the application but not the business data. One or multiple metadata repositories that contain the libraries/segments used to search structured content. 25

26 Architecture A holding must reside in an xdb metadata instance. The storage systems include metadata, content and system data. A cross-holding search is possible in one xdb instance. This platform allows for bulk ingestion, querying, as well as configuration and administration services. It also sets up the framework for the main areas of InfoArchive s functionality: Login and Logout Dashboard Reports Activity and Notification Preferences Search Design: Query Form Results Search Execution: Forms Results (ability to export) Configuration and Administration Retention Management: Retention Policies Legal Holds Ingestion The ingestion process is one of the areas in which the differences between SIP and table archiving is evident. The common base is one or more modules that enable InfoArchive to connect to the following external systems: Local/shared file system The ETL process These external systems assist in bulk loading data. After that, the code lines diverge up to the point where data is stored and indexed in XML/xDB documents, in libraries, databases or federations. 26

27 Architecture xdb is used to manage federations, databases and libraries, which is an important part of InfoArchive s architecture because it plays a vital role in: Improving scalability Organizing the retention process Lowering a customer s total cost of ownership by moving data to cheaper storage Mapping xdb data to data in HDFS Optimizing queries (knowing what data partition is in which library) What is ETL? ETL stands for Extract, Transform and Load, the three combined functions that are needed to take data from an active legacy application to another platform. Extract Transform Load Read data from an application system. Data is extracted in its natural form, whether it is in tables or file format. Applies rules or functions to covert source data extract to XML. Some systems can natively export in XML. Ingests the transformed XML data into xdb. Digital data storage (DDS) provides a toolset for loading data with assigned retention and metadata. An ETL process is needed by all in order to move legacy data into InfoArchive so that the source application can be turned off or archived. The EMC solution is open to any ETL tool that can transform data to XML or submission information packages (SIPs). Some applications have a built in facility to export to XML How to Extract Data Applications and data cannot be archived without some means of extracting and loading data into InfoArchive. Data validation is part of the ETL process. Audit points and chain of custody metadata stored with the legacy data are critical for governance and compliance. Only after validation and audit requirements have been met can a legacy system be shutdown and removed from existence, or data can be purged from the original application. 27

28 Architecture Storage What happens to data upon ingestion depends on whether the data is structured or unstructured: whereas unstructured data goes into a file system, an xdb database. Before ingestion, there is some necessary and some optional configuration for storage of structured and unstructured data. Structured data, such as AIUs or records, goes into an xdb database. A minimum of one XdbFederation and XdbDatabase need to be configured. The XdbDatabase must not. however, be the same database configured for Spring Data. Unstructured data goes into a file system and then the InfoArchive database. A minimum of one XdbFederation, XdbDatabase and FileSystemRoot need to be configured. Because ingestion is always in the context of an application, an application has to be configured. XdbLibraries and FileSystemFolders are in the XdbDatabases, under the FilesystemRoots. When data is ingested into an application, it is ingested into the application s space.which XdbLibraries and FileSystemFolders are picked for ingestion is further determined by an application s space. The following list illustrates the hierarchy for storage configuration, with the most important attributes per item: XdbFederation (name, bootstrapurl, superuserpassword) XdbDatabase (name, adminpassword) FileSystemRoot (name, path) Tenant (name) Application (name,...) Space (name) SpaceRootXdbLibrary* (name, XdbDatabase) XdbLibrary (subpath,...)... XML documents with records or AIUs Content/Blobs SpaceRootFolder* (path, FileSystemRoot) FileSystemFolder (subpath,...)... Content/Blobs A space is a collection of libraries in XdbDatabases (called SpaceRootXdbLibraires) and folders in FileSystemRoots (called SpaceRootFolders). Typically, customers will not require more than one of each. If table data is ingested, a space is constructed and configured automatically. The following list illustrates the configuration flow: 1. The Administrator configures an XdbDatabase. 2. The Administrator configures a FileSystemRoot. 28

29 Architecture 3. The Developer configures an application. 4. The Developer configures a space for the application with a SpaceRootXdbLibrary in the XdbDatabase and a SpaceRootFolder under the FileSystemRoot. 5. Ingestion starts. The FileSystemFolders and XbBlibraries (essentially, anything under a SpaceRootFolder or SpaceRootXdbLibrary) is generated by the server as part of the ingestion process. The administrator does not have to configure these entities. In the Admin UI, a space can be configured through forms for configuring SpaceRootXdbLibraries and SpaceRootFolders, In those forms, the customer can select the XdbDatabase/FileSystemRoot of the SpaceRootXdbLibrary/SpaceRootFolder from a list. How Data is Searched Using the InfoArchive interface, it is possible to create a search from scratch or to import a search. Search composition involves:, a query editor The creation of a search, search form, result list and result detail Using the Query Editor to configure a search Setting the permissions to allow specific user groups to access the search Search composition is the entry point of the search. From search composition, it is possible to create a search, run a search and retrieve elements for the search form and the search result page. Two types of searches can be executed: Synchronous search Asynchronous search (also named the background search ) A search can be directly executed as a background search or a synchronous search. If a synchronous search takes too much time, it can be switched to a background search. A search is associated with a single dataset based on the archivint type. For SIP archiving, a search is associated with an AIC and a query configuration. For table archiving, a search is associated with a schema or table. Search composition involves various components. Every search form is contained within an application, and has a name, description and state to indicate whether the search form is a draft or ready to be accessed by the End User. Each search form is also associated with a single dataset, an object that describes the target of the search (e.g., the database and tables associated with a table-archiving application). SIP-based datasets also contain reference to QueryConfiguration. Table-based search forms require a query (built upon XQueryTemplate) as well as the schema or table the search will be executed upon. SIP-based search forms require the name of the AIC and QueryConfiguration, which determines the criteria and results for the search. 29

30 Architecture The form designer defines the dataset for a given search. When creating a table-based search form, the Developer has to select either a schema or table. For SIP-based searches, the Configuration Developer has to select an Archive Collection (an AIC) and a query configuration. A search form can be associated with a user group or role to limit its availability to specific users. It can contain any number of search composition structures that configure the queries, forms, etc. The details of the objects comprising a search composition can be specific to the application type. For SIP archiving, most of these components can be synthesized from special configuration objects. For table archiving, the Developer has to manually complete the work. The SIP search uses two queries. The first one returns the suitable AIP, based on the partition key. This information appears in the wizard when selecting the field for the search criteria. The second query filters the suitable AIUs in the selected AIP list. Search performance can be improved using (and filling) at least one partition key in the search form. Forms For table-based searches, the search form designer must understand XQuery and how its related to search design. The designer of a search needs to ensure: The query is valid. The parameters used are accurate. The correct binding for elements is used in the result set. To manipulate the data bindings, an API was created and added to the XForms engine. For SIP- and table-based searches, XForms is hosted within an XHTML template. The header of the template contains the XForm instance and the bindings. The body of the XHTML document contains the element directives, decorated with classes, in-line styling and other attributes, as related to various elements. Search Results During the search form composition stage, the form designer also defines the search results for the search. For table-based searches, just as search fields are selected manually, search results are created by providing labels for columns that must match the associated xquery. For a SIP-based search, the list of available columns is provided from the search configuration (i.e., the selection of the AIC and the query configuration). The form designer has to configure the search result list (i.e., the columns of the result page), but also the detail panel. The detail panel of the SIP search contains two sub-parts: The side panel (that appears on the side of the result grid) The Inline Panel to include detail data in the result grid (useful to insert repeating information (like authors ) in a search result. 30

31 Architecture This is why the list of fields is longer when selecting from a schema in the inline panel compared to the select from schema in the side panel. A repeating field can only be added in the inline panel. InfoArchive expects a "return token" service, which is a back end service that, when given an xquery, the service returns response attributes. This is similar to the xquery analyzer and xquery validation services. The service consumes the xquery before providing the proper response. A nested search yields a special kind of result column that includes a name and result list sorting index. Binding attributes and result detail properties do not apply to nested searches. For a SIP-based search, there are two methods to search : From the user interface with the search composition, search, search form, result master resources. From the REST API, with the AIC resource, it is possible to create a search with the AIC and a result schema (see dip link on the AIC resource). This search returns raw search results in an XML format from the result schema provided. 31

32 Architecture 32

33 Core Configuration Chapter 4 Before you can start ingesting and searching data, an initial system configuration must be set up: System Data Structured Data Storage Unstructured Data Storage Initial System Configuration Storage is configured using the following: System Database (xdb) Audit Database (xdb) Structured Database (xdb) Content Database (typically not xdb but FileSystem or EMC Isilon, ECS, Centera, etc.) A tenant and an application must also be configured. For an application, specifically, the following must be configured: Holding SQL Database, schema, table layout Stored searches System and Audit Database The system database is where all system objects live (tenants, applications, spaces, searches, holdings, tables, etc.). Audit data is where audit records are stored until they are SIP-ified and archived. The system database and audit data are both configured in the.yml file. The following is from the application.yml for a system database: system: username: system password: system pagesize: 4096 cachesizetotal: 50 33

34 Core Configuration cachesizesystem: 50 xdb: datanode: name: mainfederation bootstrap: xhive://localhost:8080 superuser: password: test database: name: maindatabase admin: password: secret The following is from the application.yml for audit data: auditdata: xdb: datanode: name: mainfederation bootstrap: xhive://localhost:8080 superuser: password: test database: name: auditdatabase admin: password: secret xdb Federations and xdb Databases Every xdb federation used by InfoArchive must exist. InfoArchive can connect to a federation, but it cannot start it up. Every xdb federation and xdb databases, with the exceptions of System and Audit, must be registered in the system database. This can be done using a REST call, the IAShell or the InfoArchive web application. Structured data in InfoArchive, including AIPs/AIUs and table data, is stored in xdb databases. There are also two important repositories: A potentially large repository where PolicyApplications and HoldApplications live. It is recommended to keep this in a federation separate from the one under "system". manageditemdata: xdb: datanode: name: mainfederation bootstrap: xhive://localhost:8080 superuser: password: test database: name: manageditemdatabase admin: password: secret 34

35 Core Configuration A repository for batch-related objects. This can be in the same federation, but not in the same database, as the SystemData. batchdata: xdb: datanode: name: mainfederation bootstrap: xhive://localhost:8080 superuser: password: test database: name: batchdatabase admin: password: secret File System Root Any unstructured content is stored on a file system. A file system root is a folder under a mount point of a shared file system. It must be shared by all InfoArchive servers and under the same path. Every file system root used by InfoArchive must be registered in the system database, either through a REST call, the IAShell or the InfoArchive web application, Space Root xdb Library and Space Root Folder The space root xdb library and space root folder are the configuration items that uniquely allocate a slice of storage to a space or application. The space root xdb library is an xdb library reserved for a particular space. Anything stored in the library belongs to the same space. The space root folder is a file system folder reserved for a particular space. Anything stored in the folder belongs to the same space. Language Support InfoArchive has the ability to dynamically add a new language translation for the InfoArchive web application. The InfoArchive web application login page allows users to select a language from the drop-down list with seven language choices. The supplemental language pack installs all primary languages. For the supported language to work, a corresponding <locale_name>.json file should be populated into the classpath. The classpath is automatically modified by supplemental language pack installation. InfoArchive provides ease of configuration for new language translation. 35

36 Core Configuration The language support is improved to be more dynamic. In the new approach: 1. The languages drop-down list values are decided at run-time based on the presence of supporting language files. 2. The user can configure more languages by following the configuration steps. Also, the language drop-down list control on the login page shows only those languages for which there are corresponding <locale_name>.json files either in classpath or in the configurable location. It gives a precise picture of languages supported and avoids confusion when a user selects a language for which a corresponding <locale_name>.json is not present. Adding New Language Support Complete the following steps to facilitate support for a new language translation other than the seven primary languages. To add support for a new language, the corresponding <locale_name>.json file should be generated first. Customers should usethe en.json file from <<Installed_Dir>>/lib/infoarchive-webapp.jar under /WEB-INF/classes/static/languages folder as a reference for translation into new language. The following example illustrates how to add support for the Dutch language. 1. Create a folder named customization under <<Installed_Dir>>/config/web app folder in the distribution. 2. Create a folder named languages in the customization folder that you created in the previous step. 3. Create a file named languages.json in the languages folder that you created in the previous step. 4. Edit the languages.json file and add the following content in the file: { } "nl": "Dutch" 5. Drop the nl.json file (equivalent of en.json, zh.json for Dutch language) into the customization > languages folder. 6. Refresh the login page (using Ctrl + R or F5). There will be an option to select the Dutch language in the languages drop-down menu. If you want to add support for the Russian language, complete the following steps: 1. Add an entry for the Russian language in the languages.json file: { "nl": "Dutch", "ru : "Russian" } 2. Drop the ru.json file into the customization > languages folder. 36

37 Core Configuration 3. Refresh the login page (using Ctrl + R or F5). There will be an option to select the Russian language in the languages drop-down menu. For more information, see Setting the Customization Location when Deploying to Tomcat or Other Containers. Backing Up and Restoring of the Managed Item Database Managed items (i.e., managed items, policy and hold applications) are stored in a separate database. As part of a disaster recovery scenario, they need to be backed up and kept in sync with the system data (retention policies, holds, etc.). Unlike AIPs, managed items can change as policies are applied, holds are added and removed, so a different disaster recovery strategy is required. If configured, as managed items change, InfoArchive will automatically backup the changes to a configured store. Configuring the Managed Items Database The managed items (HoldApplication, PolicyApplication, and ManagedItem objects) are partitioned over multiple xdblibraries, with the AIP/table as the partition key. So, for example, all HoldApplications for AIUs in one AIP are together in a separate xdb library. Depending on the expected size of the managed items database, you may want to move them to a different data node: If you plan to use fine grained (AIU/record-level) retention, or expect a large number of holds on individual AIUs/records, then the managed items database may grow to even the size of the structured data. At that point, it is no longer feasible to back up the managed items as part of system data backups. The backup will simply take too long. By mapping the managed items to a different data node and configuring stores for managed item backups under Holdings/Databases, the backup process becomes: The managed items for each AIP/table are backed up separately, after each change (new or removed hold, new retention policy application, etc.). This happens immediately after the transaction where the changes took place. Configuring the Managed Item database in the application.yml file. For example: manageditemdata: xdb: datanode: storestacktraceinlock: false name: mainfederation bootstrap: xhive://localhost:8080 superuser: password: test database: name: manageditemdatabase 37

38 Core Configuration admin: password: secret Backing Up the Managed Items Database The managed items can be set to automatically backup when they are changed in InfoArchive. Setting the store for the managed items is controlled by application, and each application needs to have the managed item store configured for InfoArchive to backup managed items. By default, the managed item store is not configured. If the store is not set, InfoArchive will not backup the managed items. The backup strategy for the managed items takes into account how you are backing up the system data and how you are applying retention and holds. The system data database is where the configuration objects (retention policies, holds) and the retained and hold sets are located. No matter which method is being used to backup the managed items (InfoArchive or a scheduled backup), ensure the backups to the system data and the managed item data should be completed at the same time. InfoArchive uses a store for writing the managed items backup files. Ensure that there is sufficient space on the store and that the content can be removed (during disposition of the underlying managed item, hold application and policy application). For more information, see the Using the Stores Tab section in the Configuration & Administration User Guide. Setting the Managed Item Store For SIP archiving, you have two options when setting the managed item store: 1. Use IAShell to update the holding for the application; or 2. Use the REST interface to update the holding for the application. The attribute on the holding object is called manageditemstore and is a store object reference. For table archiving, you have two options when setting the managed item store: 1. Use IAShell to update the database for the application; or 2. Use the REST interface to update the database for the application. The attribute on the holding object is called manageditemstore and is a store object reference. XdbLibrary For every managed item, hold application and policy application that is backed up, a corresponding XdbLibrary object is created in the system data database to represent the backup of the managed item. This object has a reference to the content object that was backed up in the managed item store. Since all managed items, hold applications and policy applications for a particular AIP/table are partitioned into a single xdb library, they are represented by an XdbLibrary object. For every partition, there can 38

39 Core Configuration be three xdblibrary objects representing the three types of objects in the managed item database (managed item, hold application and policy application). The XdbLibrary object has a type defined for it called MANAGED_ITEMS. All managed item XdbLibrary objects are set to this type. The XdbLibrary object uses the name of the object to uniquely identify it and is used to determine what partition (AIP/table) is associated with the managed item and what type of object it represents. For a managed item: ManagedItem_<partitionkey> For a hold application: HoldApplication_<partitionkey> For a policy application: PolicyApplication_<partitionkey> Restoring the Managed Database When restoring the managed item database, you must first decide what needs to be restored. Do only a few of the managed items need to be restored or do all the managed items need to be restored? The method used to perform a restoration depends on the state of the managed item database. If the desire to restore the database back to a previous state due to a corrupted managed item database, use the xdblibrary list to restore. If you wish to restore just a few managed items that, for whatever reason, you want to roll back, use the managed item list itself to restore. Using the xdblibrary Identify which managed items you want to restore. To restore the entire database to an empty managed item database (that you have recreated), complete the following procedure: 1. For each application, search for all xdblibraries that have the type = MANAGED_ITEMS. 2. For each xdblibrary, select the restore link and execute a POST. Complete the following steps to restore the entire database : To a managed item database that has the managed items still in the database, and You want to replace it with the backup 1. Under the Services link, search for all xdblibraries that have the type = MANAGED_ITEMS. 2. For each xdblibrary, select the detach link and execute a POST. 39

40 Core Configuration 3. For each xdblibrary, select the restore list and execute a POST. For example: xdb-libraries?type=managed_items You would get the following: "_embedded": { "xdblibraries": [ 10] 0: { "createdby": "sue@iacustomer.com" "createddate": " T11:59: :00" "lastmodifiedby": "sue@iacustomer.com" "lastmodifieddate": " T12:41: :00" "version": 3 "name": "ManagedItem_48a2ea68-7f6b-4633-a44b-87d814bd3ddf" "subpath": "ManagedItem/48a2ea68-7f6b-4633-a44b-87d814bd3ddf" "detached": false "readonly": false "detachable": false "concurrent": false "cachesupport": true "cacheincount": 0 "size": "indexsize": 0 "type": "MANAGED_ITEMS" "aipcount": 0 "aiucount": 0 "closed": false "closerequested": false "xdbmode": "PRIVATE" "_links": { "self": { "href": " 4fed-b74e-9b02fc6e4849" } - " { "href": " } - } - } To detach, call the following: 9b02fc6e4849/detach To restore, call the following: 40

41 Core Configuration 9b02fc6e4849/restore Selected Managed Items You have identified the selected managed items that need to be restored. The partition key to identify the xdblibrary will be based on the AIP/table ID. You have to restore all managed items for a particular AIP or table. You cannot restore individual managed items (for example an AIU) under a particular AIP. The entire AIP s managed items would have to be restored. Complete the following steps to complete the restoration: 1. Find the three xdblibraries that correspond to the partition key. To find them, you would first have to get the AIP or table ID for the partition that you want to restore and issue the following commands to get the three xdblibraries (for example, the partition key is 6b20bb40-671e-4fbe-a b: a. 4fbe-a b/xdb-libraries?name=ManagedItem_ 6b20bb40-671e-4fbe-a b b. 6b20bb40-671e-4fbe-a b c. 6b20bb40-671e-4fbe-a b 2. Once you have the xdblibrary object, issue the detach and restore commands: a. 4fed-b74e-9b02fc6e4849/detach b. Using the Managed Items If the managed item list is available from an application, there are links off the managed item to detach and restore a managed item. The following example illustrates how to restore a managed item: 1. Get the list of managed items from an application. 41

42 Core Configuration 2. Each of the managed items that you want to restore, the links will be on the managed item object: a. 65aa0e309073/detach b. 4c2d-b2ad-272cde64ef4c For each of the policy applications that is off the managed item, there are links off the policy application to detach and restore a managed item. The following example illustrates how to restore a managed item: 1. Get the list of policy applications from the managed item a. bf8a-65aa0e309073/retention-applications 2. Each of the managed items that you want to restore, the links will be on the managed item object: a. a2f9-456d-8268-c6af5d389fba/detach b. a2f9-456d-8268-c6af5d389fba/restore?xdblibraryid=ddaea662-b79a bd0d-5e142440fbe3 For each of the hold applications that is off the managed item, there are links off the hold application to detach and restore a managed item. The following example illustrates how to restore a managed item: 1. Get the list of policy applications from the managed item: a. bf8a-65aa0e309073/hold-applications 2. Each of the managed items that you want to restore, the links will be on the managed item object: a. 42

43 Core Configuration b. bd0d-5e142440fbe3 To restore the managed items, identify what items will need to be restored. Currently, the only way to initiate the restoration is through the REST interfaces. The restoration of the system data at the same time needs to also be done to ensure that the configuration data and the retained and hold sets stay in sync with the managed item database. If you want to manage the backups separately from InfoArchive, do not configure the managed item store and backup the managed item and system data xdb databases separately. 43

44 Core Configuration 44

45 Chapter 5 Creating a Search The Developer is able to create a search template for: A SIP archive A table archive An application can contain multiple search templates. It is responsibility of the search designer to ensure that only the correct groups have access to certain search templates and to ensure that sensitive information is marked correctly during search composition. InfoArchive supports open-ended date ranges in a search. Creating a Search for a SIP Archive Application The Developer has the ability to create a search for a SIP archive application, either from scratch or by importing an existing search (Zip file). 1. Select the application in which you are creating a search. The application should be configured enough to build the search. For example, it is not possible to create a SIP search after manually creating an application without creating the configuration objects (at least an AIC and a query configuration objects). 2. Click Add Search and select one of the following: Create New: Proceed to the next step. Import from ZIP file: Select the ZIP file and click Open. If required: Any permissions specified need to be added to the search. Set the state of the search to Ready. Proceed to step 7. Tip: Exported searches are associated with an application and the application must match. 3. Enter the following information: 45

46 Creating a Search Field Search Name Description Enter a name for the search that is unique within the application. Description The following special characters are not supported: #, <, >, \, /,!, =. Enter a description for the search. Type Indicate whether you are creating a: Primary search, which can be accessed directly by users. Archival Collection Configuration Nested search, which is a search to be linked from other searches to retrieve additional information on a result. Select the AIC. Select the query configuration that will be searched. The query configuration contains information to build the search from (i.e., the list of queryable fields, the partition key, the indexed fields, the field format, the query quota, etc.) and the result page (the list of result fields). 4. Click Next. 5. The Criteria page contains information returned by the query configuration object, which includes: Any fields returned by the query configuration object. The Partition Key column indicates the field designated as the partition key. While not obligatory, a search form that uses a partition key works more efficiently. The Index column indicates fields that are indexed. The Criteria page may be blank if information was not set up in the query configuration object. On the Criteria page: a. In the Show in Form column, indicate which fields should be included in the search form. b. In the Required? column, indicate which fields the user will have to complete prior to running the search. Tip: It is not mandatory to make any of the criteria required, but not specifying any criteria will result in all of the AIUs being returned. For more information, see Query Quota. c. Click Next. 6. The Results page contains the column names returned by the query configuration object. The Results page may be blank if information was not set up in the query configuration object. On the Results page: a. In the Include in Results column, indicate the fields that will be shown as column in the search results. b. For the selected fields, indicate whether the column should include a default sort. c. Click Finish. 46

47 Creating a Search 7. You are now able to further refine the search. Click Save at any time to save your changes. In the Search Form tab, you have the options of: Manually adding fields to the search form. Click Add Form Field. A pallet is displayed that contains the various elements that can be added to the search form. Rearrange the search form by dragging and dropping an element. To remove an element from the search form, click X in the top right corner of the element being removed. To further configure an element, click. For more information, see Configuring Search Form Fields. Selecting fields from a schema to add to the form. Click Select from Schema and repeat step 5. Once you start adding fields, click Preview to see how the search form will appear. 8. In the Result List tab, you have the options of: Manually adding columns to the search results. Click Add Column. To further configure a column, click. For more information, see Configuring a Result Column. Selecting columns from a schema to add to the form. Click Select from Schema and repeat step 6. To remove a column, click X. 9. The Result Detail tab allows you to enter information that will appear in a panel when a row in the search results is selected. In the Result Detail tab, you are able to add tabs. If the results contain a lot of fields, consider organizing them in tabs. To add a tab: a. Select whether you want the tab to appear in a Side Panel or an Inline Panel. Repeating fields can only appear in the inline panel. b. Click Add Tab. c. Click to: Enter a Detail Field Label, Enter a Detail Field Name, or To remove a tab, click X. 10. In the Permissions tab, indicate which group names will be able to access the search form. Updating a Search Template Status to Ready When a search template is created, the status remains in Draft mode until it is updated by the Developer. When a search template is in Draft mode, the End User will not be able to access it until the template s status is set to Ready. 1. Select the application in which the search is stored. 2. Click and select Set to Ready. Whenever you edit a search template, the Status of the template returns to Draft. You are, however, able to update the Status of a search template being edited. 1. The current Status of the search template is displayed beside the name of the template. If any changes have been made, the Status is listed as Draft. Once you have finished editing the 47

48 Creating a Search template, click. Changing any data, including the query or columns, resets the status to Draft, which allows the Developer to test the changes prior to setting the status to Ready. 2. On the Edit Search page, update the Status field to Ready. 3. Click OK. 4. Click Save. Creating a Search for a Table Archive Application The Developer has the ability to create a search for a table archive application. 1. Select the application in which you are creating a search. 2. Click Add Search and select one of the following: Create New: Proceed to the next step. Import from ZIP file: Select the ZIP file and click Open. If required: Any permissions specified need to be added to the search. Set the state of the search to Ready. Proceed to step 5. Tip: Exported searches are associated with an application and the application must match. 3. Enter the following information: Field Search Name Description Type Archival Collection Description Enter a name for the search that is unique within the application. The following special characters are not supported: #, <, >, \, /,!, =. Enter a description for the search. Indicate whether you are creating a primary or nested search. In this section: a. Select a Database the search form will access when executed. b. Select a Schema the search form will access when executed. c. If desired, select a Table the search form will access when executed. Tip: Specify the table if the search will be used to apply a hold to the results. 48

49 Creating a Search 4. Click Create. 5. You are now able to further refine the search. Click Save at any time to save your changes. 6. In the Search Form tab, click Add Form Field. A pallet is displayed that contains the various elements that can be added to the search form. Rearrange the search form by dragging and dropping an element. To remove an element from the search form, click X in the top right corner of the element being removed. To further configure an element, click. For more information, see Configuring Search Form Fields. Once you start adding fields, click Preview to see how the search form will appear. 7. In the Result List tab, click Add Column. To edit a column, click. For more information, see Configuring a Result Column. To remove a column, click X. 8. The Result Detail tab allows you to enter information that will appear in a panel when a row in the search results is selected. In the Result Detail tab, you are able to add tabs. If the results contain a lot of fields, consider organizing them in tabs. To add a tab: a. Select whether you want the tab to appear in a Side Panel or an Inline Panel. Repeating fields can only appear in the inline panel. b. Click Add Tab. c. Click to: Enter a Detail Field Label, Enter a Detail Field Name, or To remove a tab, click X. 9. Use the Query Editor to compose the search components, including the search form, results and result details. For more information, see XForms, XQueries and Query Results. 10. In the Permissions tab, indicate which group names will be able to access the search form. For more information, Exporting Search Results. Editing a Search The Developer is able to edit a search. Any changes to a search in the Ready state causes the search form to revert to Draft mode. 1. Select the application in which the search being edited is stored. 2. Click and select Edit. 3. Edit the search, as desired. 4. Click Save. Deleting a Search The Developer is able to delete a search. 49

50 Creating a Search Searches in use cannot be deleted. Searches are considered in use when a user has: Executed a background search. Applied a hold to the results of a background search To delete a search that is in use: Ensure that the hold set is removed, if you applied a hold to the search results, or On the Background Results tab, click on x to delete the background search. 1. Select the application in which the search being deleted is stored. 2. Click and select Delete Search. 3. When prompted to verify that you want to delete the search, click Delete. Configuring Search Form Fields The Developer is able to further refine a search form field, as well as the error messages that are issued when the user enters incorrect information 1. In the Search Form tab, click to configure a field and the error messages that are issued when the user enters incorrect information. In the Result List tab, click to configure a result column. 2. In the Properties tab, enter the following information, depending on the element being configured: Field UI Control Data Binding Description A read-only field that is set to Input for the search form criteria. For table searches, enter the name of variable in the xdb query. For SIP searches, define the value as a criterion. The name value has to match in order to use the criterion for the SIP search. Field Label Required Tooltip Text Regex Pattern For more information, see Example of Data Binding. Enter the name for the field shown on the form. Indicate whether an input value is required. Enter concise, helpful information about the field that appears in a small hover box when the user hovers the cursor over the control. Enter the regular expression used for validation on the field. The Messages tab allows you to enter an error message in the event that a user entry does not match the regex expression. 50

51 Creating a Search Field Minimum Characters/ Maximum Characters Search Description Enter the minimum and maximum number of characters the user can enter on the search form. Check the box to allow a wildcard search using asterisk in this field. Tip: If a wildcard search is allowed, and a regex pattern is specified, ensure that the regular expression allows for * in the field. Example of Data Binding The following example illustrates how it is defined in the query: declare variable $lastname external := "" The following example illustrates the AIC definition under resources for the PhoneCalls application: <criterias> <name>callstartdate</name> <label>call Start Date</label> <type>datetime</type> <pkeyminattr>pkeys.datetime01</pkeyminattr> <pkeymaxattr>pkeys.datetime02</pkeymaxattr> </criterias> <criterias> <name>callenddate</name> <label>call End Date</label> <type>datetime</type> </criterias> <criterias> <name>customerid</name> <label>customer ID</label> <type>string</type> <pkeyvaluesattr>pkeys.values01</pkeyvaluesattr> </criterias> <criterias> <name>customerlastname</name> <label>customer Last Name</label> <type>hashed</type> </criterias> <criterias> <name>customerfirstname</name> <label>customer First Name</label> <type>string</type> </criterias> <criterias> <name>representativeid</name> <label>representative ID</label> <type>hashed</type> </criterias> <criterias> <name>callfromphonenumber</name> <label>from PhoneNumber</label> <type>string</type> </criterias> <criterias> <name>calltophonenumber</name> <label>to PhoneNumber</label> 51

52 Creating a Search <type>string</type> </criterias> 3. In the Messages tab, enter the messages you want to issue. The messages will depend on the element being configured. 4. Click OK. Configuring a Result Column The Developer is able to further refine a result column from the Result List tab. 1. Click to configure a column. 2. Enter the following information: Field Column Label Column Type Description Enter the label of the column, which will appear as the header in the result details. If Schema Column Name is selected, complete the rest of the fields. If Linked Column (Nested Search) is selected: a. Select a Column Name. b. In the Nested Search Mapping section, click Add. c. Select a Result Column Binding Name and a Search Field Binding Name. d. Click OK. If Linked Column (External URL) is selected: a. Enter the URL. b. In the External URL Parameters Mapping section, click Add. c. Enter an External URL Parameter Name. d. Select an External URL Parameter Name. e. Click OK. If Downloadable Content is selected: a. Select a Column Name. Include in Export If selected. allows the user of the search to export the search results of the selected column. 52

53 Creating a Search Field Column Name Sort Description For a table search, the column name is the column name defined in the row. For a SIP search, the column name is the name of the field. This is the name (case sensitive) that is specified in the criterias section in XML and is also defined in the query XML file: <operands> <name>callstartdate</name>... Indicate whether: Sort will be disabled for the column. Sort will be enabled for the column. Data Type Hide Column The column is to be displayed as a default sort. Select the data type for the column. Indicate if you want the result column to be hidden from the user. Creating a Duplicate Search The Developer is able to create a duplicate of an existing search and refine it to create an entirely new search. 1. Click and select Create Duplicate. 2. Enter the following information: a. A unique Search Name for the new search form. The following special characters are not supported: #, <, >, \, /,!, =. b. A brief Description of the new search form. c. Click OK. The new search form appears in the list of search forms on the Record Search tab. 3. To further refine the new search, click. For more information, see Editing a Search Form. Table-Based Search XForms, XQueries and Query Results The following section provides XQuery examples and how they relate to XForms (i.e., how an XForm can be derived from an XQuery, or how form output can become query input). The section also examines query result configuration for an XQuery. 53

54 Creating a Search Example: First Name and Last Name with External Variables The following XQuery represents a search on a person s first and last names, returning an address, each in a <row> element. Specify the schema, the table and the case matters: declare namespace table="urn:x-emc:ia:schema:table"; declare variable $firstname as xs:string external; declare variable $lastname as xs:string external; for $elem in /BASEBALL/BASEBALL_MASTER/ROW return <row id="{string($elem/@table:id)}"> <column name="lastname">{ $lname }</column> </row> Incomplete Forms Even if an end user executes an incomplete search form, search results will be returned. For instance, if the end user is searching for a person s name, but only enters the last name, the correct XQuery is executed: Incorrect XQuery for $row in //row where $row /firstname = "" and $row/lastname = "Smith" Correct XQuery for $row in //row where $row/lastname = "Smith" XQuery extension functions are used to construct and execute a string to return search results. The extension function to execute an XQuery string (xhive:evaluate) comes with xdb. Other extension functions are incorporated into InfoArchive s search composition functionality, where ia:condition constructs a condition clause for a form field if the field has a value, and an empty string. ia:where concats non-empty condition clauses by, for example. "and" The benefit of this construct is that it gives the developer complete freedom in XQuery construction. The InfoArchive server only has to provide the form output to XQuery to provide the "binding". Nested Searches While the Developer can easily create a nested search for a SIP archive, creating a nested search for a table archiving requires configuration. 54

55 Creating a Search The developer must: Add a column with a header to indicate the associated nested search, and Add an icon in each cell under the header to trigger a nested search with the appropriate parameter settings. For instance, suppose that for each returned city, a user can search for any tourist destinations that city, the resulting table would appear like the following: Street City Tourist Destinations West 44th Street Eighth Avenue New York New York Cicking the lightbulb triggers an additional search for tourist destinations. In order to make this work, the developer needs to know: The number of columns and the column headers. Which columns represent nested searches. For cells that contain a lightbulb, which REST request should be sent to the server for the nested search, which includes any search parameter values (in this case New York ). To assist the developer, InfoArchive can either: Put the information in the XQuery result, or Put the information in the search configuration. If the information is in XQuery, it would be rendered like the following: (<columns><column name="street" label="street"/><column name="number" label="number"/> <column name="city" label="city"/><column label="city Stuff" linkrel="city-stuff" parameters="city"/></columns>, for $row in firstname $row/firstname = lastname $row/lastname = return <row><column>{$row/street}</column><column>{$row/number}</column><column>{$row/city} </column><column> {$row/city}</column></row>) The above query first returns a <columns> element that contains more <column> elements, followed by a number of <row> elements. InfoArchive lets the developer know that, in the Tourist Destinations column, the "linkrel" attribute is a linkrel to follow on the search configuration response, which contains a URL to a nested search (e.g..../nested-searches/{search-id}). The input of the nested search is bound to the query parameters of the REST call. You do not require dedicated column to invoke the nested search. You can bind the nested search to an existing column where the value in the column will be hyperlinked, which avoids sacrificing a column for the nested search. Also, any column can invoke nested search (i.e., values shown in the Side and Inline panels can invoke a nested search as well). Also, any column can invoke nested search (i.e., values shown in the Side and Inline panels can invoke a nested search as well). 55

56 Creating a Search Authorization The Developer can limit the information that an end user sees by: Hiding columns from the result set. Masking a field. Hiding rows: Based on a single row (i.e., the developer decides that a particular row in a table is only accessible by users in certain roles. Based on some criterion (i.e., only users in certain roles are allowed to see rows matching a certain predicate). Multiple XQueries/XForms per Search The developer must associate multiple XQueries and XForms for a search. The following XQuery allows the specific role to see only non-vips, and only their cities, not address details: declare variable $firstname as xs:string external; declare variable $lastname as xs:string external; for $row in //row where $row/firstname = $firstname and $row/lastname = $lastname and $row/vip = false return <row><column>{$row/city}</column></row> Using ANT Tasks to View, Create, Delete or Update XQuery Modules There is no way to view, create, delete or update XQuery modules in the InfoArchive web application. For this reason, a number of ANT tasks were introduced. The following ANT tasks can be run from the tools/applications/<application > directory</application>. ANT Target Description Option view-xquery-modules create-xquery -modules create-xquery-module update-xquery -module Views all XQuery modules on tenant and application levels. Creates all XQuery modules, located under the <application>/xquery -modules directory. Creates an XQuery module from a file. Updates an XQuery module from a file. option = location Required. The location of the file. option = location Required. The location of the file. 56

57 Creating a Search ANT Target Description Option delete-xquery-module Deletes an XQuery module. option = name Required. The name of the XQuery module export-xquery-module Exports the XQuery module object, including all properties to an XML file. option = context Optional. Context can be tenant or application. option = name Required. The name of the XQuery module. option = context Optional. Context can be tenant or application. option = location Required. The location of the output file. As XQuery modules can be stored on both tenant and application level, an XQuery module with the same name can exist on both tenant and application level. For this reason, ANT targets export-xquery-module and delete-xquery-module have the following strategy when no context option is set: If an XQuery module is found on the application level, the operation is applied to this module. If no XQuery module is found on application level, the operation is applied on the XQuery module on tenant level, if found. Background Searches InfoArchive s background search functionality allows a user to execute an asynchronous search. A search must be executed asynchronously when: The search scope is too wide and the result quota for SIP searches will be exceeded. The quota limits the number of results. Reaching the search quota does not necessarily force the search to be run as a background search. Only a time-out (30 seconds) is the only criteria requiring a background search. Quotas are unavailable for table searches. After an initial search times out. If a synchronous search controller is taking too long, the user is notified that it will take time to locate the results of the search. The user can opt to: Cancel the search or Run the search in the background. If desired, the user can change the default search name (also referred to as the order name) and must then submit the search order. It is best to use a unique name for the order to helps locate the results. While the search is running, the user is able to complete other InfoArchive tasks. Once the results of the search are available, the 57

58 Creating a Search Background Results tab appears in the main header. The user is able to view or export the background search results into a.csv file. A user can also opt to execute a search to run in the background when the search form is first selected. Instead of entering the required search parameters and clicking Search, the user would click Run in background. Order Configuration object that factorize a set of properties to apply on order item objects. For SIP asynchronous search, the order is retrieved based on the Search used to create the orderitem. Field id application Description Type: UUID Label: ID Type: Application Label: Application name Application of this order. Type: String Label: Name priority Name of this order. Type: Integer Label: Priority retentionperiod Order Items created from this order configuration will have this priority. Type: Integer Label: Retention Period Order Items created from this order configuration will have this Retention Period if no Retention Policy is set. 58

59 Creating a Search Field retentionpolicy Description Type: String Label: Retention policy orderitempermission Order Items created from this order configuration will have this Retention Policy. Type: Permission Label: Permission to apply on OrderItem Order Items created from this Order configuration will have this Permission. The Clean job removes order items from the system. For more information, see Clean Job. Using the Background Results Tab Once a background search completes its run, the results can be accessed on the Background Results tab. Search results are displayed in a table that contains the following information: Column Submission Date Type Description Indicates the date and time the background search was executed. Possible values include: Search: Displayed when the results are from a background search Export: Displayed when a user exported search results into a CSV file Apply Hold: Displayed when a user applied a hold policy to an item or search results. Application Duration Remove Hold: Displayed when a user has removed a hold policy from an item or search results. Indicates the application. Indicates the time the operation took to finish. Is not updated until the status is complete. 59

60 Creating a Search Column Name Status Description Indicates the name of the task. For removing a hold, after then name of the holdset, the name will either have removeholditems or removehold. Status has one of the following values: Submitted: The request has been submitted. As the framework picks up tasks every 10 seconds, this status may be rare. In Progress: The request is still running. Completed: The task completed with no errors Exception: The task could not be completed. To determine what happened, refer to the ia.log file. Deleting a Background Task 1. For the background search results being deleted, click. 2. When prompted to verify that you want to delete the background search results, click Delete. Tip: Removing background tasks can be cleaned up after a configurable number of days. Exporting Search Results It is possible to select all or only some of the search results to export. The number of items returned during the search is displayed in the Select All button. Once one or all of the search result items are selected, the Export button is displayed. No matter how many search result items have been selected, access the exported files in the Background Results tab. To select all of the search result items for export: 1. Click the Select All button. 2. Click Export and select the export option. 3. Enter a Name for the exported file or use the default name provided and click OK. To select some of the search result items for export: 1. Click the box or boxes beside the items being exported. The number of items selected is calculated beside the Export button. 60

61 Creating a Search 2. Click Export and select the export option. 3. Enter a Name for the exported file or use the default name provided and click OK. Configuring Export Functionality in the InfoArchive Web Application The configurable export feature enables the ability to configure export search results for one or more records. The export functionality can be selectively exposed on the main result master grid or the individual tab on side or in-line panels at the tab level. The Export action can be selectively enabled or disabled per tab. Export configuration can be used to enable users to download search results in a different format with or without associated contents (attachments/blobs). A number of common formats are available out of the box and are installed, as mentioned in the installation step. During search composition in the InfoArchive web application, to add the export action for each tab, the Developer can use the ADD ACTION button at the main result list or at tab level in the result detail section. For more information, see Adding the Ability to Export Search Results. Using the gear button, it can be further configured for each option, by choosing to enable or disable export configuration at any given level. The Developer can include more configuration objects with the search template. Beside common export configuration, each application can have its own special configuration object from which the Developer can choose. Export search result continues to be an asynchronous activity. The option to choose between asynchronous or synchronous options is shown in the UI but is disabled. At runtime, users are given a choice of exporting search results according to the options selected during composition time. When exporting at the tab level in the detail section, one row of data (selected row) is available for the export operation. From the main search result screen, the user is required to select an export option from the Export menu, which contains options selected by the search designer. The download option also depends on the browser settings. For instance, in some browsers, a save as menu may be presented or the download may start when the download button in the background items listing page is selected. The gzip option downloads with a.gz extension. In case of included content or multiple files in the archive for other reasons, the content of the.gz file will be a tar file and the full file extension will be.tar.gz. It is also possible to download as.tar or.zip files instead, assuming the corresponding export pipelines are enabled. Composition (Edit) Mode In the main grid, when user clicks on the RESULT LIST tab, the available actions toolbar will appear and a default configuration (i.e., csv, gzip format with no content) is selected. The Developer may choose to change this setting. The Developer can disable the export feature for the main result listing page or individual tabs. By default, it is off for tabs in the detail section. 61

62 Creating a Search Click next to EXPORT to launch the Export configuration selection dialog (the Developer can disable the export functionality for the main grid and/or individual tab using the top switch): When EXPORT is disabled: 62