InfoArchive Technical Deep Dive

Transcription

1 InfoArchive Technical Deep Dive Andreas Kalogeropoulos Tord Svensson Copyright 2016 Dell.. All rights reserved. 1

2 MOMENTUM BARCELONA APP AND WIN! BEYOND SILOS Play the BEYOND Game and win a Raspberry Pi pre-loaded with InfoArchive 2

3 Safe Harbor Disclaimer This presentation contains forward-looking statements as defined under the US Federal Securities Laws. Dell EMC makes no representation and undertakes no obligations with regard to product planning information, anticipated product characteristics, performance specifications, or anticipated release dates (collectively, Roadmap Information ). Roadmap Information is provided by Dell EMC as an accommodation to the recipient solely for purposes of discussion and without intending to be bound thereby, and is subject to change without notice. Roadmap information is Dell EMC Confidential Information, and is provided under the terms, conditions and restrictions defined in the Dell EMC Non- Disclosure Agreement in place with your organization. 3

4 Objectives High-level and dive into details of the architecture Storage considerations Disaster Recovery and High availability Compliance 4

5 JOIN THE CONVERSATION! #MMTM16 ECD SERVICES Take the LEAP personality quiz and win! Genius Labs Garden Level Foyer INTERACT WITH THIS SESSION #infoarchive Connect with us 5

6 EMC INFOARCHIVE An Enterprise Information Archiving Platform that unlocks data of all types, trapped in siloed applications, lowering IT costs, preserving compliance and putting application data to work. Leave No Application Data Behind 6

7 InfoArchive Main Value Propositions Application Decomissioning Active Archiving Information Transformation & Reuse Compliance Engine for All Data Types Cost creep legacy apps Aging portfolio Uncontrolled file shares Merge & Acquisitions Migrations Unsupported software stack Maintenance renewals Loss of talent/knowledge Backup windows too short Production System Performance Scalability Challenges Rising costs due to static data Extended Access Push for agility Information Integration for Improved reuse Mobile strategy Need for Analytics Compliance gaps Centralized compliance control New legislation, e.g. Dodd-Frank, EU Data Protective Directive Uncontrolled file shares Long term access challenges 8

8 InfoArchive High-level Architecture SOURCE APPLICATION connectors XML INGESTION MANAGEMENT VALIDATION A C C E S S REST API ACCESS Table Archiving BATCH SQL/JDBC InfoArchive UI PRODUCTION XML Data Record Archiving SECURITY AUDIT ARCHIVE SERVICES ANALYTICS CONFIG RETENTION CONSUMING APPLICATION File Archiving ENCRYPTION STORAGE PLATFORM SEARCH /EXPORT DATA & ANALYTICS UI LEGACY EMC Isilon, Centera, ECS NAS / SAN Compound Record Archiving HDFS PRODUCTION 9

9 InfoArchive in 2016 RECOGNIZED AS A LEADER IN THE GARTNER MQ FOR STRUCTURED DATA ARCHIVING AND APPLICATION RETIREMENT June 4.1 Sept 4.2 Dec Extreme Archiving New Architecture User Centric Design In-Place Compliance Solutions Clinical Archiving 1.9 Accessibility Synchronous Ingestion and access Clinical Archiving 2.0 HIMVision Cloud-scale capabilities with ECS WORM compliance with Centera Compliance Cloud Storage Capabilities Amazon S3 Customized Search and Export Find what you want Export what you need ediscovery Functionality Collections and production sets InfoArchive for SAP 10

10 Ingestion Extraction (Outside IA) Ingestion (RESTful) Access Table Archiving ETL Process Archive ORACLE, MVS, DB2, etc. Structured Data Data Record Archiving XM L ORACLE, Baan, SAP, etc. Structured Data A SIP container InfoArchive File Archiving Documentum, StreamServe, Sharepoint, etc. Unstructured Content A SIP container Reception Archive Ingest Commit Compound Record Archiving , HrAccess, Siebel, Lotus Notes, etc. Multiple Content A SIP container 11

11 InfoArchive Client Application: Role Based Business Owner Knowledge User Retention Manager Configurator Administrator IT Owner Search, View Dashboard Search, Background Search, Export Search, Create / Apply/Remove Retention, Hold, Approve Purge Compliance Dashboard Compose Collaborate Search, Ingestion, Admin activities Optimized for functional needs Administration Storage, Space, Job, Permission, Encryption View Storage Dashboard 12

12 Retention Consultant/Developer End Administrator user Manager 13

13 InfoArchive and OAIS PRESERVATION PLANNING X M L Descriptive Info INGEST DATA MANAGEMENT Descriptive Info ACCESS queries result sets orders PRODUCER SIP X M L AIP ARCHIVAL STORAGE X M L AIP X M L DIP CONSUMER ADMINISTRATION MANAGEMENT 14

14 Logical Architecture 4.1 XML Archive packages (SIP/Tables) </> Reception / Ingestion MetaData Repository Dashboard REST Services (thru Gateway) Administration (Jobs/Audits/Encryption/ Security) In f oar c hi v e Se r v er System Repository CLI JDBC Driver Search / Export Content/Data Repository SAN/NAS and/or Isilon, Centera, ECS GUI Application Server Content Retrieval Gateway pattern for single point of access and authentication Highly scalable due to stateless authentication Modern pattern for emerging micro services based architecture, Cloud environment friendly OAuth2 Based Stateless authentication using modern JSON Web Token JWT Self contained, signed, secure tokens for Clustering support 15

15 InfoArchive 4.1 Base Architecture ( 3 jars ) IA Web UI AngularJS IA Web Server Spring Boot or.war on Tomcat IA Server Spring Data IA Server REST Services Authentication (OAuth2/JWT) xdb IA xdb XML Database Storage Systems Storage System(s) Metadata, Content and System data 16

16 InfoArchive 4.1 Scalability IA Web UI Cluster IA Web Server IA Web Server IA Web Server Stateless, Standard AppServer load balancing IA Server xdb Database servers Storage Systems IA Server IA Server Storage System IA xdb IA xdb IA xdb Storage System IA Server Storage System Stateless, scale-out And specialization Stateful, scale-out (Last to scale horizontally) Stateful, scale-out (Last to scale horizontally) 17

17 InfoArchive Storage Breakdown System Repository Holding Properties & Configuration (e.g Searches) AIP Information Table Information (with chain of custody) Partition Criteria Configuration Information Security information Separate Compliance Repository : Retention & Hold information (Can grow based on retention rules) Cache-In / Cache-Out MetaData Repository XDB Library PDI XML SIP XML or Table(s) XML RI XML (can grow based on indexes) Content/Data Repository XDB Library Backup RI XML CI Container OAIS Terminology : PDI = Preservation Description information RI = Representation Information CI = Content Information Ingestion logs PDI XML GZIP (optional) Analytic Rendition (optional) SIP GZIP (optional) 18

18 InfoArchive Storage BreakDown and RTO/RPO SYSTEM REPOSITORY < 1% of the original XML (small) *(Can increase based on the number of retention policies) Keep the system/technical information (System) NAS/SAN or SSD Applicative replication for Consistency METADATA REPOSITORY 50% of the original XML- xdb Library *(Can increase with indexes) Keep the XDB segments/libraries of the data for search NAS/SAN Optional applicative replication for RTO CONTENT/DATA REPOSITORY 40% of the original XML - xdb Library.zip Aggregated Content CI.zip (10% of the original XML - OriginXML.zip) Keep all the content of an AIP/ Table WORM or NAS/SAN (*Can also be XDB BLOB) Storage replication for RPO All events/audits (record lifecycle and administrative journal) are created and pushed to an audit holding, hence no increase in system repository. 19

19 Scalability & Sizing Until now, with SIP the sizing has been driven by the demanded ingestion throughput/sla not the search & retrieval activity to serve The key for increasing the global batch ingestion throughput is to be able to run an higher number of concurrent ingestions The search & retrieval workload unlikely to significantly impact the global sizing The ingestion workload profile is very different between structured data and unstructured data Performances in SIP are not sensitive to the already archived volume Each ingestion works in a single xdb partition A search including a partitioning criteria quickly narrows the XQuery scope to a subset of xdb partitions Unstructured data retrieval is processed without any underlying XQuery execution Performance in Table are traditional sensitive to already archive volume and number of concurrent searches HTTP multipart can be used for large files (>2Gb), but we recommend a local file transfer before starting the ingestion (through REST Services) 20

20 23

21 LET US KNOW WHAT YOU THOUGHT Take the Session Survey 1. Open the schedule with the Momentum App 2. Go to the session you attended 3. Open Session Survey 4. Answer the 4 questions and submit. Thank you! 24

22 InfoArchive Sessions Wed Nov 2 nd Room H hrs. Accelerate your Digital Transformation Journey with InfoArchive 0930 hrs. InfoArchive: A Customers Journey 1045 hrs. InfoArchive as Part of your Digital Transformation and Move to the Cloud Journey - Atos 1145 hrs. InfoArchive Deep Dive 1345 hrs. Get Your Documentum, SAP, SharePoint Data Under Control Hackathon - Configuration 1445 hrs. Digital Transformation Journey of Sanofi 1615 hrs. Sneak Peek: Future InfoArchive Functionality Thur Nov 3 rd Room H3 Hackathon Docker 0900 hrs. Financial Services: Regulations and Compliance with InfoArchive 1000 hrs. InfoArchive: Ensuring Big Data Compliance and Reducing Risk with Real Time Analytics Copyright 2016 Dell.. All All rights reserved. 25

23 InfoArchive Technical (Room J) Tue Nov 1 st 0900 hrs. Hands On Lab 1515 hrs. Hackathon Wed Nov 2nd 1345 hrs. Hackathon 1615 hrs. Hackathon: Docker Hands On Lab Thur Nov 3rd 0900 hrs. Hackathon 1100 hrs. Hands On Lab Copyright 2016 Dell.. All All rights reserved. 26

24 27