IM&T Data Management Strategy Overview March 2013 CSIRO INFORMATION MANAGEMENT & TECHNOLOGY

Transcription

1 IM&T Data Management Strategy Overview March 2013 CSIRO INFORMATION MANAGEMENT & TECHNOLOGY

2 What is the CSIRO Research Data Service? RDS is a newly established service developed by IM&T that delivers a data management capability (people, processes, and technologies) to all of CSIRO Discover It enables CSIRO researchers and others to manage, share, discover and re-use Research Data Assets CSIRO Research Data Assets Share Manage 2

3 Technology capability: Data Access Portal Deposit Describe Cite Restrict License Approve Publish Discover Download

4 What are Research Data Assets? Data collections of value from scientific research that are intended to be retained for future use or reference.

5 What does data management involve? Transformation of research data as follows From data that is: to structured collections that are: Unmanaged Disconnected Invisible Single-use Managed Connected Findable Reusable

6 DAP Workflow

7 CSIRO Logical Collection Manager (LCM) Large unstructured but related sets of research data Key threats to large science data sets are 'silent data loss' and 'silent data corruption'. LCM provides integrated management of these two key threats to research data. My Logical Collections Profiles enable LCM users to designate 'source' and 'destination' locations. Destinations may be segregated by 'size' so for small collections it may be desirable to keep on high speed storage, while large collections are stored on lower cost storage.

8 CSIRO Logical Collection Manager (LCM) 1. Encapsulate the largely unstructured data into virtual file systems or 'DVD's (Virtual DVDs) 2. Maintain an external and searchable index of the virtual DVD. 3. Maintain one or more crypto graphic hashes of the virtual DVD to detect corruption and or loss of data. 4. Enable automatic storage of each logical collection to the physical location(s) of best fit.

9 DAP High Level Design

10 Support capability RDS Support Team Advanced Scientific Computing Outreach Metadata assistance Collection definition Deposit assistance Data Conversion Assistance with structuring data Communications, awareness raising Training Presentations

11 Statistics collections deposited via pilot projects collections deposited since go-live April collections in total Publishing CMIS FNS CASS CPI CMAR CLW ICT Draft Submitted Published Public Published CSIRO only Published Restricted CES CMSE IM&T CESRE CAFHS

12 Integration and Connection 1 2

13 What is the Storage and Compute Cloud (STACC)? STACC is implementing services and supporting frameworks - providing storage and computational acceleration tools, infrastructure, and systems Enabling researchers to create, manage, scale and retrieve virtual IT platforms ; Increasing CSIRO productivity with flexible and higher performing technology creating a continuum scale for computational science and data maximization; Technology Neutral Self-managing Automation Updating technology Protecting research data with rapid, reliable and appropriately structured infrastructure platforms Providing frameworks for the delivery of autonomous research IT tools, including the use of cloud orchestration, high speed data storage and management and HPC service accessibility Right Sizing High performance Self-service Dynamic provisioning 1 3

14 Persistent Auto Managed StACC Visual Un-managed Non-persistent On-demand Compute Non-persistent Virtual Utilities Read only Standard Images/Config E B A C Optimised Storage Manager (OSM) Projects G Profile Driven F D Virtual Private Data Centres Optimised STACC Managed Storage and Compute Logical Collection Manager (LCM) F & 7 Workflow Dependent Optimised Flagships Profile Specific Images Storage and Compute CSIRO and Partner Compute Locations & x X W Y Best geograhpical fit Hypervisors Divisions Z S V U Scalable T High Performance Storage Facility (HPSF) CLOUD ORCHESTRATION Right Sized through Data Categorisation WA QLD ACT VIC 1 4

15 STACC Performance Information Management and Technology (IM&T) OSM

16 Data Categorisation why? New tools and technology being delivered under STACC, are being designed specifically to meet the unique needs of CSIRO research. To support the underlying capabilities being delivered, a data classification framework has been developed to enable information to be stored appropriately, managed automatically and provide the most cost effective methods of storage. Dynamic provisioning of infrastructure through the implementation of storage categories provides researchers with the flexibility to meet changing demands and automatically scale storage as the use of a category increases. Tools optimise data assets - automatically managing the process, calculate growth and future needs for the entire environment and initiate tuning activities based on the use of categories. 1 6

17 Categorisation - accelerating Science workflows: Existing workflows are structured around constrained business systems, using limited bandwidth storage (i.e. USB hard drives) and compute (i.e. high end desktops). As an example, one of the things we have see with the initial implementations of STACC Compute are 2 fold increases in 'bandwidth' in the processing but 10 fold increases in throughput as data is used in place rather than copied back and forth between facilities, desktops, USB hard drives and enterprise storage. So simply by 'categorising' the data and optimising the workflow on a Project, Team or Individual Scientist level we see significant reduction in 'resource utilisation' with an a side benefit of increase in productivity. Data 17 Management Management Briefing

18 Categorisation is a first step in the process.. 18

19 Data Categories (examples) Category Description Default Storage Attributes** Commonly used Categories are listed first: Pre-Quality Assured Data /preqa/ This is data which has been generated by an automated process (eg. robot) but has not yet been subject to a formal quality assurance process to ensure the relevance of the data (ie. might just be white noise and not valuable). Hardware level protection No Tape Backup Standard Disk Source Data/ Input Data /Source/ Reference Data /Reference/ Working Data /Work/ User Data /User/ Common Data /Common/ This is data which is immutable reference data, data which may be difficult, impractical or impossible to recreate. This is the authoritative source data. This data is part of a reference set. It may be impractical to access the reference set from its authoritative source, hence a local copy may be required. Data sets that are currently being operated on This is data associated with a Linux user - temporarily because the user has a private copy of some other collection of data for the purpose of testing, development, quality assurance, reviews, etc This is data associated with a user but is to be shared within a project of group of internal users. Written to long term storage definition here Protected by storage at geographically diverse location Usually read only Written to long term storage Protected by storage at geographically diverse location Daily snapshot 28 days retention Standard disk Hardware level protection No Tape backup Daily Tape backup Retention on Tape for 1 year Standard disk ** Note: The Default storage attributes listed below will be used for that Category, unless a variation for your specific requirements is advised during the design/implementation phase. 19

20 Workflow/Process Map Example Example Data Collection/Processing Workflow Process QA Process Calibrate images Multiple/parallel Researcher Receive raw Data (from client) Terrain Correction Mosaic Images together Cloud Masking Save calibrated/ output data Visualisation (Analysis) Process Data Prepare output/report for customer Publish avail for external Collaboration (via RDS) Partner/Customer COMPUTE Category Processing DATA Category PRE-QA External Data Collection Collect raw Aerial data (Landgate) Transport Raw Data to CSIRO (via USB) CPU & Memory CPU MEMORY IO RAW DATA CPU MEMORY 60TB Annually IO IO SOURCE REFERENCE WORKING EXPORT CPU Visualisation GeoPixie MEMORY Memory IO CPU Process data CPU MEMORY IO Request/ Receive Report Customer) CPU RDS MEMORY Memory & IO IO Data 20 Management Management Briefing

21 Next Steps System usage Consolidate resources to increase focus/activity on data depositing CASS activities Continued presentation to business units management teams raising awareness of Data Management lifecycle StACC releases 2 4 (through until June 2014) targeted workshops Revamp of RDS intranet presence Establishing a Data Management culture in the organisation Holistic approach: create process archive re-use

22 Our partners in developing the RDS CSIRO Astronomy & Space Science CSIRO Land & Water CSIRO Australian Animal Health Laboratory CSIRO Sensor & Sensor Networks TCP