Data, Data, Everywhere. We are now in the Big Data Era.

Transcription

1 Data, Data, Everywhere. We are now in the Big Data Era.

2 CONTENTS Background Big Data What is Generating our Big Data Physical Management of Big Data Optimisation in Data Processing Techniques for Handling Data Conclusions 2

3 BACKGROUND 3

4 Current Status Maturing Subsea Vehicle Market AUV s moving towards commodity In most categories there is growing choice of supplier. More manufactures moving into the market ROV s at commodity status Multiple manufacturers across categories 4

5 Current Status - Sensor s and Payloads Commodity Market Growing richness in range and suppliers of acoustic sensors. Growing range of Optical and Laser sensors. Rapidly growing range Positioning and Navigation Sensors. Some of the bigger players becoming acquisitive Some of the manufactures starting to connect the dots in their portfolios from sensors to solutions. Some vehicle providers tightly integrating sensors to provide a package solution 5

6 Current Status - Summary Subsea Platforms are commodities ( just not cheap ones) Rich sensor / payload market The toolbox is available Operational Challenges Remain Experienced People. Launch & Recovery ( AUVs) Autonomy The big challenge for the Survey and Inspection contractors is the efficient handling of all this data. Big Data 6

7 BIG DATA 7

8 What is Big Data Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions. 8

9 What Big Data Challenges do we face Data Management. Protect Transport Deliver Efficient Data Handling (Processing of the Data) Rapid Delivery of Results More Automation. Less Intervention Integration of Data Sets Temporal Analysis Products for Clients - Visualisation 9

10 What is our Big Data Imagery Digital Video -> HD Digital Video Photo Stills New Survey and Inspection Sensors Laser Profilers Synthetic Aperture Sonars MBE Water Column Logging Optical Sensors Peripheral Sensors Inertial Navigation Sensors Hydrocarbon Sniffers Magnetometers Forward Looking sonars SVP/ CTD 10

11 WHAT IS GENERATING OUR BIG DATA 11

12 Big Data Generators VIDEO Typical ROV Inspection 3-4 Camera Data Volumes SD per Month DVM 6TB Data Volume HD per Month DVM 11TB Growth in move from SD to HD to 4K??? Move away from Video to Stills? Fast ROV Growth in Video Resolution PHOTO STILLS 172GB per day. ( 1 MB Image) DVM 3.4TB Extensive Processing ( duplications) DVM 12TB Each process results in a new image DVM Data Volume per Month. That s how long we are at sea for. 4K HD Video moving mainstream 12

13 Unique Challenges of Big Data Offshore Cruise lengths of days not uncommon. Typically with limited bandwidth on vessels. Not commercially practical to stream all data to beach. Port calls are expensive down time so need to turn vessel around fast. Port calls are often in in remote locations. Arctic, West/East Africa. Shipping logistics can be difficult or time consuming. No local cloud ingestion point Long duration uptime. 24/7 acquisition. Complex data that needs to be rapidly accessible often with random access. (Jump to the middle of a video file) How and what data products we deliver to clients Clients want results fast!!! 13

14 PHYSICAL MANAGEMENT OF BIG DATA 14

15 Data Storage Optimisation RAID array of disks for redundancy and capacity Academic Definition: Redundant Array of Inexpensive Disks Manufacturers Definition: Redundant Array of Independent Disks 15

16 Data Storage Optimisation - Speed Tiered Storage approach for speed optimisation. Layers of disks optimised for speed or storage. Optimised solutions fast become expensive solutions. 16

17 Data Storage Optimisation - Backup Solution for DR and Archive Replication to another RAID. LTO Tape Drives 6.25TB on a tape 17

18 Data Storage Optimisation - Office Data Live on Vessel for Weeks or Months Data Live in Office for Longer. Final Report issued. Client Comments Office Data Storage may need to be significantly larger s TB Disaster Recovery Solutions. What if the building burnt down. What if the building got flooded. What if the power went off. How will this data be archived for the long term Medium Duration (and who pays?) Data Security requires rigorous processes and redundancy 18

19 OPTIMISATION IN DATA PROCESSING 19

20 Data Processing Optimisation Data is Secure and Backed Up but we need fast access to this data Often software vendors recommend storing data locally on SSD disks for fast access. Non starter in a large Survey Data Centre Danger of data loss. Danger of sync issues. Where is the most recent data Single user access. What if it s a large project and needs multiple user access Not a responsible method of managing data But we still need fast access? 20

21 Data Access Optimisation - Network Workstation PCs with Fast Ethernet connections to server Minimised bottle necks in network to maintain available Ethernet connection from PC to Storage. 21

22 Data Access Optimisation - Software Optimised Software enables cluster processing Processing tasks distributed across multiple cores on multiple PC. Significant time savings on computational intensive tasks. Factor of 10 improvement in a small cluster 22

23 Delivery of Data to the Client Generally there is a requirement to deliver significant number of data products to the client. With Inspection activities that has meant dozens of hard disks or RAID arrays full of video files. 23

24 Evolution of Video Deliverables 24

25 Evolution of Video Deliverables 25

26 Video Deliverables Both VHS Tapes and Digital Video Files are essentially RAW products Digital Video provides significant improvements for data review and QC Possible to sync Digital Video with Survey Data 26

27 Video Deliverables Digital Video integration with Survey Data tends to have a limited life span and user base. Event Listing becomes the product that is circulated. Compact There is a better way Convert the Digital Video into a Geo-referenced Image Image can be loaded into GIS software Desktop or web based Product that can be shared within a company, access available to all that require it. 27

28 Video Processing INPUT: Digital Video and USBL Positioning OUTPUT: Referenced Image for GIS Aerial Photo of Seabed 28

29 Video Processing As-Built photo record of a pipeline. Enables Temporal Comparisons Layers in GIS Automated Classification/ Comparison Comparisons with other platform image data sets (AUV Stills Camera) Benefits Compact deliverable (relatively) Optimised for GIS Optimised for Web delivery Resource that can be widely distributed Longer life. As-Built record Annual Comparisons Spatial 29

30 AUV Image Handling FliMap. Image and Laser Profiling Solution from Helicopter Very close analogy to AUV 50 knots v 4 knots 50m altitude v 4m altitude Still images High resolution topographic profile. Very large image datasets 2Hz - 24 hours =172,000 Images Optimised Image Handling Tools Radiometric and Topographic Corrections GIS friendly outputs. Tiled Images 30

31 AUV Image Handling 31

32 Image Handling Convergence in Products - GIS Tiled Image ROV Video Input AUV - Still Image Input Processing Flows that evolve towards minimal human interaction Challenges Remain Computationally intensive. Optimised SW and HW solution 32

33 Conclusion Big Data is here and its only going to get BIGGER. Rich sensor market of acoustic and now laser/optical sensors. Tools are available for acquiring the data but the processing is the big challenge. Investment in infrastructure for secure management of data is not insignificant. ( Should not be overlooked). New approaches required for the handling of these large data volumes. Emerging techniques for improved handling of Video and creating new geospatial products Convergence in deliverables AUV and ROV GIS friendly products. 33

34 Thank You