Hawaii Energy and Environmental Technologies (HEET) Initiative

Transcription

1 Hawaii Energy and Environmental Technologies (HEET) Initiative Office of Naval Research Grant Award Number N Task 8. ENERGY-NEUTRAL ENERGY TEST PLATFORMS 8.3 Advanced Database Research, Development and Testing (RD&T) Prepared by: University of Hawaii at Manoa, Hawaii Natural Energy Institute September 2016

2 Advanced Database Research, Development and Testing (RD&T) As data streams expand in size, the ability to manage, visualize and apply data weakens. Big data can be found in both structured and unstructured formats. Structured data can be easily parsed and organized in preparation for analysis, a process that can often be automated. Formats for these data include but are not limited to: Comma Separate Values (CSV), Excel (XLS), extensible Markup Language (XML), Geographic Information System (GIS), JavaScript Object Notation (JSON) and other less common structured formats. Unstructured data appear in visual, symbolic, textual, or other non-numeric formats. Recent market research has attempted to quantify the split between structured and unstructured data. Recent estimates indicate that between 80-85% of an organization s data are unstructured. However, most organizations lack the processes and system requirements to effectively manage this information leaving this potential untapped. Unstructured data files do not follow structured data formats and thus inhibit facile aggregation with structured data models. Unstructured data are generally in a text format although video, image and audio files are common as well. There is great value in analyzing unstructured data independently, as well as converting them to a more usable, structured format and combining them with structured data. HNEI subcontracted this subtask, Advanced Database Research Development and Testing (RD&T), to MKThink. With support from RoundhouseOne, MKThink expanded its Database Management System (DBMS) called 4Daptive 1.0 platform a cloud-based data management system for architects, sustainability coordinators, facilities managers, and other advocates for sensor-driven building technology and building occupant health and comfort. This project advanced the capabilities of 4Daptive 1.0, by developing techniques to improve the acquisition, management, and analysis of large volumes of data, both structured and unstructured. The proposed effort assumed prior collection and validation of the data, that is, it did not consider real-time data collection and analysis. The final output of the project is a platform called 4Daptive 2.0. The objectives of the Advanced Database Research Development and Testing project were to: 1. Develop techniques to auto-acquire, manage, and visualize large volumes of data (> 1 billion data points per month) across multiple dimensions of both structured and unstructured data. 2. Develop an advanced Database Management System that follows the DAMA-DMBOK framework: data architecture; data development; database operations; data security; reference and master data; data warehousing and business intelligence (visualization, analysis, reporting); document and content management; meta data; data quality. 1

3 3. Develop techniques to visualize structured and unstructured data in combination to improve usability for future decisions. The effort considered multidimensional data in both the structured and unstructured data sets. The range of the dimensions included cultural, environmental, and technological sources and types. Accomplishments under HEET10 include: Structured data market research The Data Management Association Data Management Body of Knowledge (DAMA- DMBOK) framework was utilized to evaluate software tools and select those that satisfied the data management requirements. From the DMBOK framework, 12 core database and support functionalities (subcategories) were chosen. Over 500 companies and/or software tools were evaluated to determine 75 competitive companies, for which a document was created to compare capabilities. Unstructured data market research Market research was performed for unstructured data types including video, text, image, audio and handwriting. Over 58 companies and/or software tools were evaluated to determine 14 viable companies across five different unstructured data types. Structuring text files Text analysis module Based on research, the relevant technologies for text were found to be classification, data extraction, keyword extraction and sentiment analysis. The process for integrating unstructured text data into 4Daptive was determined to be: 1) collect unstructured text, 2) import text via API, 3) export in data interchange format, 4) analyze data quality, 5) group similar results, 6) create tabular dataset, 7) integrate with database. A prototype of a text analysis module was developed in 4Daptive, with the ability to extract keywords from text sources. Structuring video files Video analysis Based on research, the relevant technologies for video were found to be demographics, dwell analysis, heat mapping, object detection, path mapping and people counters. The process for integrating unstructured video data into 4Daptive was determined to be: 1) collect unstructured video, 2) import video feed to 3rd party program, 3) collect metrics from video feed, 4) create and store tabular dataset in the cloud, 5) integrate tabular data into database. The 3rd party program selected for integration has the ability to extract average occupancy, average dwell time and other metrics from video feeds. 2

4 Flexible feature selection 4Daptive 1.0 was restructured to allow different modules to become active or inactive depending on the users needs. New 4Daptive modules were developed during the project Project Setup module Setup client and project information on the platform to organize data and content securely; create project teams to organize and manage all data and content across a project using security levels. Auto Importer module Using Extract-Transform-Load (ETL) tools found in the Import Scheduler, users can schedule automatic structured data uploads; capability to import 180,000 rows of data from 60 plus device types, tested using the Maui Meadows Smart Grid data provided by HNEI. Manual Importer module Map item attributes to CSV columns and preview prior to data import. Data Grid module Display user selected items, attributes and associated data that exist in a project s repository; sort rows alphabetically and numerically depending on attribute data; filter rows by any attribute and/or value or multiple attributes and/or values. Analyzer module Create data visualizations using numerous aesthetic and quantitative visual customizations associated with representing, exploring and expressing the data; plot values over time, values grouped by category and on scatter plots; create custom formulas to perform calculations on data being visualized; calculate statistics on data being visualized; create data cards containing visualization, contextual metadata, statistics and findings associated with data; summarize numeric columns using statistics. Unstructured text tool Process unstructured text data and output tabular structured data for easy integration with other analysis tools. Event detection tool (EDT) module Utilize customized formulas to identify events in the data to quickly find trends and anomalies; automatically calculate distribution statistics on full data series, a zoomed range or selected events; accurately visualize 1 billion data points in a readable format; tested using the Maui Meadows Smart Grid data provided by HNEI. File exchange module Distribute and share drafts and items of any file type. Inventory manager module Store all findings related to a project all in one location along with the data card to verify the authenticity of the discovery; manage content in one centralized location with content briefcase; share and distribute drafts and other work in progress items across a respective project. Publisher module View and edit PDF documents uploaded to project. 3

5 Nonfunctional 4Daptive capabilities Data Architecture management Utilize 4Daptive s big data capability scalable to at least 500 GB, tested using the Maui Meadows Smart Grid data provided by HNEI. Database Management utilities Utilize cloud-based system with https encryption and an effective replication process with multi-site backup. Data Security management Database management system contains an integrated firewall; 4Daptive 2.0 is compliant with ISO 27001, which is a security management standard that specifies security management best practices and comprehensive security controls. Database Operations management Ability to retain original raw uploaded le for up to two years after its last use. Details of the work can be found in the report prepared by MKThink, Advanced Database RD&T: Task 5 Final Integrated Report, (August 2016), available on the HNEI website. References 1. M. Mosley, DAMA-DMBOK Functional Framework, Version 3.02, September 10, 2008, available on 4