Building Simulation Modelers Are we big data ready? Jibonananda Sanyal and Joshua New Oak Ridge National Laboratory

Transcription

1 Building Simulation Modelers Are we big data ready? Jibonananda Sanyal and Joshua New Oak Ridge National Laboratory

2 Learning Objectives Appreciate emerging big-data needs in building sciences Identify bottlenecks early and devise solutions for unanticipated situations using large amounts of data ASHRAE is a Registered Provider with The American Institute of Architects Continuing Education Systems. Credit earned on completion of this program will be reported to ASHRAE Records for AIA members. Certificates of Completion for non-aia members are available on request. This program is registered with the AIA/ASHRAE for continuing professional education. As such, it does not include content that may be deemed or construed to be an approval or endorsement by the AIA of any material of construction or any method or manner of handling, using, distributing, or dealing in any material or product. Questions related to specific materials, methods, and services will be addressed at the conclusion of this presentation.

3 Outline Background and scope Motivation Big data in building sciences Data from simulations Data from sensors Conclusion 3

4 Sustainability is the defining challenge Buildings in U.S. 41% of primary energy/carbon 73% of electricity, 34% of gas Buildings in China 60% of urban building floor space in 2030 has yet to be built Buildings in India 67% of all building floor space in 2030 has yet to be built 4 4

5 Energy Consumption and Production Commercial Site Energy Consumption by End Use 5 5

6 Whole test buildings for system/building integration research Evaluating emerging energy efficiency technologies in realistic test beds is an essential step before market introduction. Some technologies (whole-building fault detection and diagnostics, etc.) benefit from use of test buildings during the development process. Fleet of Residential Test Buildings Two Light Commercial Test Buildings 6

7 Real demonstration facilities Residential homes 2800 ft 2 residence minutes 50-60% energy savers Heavily instrumented and equipped with occupancy simulation: Temperature Plugs Lights Range Washer Radiated heat Dryer Refrigerator Dishwasher Heat pump air flow Shower water flow 7

8 What is Big-Data? Volume Scale Velocity Streaming Variety Data Types Veracity Quality But still, what is big-data? 8

9 Why do we care? Trending technologies Simulation playing bigger role Parametric analysis determinants Understanding uncertainty Sensor data Unprecedented levels of resolution New control algorithms Calibration Internet of things Demand response B2G integration Common/traditional tools and methods of analysis break 9

10 How big is big data in the building sciences? Depends on Size Capability of tools for conventional analysis What is the purpose of the data? Where the management and analysis of data poses a non-trivial challenge 10

11 For simulation output Scalability, analysis requirements, and adaptability of data storage mechanism to changing needs Unique situations Non standard EnergyPlus time-stamps :00:00 Data movement and network performance Lag, bandwidth, connection stability Logical aspects Synchronization, storage schema, logical partitioning Analytics on the data 11

12 Additionally, for sensor data Fault detection, management, correction 3-sigma rule for automated outlier detection Slopes and trends Physical redundancy Chicken and egg problem Quality control/quality assurance Filtering, statistical filling in, machine learning approaches 12

13 Managing simulation ensembles Uncertainty Simulation input design of experiments Random sampling Uniform sampling Markov order sampling Latin square designs Fractional factorial designs Simulation output Larger in size, post-processing overheads Saving raw output vs. summary Trade-off of re-computing vs. storage and retrieval 13

14 Big-data management Generic considerations Weak to unstructured collection of data units Balance of storage to computational needs (e.g. Cost of unzipping) Access patterns Do you access individual files or groups of files? Do you calculate summaries often? Do you repeat the same calculations? Physical location on disk Can you exploit parallelism? Design for the generic analysis/use case Design for fault resilience 14

15 Data transfer and storage methods Moving big-data is expensive 10 days to move 45TB, only 68 minutes to generate! Logical partitions in data movement Over arching logical data schema Use of parallel file transfer tools like bbcp, GridFTP, rsync Traditionally, building simulation data are csv files Use database technologies Atomicity, Consistency, Isolation, Durability (ACID) SQL vs. NoSQL Compression in databases, row vs. columnar 15

16 Performance metrics for simulation data 15 minute EnergyPlus output 35,040 records of 96 variables, ~35MB 7-8MB compressed, ~20-22% compression rate 200 CSVs inserted into MySQL database, w/ row compression 7M records, MB average Read only: 6.8 MB ALTER TABLE command on a 386 GB table 8 hrs 10 min with 12 partitions >1 week when unpartitioned HADOOP approaches Key-value pairs, no ACID compliance 16

17 17 Comparison of database performance

18 Other considerations Data sensitivity and user permissions Backups Sensor data Simulation data Is it worth it to back up? Provenance Analysis to be applied Change in simulation code Reference for derivative products Workflow Tools 18

19 Case study Parametric ensemble 2 Residential buildings Commercial buildings: medium office, warehouse, stand-alone retail Several supercomputers used Titan (299,008 cores) Frost (2048 cores) Nautilus (1024 cores) Several challenges EnergyPlus on supercomputers File system, data transience and its movement Analysis 19

20 Tipping point Wall-clock Time Data Processors E+ simulations (mm:ss) Size 16 18:14 5 GB :19 11 GB :34 22 GB :22 44 GB :30 88 GB 1, : GB 2,048 1,024 21: GB 4,096 2,048 21: GB 8,192 4,096 20: TB 16,384 8,192 26: TB 32,768 16,384 26: TB 65,536 32,768 31: TB 131,072 65,536 44:52 23 TB 262, ,072 68:08 45 TB 524,288 20

21 Conclusion What is big data? Why should we care? Considerations in working with big-data Analysis requirements Managing big-data Anticipating issues of scale 21

22 Questions? Jibonananda Sanyal