Developing a System for Real-Time Numerical Simulation during Physical Experiments in a Wave Propagation Laboratory

Transcription

1 Developing a System for Real-Time Numerical Simulation during Physical Experiments in a Wave Propagation Laboratory Dr. Thomas Blum Prof. Stewart Greenhalgh Prof. Johan Robertsson Dr. Dirk van Manen Marlies Vasmel ETH-Zurich D-EDRW Geophysics March 27, 2014 Darren Schmidt (presenter) Lothar Wenzel NI Scientific Research & Big Physics

2 The International Team NI Team Barry Hutt Darren Schmidt Balazs Toth Lothar Wenzel ETH Team Christoph Baerlocher Dr. Thomas Blum Prof. Stewart Greenhalgh Heinrich Horstmeyer Prof. Johan Robertsson Dr. Dirk van Manen Marlies Vasmel ` www. 2

3 Problem: Existing Laboratory Limitations Size of physical experiments limit the study of longer wavelengths or lower frequencies Rock Boundaries of physical experiments introduce undesired reflections reflection 3

4 Goals A new lab needs to: Address existing limitations Avoid introducing new barriers Allow for applications outside of geophysics 4

5 Solution Virtualize the Physical Experiment Immerse the physical experiment in a live real-time 3D simulation much like a person wearing a virtual reality suit is plugged into a visual simulation. Simulated Environment Rock 5

6 Solution Virtualize the Physical Experiment Immerse the physical experiment in a live real-time 3D simulation much like a person wearing a virtual reality suit is plugged into a visual simulation. 6

7 Solution Virtualize the Physical Experiment This new system allows Investigation of larger wavelengths Full control over wave propagation Example: Noise cancellation for undesired reflections. 7

8 Solution Virtualize the Physical Experiment This new system allows Investigation of larger wavelengths Full control over wave propagation Example: Noise cancellation for undesired reflections. hear 8

9 Solution Virtualize the Physical Experiment This new system allows Investigation of larger wavelengths Full control over wave propagation Example: Noise cancellation for undesired reflections. cancel 9

10 Solution Virtualize the Physical Experiment This new system allows Investigation of larger wavelengths Full control over wave propagation Example: Noise cancellation for undesired reflections. cancel 10

11 How? The physical components of the experiment are: A 1.5m cube rock sample 1.5m 1.5m Rock 1.5m 11

12 How? The physical components of the experiment are: A 1.5m cube rock sample A 2m cube of water containing the rock sample 2m Water 2m 2m 12

13 How? The physical components of the experiment are: A 1.5m cube rock sample A 2m cube of water containing the rock sample sensors on or close to rock surface o Sensors are equally distributed across six recording surfaces o Each sensor measures pressure sensors 13

14 How? The physical components of the experiment are: A 1.5m cube rock sample A 2m cube of water containing the rock sample sensors on or close to rock surface o Sensors are equally distributed across six recording surfaces o Each sensor measures pressure actuators on or close to water surface o Actuators are equally distributed across six emitting surfaces o Each actuator generates acoustic waves actuators 14

15 How? The physical components of the experiment are: A 1.5m cube rock sample A 2m cube of water containing the rock sample sensors on or close to rock surface o Sensors are equally distributed across six recording surfaces o Each sensor measures pressure actuators on or close to water surface o Actuators are equally distributed across six emitting surfaces o Each actuator generates acoustic waves 15

16 How? The simulation part of the experiment involves: Acquisition: channels Data represents pressure 16-bit resolution 16

17 How? The simulation part of the experiment involves: Acquisition: channels Computation: Large PDE solver Based on a pre-computed Green s functions (two 1K x 1K x 250 matrices ) Formulation requires pressure and velocity data 17

18 How? The simulation part of the experiment involves: Acquisition: channels Computation: Large PDE solver Control: channels Response represents generator values 16-bit resolution 18

19 How? The simulation part of the experiment involves: Acquisition: channels Computation: Large PDE solver Control: channels Real-time constraint: 50 ms cycle time Dictated by the 20 khz sampling rate Includes a complete acquire compute control cycle 19

20 Why. Water? So the system has the Time to act 20

21 Slower Wave Propagation Why. Water? So the system has the Time to act Distance from recording to emitting surface = 25 cm Propagation time in water ~ 170 ms 21

22 Slower Wave Propagation Why. Water? So the system has the Time to act Ability to react 22

23 Slower Wave Propagation Why. Water? So the system has the Time to act Ability to react Impedance differential impacts actuator performance Wave power in water is stronger than air 23

24 Slower Wave Propagation Why. Water? So the system has the Time to act Ability to react Impedance differential impacts actuator performance Wave power in water is stronger than air 24

25 Why. Water? So the system has the Time to act Ability to react Knowledge to respond Wave propagation in water is well-understood Defined by computable Green s functions 25

26 Why. Water? So the system has the Time to act Ability to react Knowledge to respond Wave propagation in water is well-understood Defined by computable Green s functions 26

27 Why the Rock? Predicting wave propagation in water is easy Behavior is known Models are based on linear operators Many computational optimizations are possible 27

28 Why the Rock? Predicting wave propagation in water is easy Predicting wave propagation in rock is hard Behavior is under exploration Models involve inherently non-linear functions Computational complexity is high 28

29 Why the Rock? Predicting wave propagation in water is easy Predicting wave propagation in rock is hard As a result, this real-time system: Leaves the nonlinear behavior to Mother Nature Captures the results at the recording surfaces Manipulates the physical experiment using known models 29

30 Why Pressure & Velocity? Dictated by the physics (i.e. the Green s functions) 30

31 Why Pressure & Velocity? Dictated by the physics (i.e. the Green s functions) velocity pressure 31

32 Why Pressure & Velocity? Dictated by the physics (i.e. the Green s functions) velocity pressure How Do We Get Velocity? 32

33 Challenge 1 How Do We Get Velocity? Option 1: Duplicate Recording Surfaces Benefit o Compute velocity using six additional recording surfaces o New sensors measure pressure along the surface normal = sensor (for pressure) = sensor (for velocity) 33

34 Challenge 1 How Do We Get Velocity? Option 1: Duplicate Recording Surfaces Benefit Work o Compute velocity using six additional recording surfaces o New sensors measure pressure along the surface normal o Compute the velocity using common difference method = sensor (for pressure) = sensor (for velocity) 34

35 Challenge 1 How Do We Get Velocity? Option 1: Duplicate Recording Surfaces Benefit Work Issue o Compute velocity using six additional recording surfaces o New sensors measure pressure along the surface normal o Compute the velocity using common difference method o Sensor data is double and computation is increased. = sensor (for pressure) = sensor (for velocity) 35

36 Challenge 1 How Do We Get Velocity? Option 1: Duplicate Recording Surfaces Option 2: Duplicate Surfaces Using Staggered Sensor Layout Benefit o Sensor data is not doubled o Gaps in one recording surface are filled in the other o Layout still leverages the surface normal = sensor (for pressure) = sensor (for velocity) 36

37 Challenge 1 How Do We Get Velocity? Option 1: Duplicate Recording Surfaces Option 2: Duplicate Surfaces Using Staggered Sensor Layout Benefit Work o Sensor data is not doubled o Gaps in one recording surface are filled in the other o Layout still leverages the surface normal o Reconstruct all missing pressure data using interpolation = sensor (for pressure) = sensor (for velocity) = virtual sensor (for pressure) = virtual sensor (for velocity) 37

38 Challenge 1 How Do We Get Velocity? Option 1: Duplicate Recording Surfaces Option 2: Duplicate Surfaces Using Staggered Sensor Layout Benefit Work Issue o Sensor data is not doubled o Gaps in one recording surface are filled in the other o Layout still leverages the surface normal o Reconstruct all missing pressure data using interpolation o Computation increases. = sensor (for pressure) = sensor (for velocity) = virtual sensor (for pressure) = virtual sensor (for velocity) 38

39 Challenge 1 How Do We Get Velocity? Option 1: Duplicate Recording Surfaces Option 2: Duplicate Surfaces Using Staggered Sensor Layout Option 3: Optimize the Mathematical Model Benefit o Staggered layout is accounted for in the numerical simulation = sensor (for pressure) = sensor (for velocity) 39

40 Challenge 1 How Do We Get Velocity? Option 1: Duplicate Recording Surfaces Option 2: Duplicate Surfaces Using Staggered Sensor Layout Option 3: Optimize the Mathematical Model Benefit Work o Staggered layout is accounted for in the numerical simulation o Velocity is implicitly computed via a new Green s function = sensor (for pressure) = sensor (for velocity) 40

41 Challenge 1 How Do We Get Velocity? Option 1: Duplicate Recording Surfaces Option 2: Duplicate Surfaces Using Staggered Sensor Layout Option 3: Optimize the Mathematical Model Benefit Work Issue o Staggered layout is accounted for in the numerical simulation o Velocity is implicitly computed via a new Green s function o No physical meaning to the new simulation model = sensor (for pressure) = sensor (for velocity) 41

42 Option 3 Optimize the Mathematical Model 42

43 Option 3 Optimize the Mathematical Model velocity pressure 43

44 Option 3 Optimize the Mathematical Model pressure 44

45 Option 3 Optimize the Mathematical Model pressure computed in advance 45

46 Option 3 Optimize the Mathematical Model The new Green s function: Avoids increasing the sensor data Reduces the computation by 50% 46

47 Challenge 2 How to Simulate in Real-Time? Given a cycle time of 50 ms Latency Bandwidth 47

48 Challenge 2 How to Simulate in Real-Time? Given a cycle time of 50 ms Latency o Is an issue for data movement within the system Ethernet USB PCI PCI Express Bandwidth (MB/s) 125 (Gigabit) 600 (SuperSpeed) (x16) Latency (ms) 1000 (Gigabit) <112 (SuperSpeed).7.7 (x4) NOTE: This table includes off-the-shelf bus & computer network technologies specific to instrumentation. 48

49 Challenge 2 How to Simulate in Real-Time? Given a cycle time of 50 ms Latency o Is an issue for data movement within the system o PCI Express bus offers high throughput and low latency o Transfers are synchronized using FPGA technology Ethernet USB PCI PCI Express Bandwidth (MB/s) 125 (Gigabit) 600 (SuperSpeed) (x16) Latency (ms) 1000 (Gigabit) <112 (SuperSpeed).7.7 (x4) NOTE: This table includes off-the-shelf bus & computer network technologies specific to instrumentation. 49

50 Challenge 2 How to Simulate in Real-Time? Given a cycle time of 50 ms Latency Bandwidth o Is an issue for the Green s functions computation 50

51 Challenge 2 How to Simulate in Real-Time? Given a cycle time of 50 ms Latency Bandwidth o Is an issue for the Green s functions computation o Pre-computed Green s functions data is large 1000 sensors & 1000 actuators 1K x 1K Experiment runs for 250 time steps

52 Challenge 2 How to Simulate in Real-Time? Given a cycle time of 50 ms Latency Bandwidth o Is an issue for the Green s functions computation o Pre-computed Green s functions data is large 1000 sensors & 1000 actuators 1K x 1K Experiment runs for 250 time steps {1K x (1K x 250)} sgemv ops 52

53 Challenge 2 How to Simulate in Real-Time? Given a cycle time of 50 ms Latency Bandwidth o Is an issue for the Green s functions computation o Pre-computed Green s functions data is large o Consume 4GB of memory Storage = 4GB 53

54 Challenge 2 How to Simulate in Real-Time? Given a cycle time of 50 ms Latency Bandwidth o Is an issue for the Green s functions computation o Pre-computed Green s functions data is large o Consume 4GB of memory o Computation requires 100TB/s memory bandwidth Estimated with data movement overhead of 10 ms System bandwidth = 100TB/s 54

55 Challenge 2 How to Simulate in Real-Time? Given a cycle time of 50 ms Latency Bandwidth o Is an issue for the Green s functions computation o Pre-computed Green s functions data is large o Consume 4GB of memory o Computation requires 100TB/s memory bandwidth o To perform the computation in real-time Leverage multiple NVIDIA Tesla K40 GPUs Connected via PCI Express Optimize the Green s functions implementation to reduce the number of GPUs required by an order of magnitude Telsa K40: 288GB/s 55

56 Real-Time 3D Simulator Components Emitting surface GPUs PXImc PCIe Switch MXI Gen2 MXI Gen2 Recording surface PXI FPGA LabVIEW Timing/Synch Network over PCIe GPU acceleration PCIe enclosures PCIe switches 56

57 Demo: System Simulation & Green s Functions Computation Simulation of the System Green s Functions Computation Sensor Data 57

58 Special Thanks NVIDIA Jerry Chen Cliff Woolley Duncan Poole One Stop Systems Jaan Mannik Dell Aron Bowan 58

59 References Vasmel et al (2013). Immersive experimentation in a wave propagation laboratory, J. Acoust. Soc. Am. 134, EL492-EL Big Physics at National Instrument s 59