Cooperative Visual Monitoring in Energy- Constrained Wireless Sensor Networks

Cooperative Visual Monitoring in Energy- Constrained Wireless Sensor Networks Zichong CHEN AudioVisual Communications Laboratory École Polytechnique Fédérale de Lausanne (EPFL), Switzerland Thesis supervisor: Prof. Martin Vetterli Thesis co-supervisor: Dr. Guillermo Barrenetxea

Outline Motivation Cooperative sampling framework Cooperative coding for distributed image sources Cooperative coding for distributed video sources Event-driven video coding using smart cameras Conclusions and future directions 2

Wireless visual monitoring CMOS cameras are cheap, emerge in wireless sensor networks (WSNs) Visual access is helpful for monitoring applications: BirdNest [Hicks2008] Environmental monitoring: e.g., avalanche detection The Swiss environmental research using Sensorscope wireless sensor networks [Ingelrest2010] 3

Why camera network? Robustness Risk imposed by the surrounding environment: strong winds, avalanches, or rock slides In surveillance applications, the intruders can destroy monitoring cameras on purpose One camera is not enough! 4

Why camera network? Energy constraint Power source: energy-harvesting solar cells WSN node is becoming smaller and cheaper, hence solar cells must shrink in size Using Motion JPEG, the energy constraint only allows for transmitting less than two frames every minute Multiple cameras can share the energy burden! 5

What this thesis is about? How to design an efficient multi-camera monitoring system Base station (BS) Cameras Event Detected scene Time Camera failure Sampling Problem (Part 1): how to coordinate multiple cameras for an optimal event-detection probability? Coding Problem (Part 2-4): how to exploit signal statistics locally and across different cameras for reducing communication costs? 6

Interleaved sampling Two cameras with overlapped views, high redundancy!! Target: maximize the probability to catch a random event?. Camera 1 1 2 Camera 2 τ 2/f EVENT len: Δ Time Optimal strategy: two cameras sample in a symmetric interleaved manner Time Camera 1 Camera 2 8

How to synchronize the network To form the interleaved sampling configuration A distributed synchronization mechanism that adapts to clock drift, camera arrival/loss How: overhearing Tx a beacon when sampling Broadcast graph 9

Self-organized sampling (1) DESYNC Algorithm [Degesys2007]: corrected Tx scheduled Tx Time T Sampling interval of each camera Converge exponentially if there is a Hamiltonian cycle E.g., a fully connected network 6 5 4 1 2 3 10

Self-organized sampling (2) A partially connected network: Find a Hamiltonian cycle (if exists) 4 4 5 3 5 3 1 2 Hamiltonian 1 2 Non-Hamiltonian Three components in addition to DESYNC Find Hamiltonian cycle [Levy2005]: linear time on dense network Coordinate initial schedule order (otherwise converge to sth. else) Recovering algorithms for camera failure or merging 11

How to deploy cameras? Constraint: the broadcast graph must be Hamiltonian Option A: to place cameras evenly in a circle around the area of interest Area of Interest BS Option B: the radio transmission range of each camera should cover at least half of the network 12

CHANNEL time Cooperative Visual Monitoring in Energy-Constrained Wireless Sensor Networks Cooperative coding of multi-view images Transmit the stereo-view images X and Y to BS DiSAC2: Split the coding procedure into successive phases imitate the interleaved sampling configuration X ENCx C 1 BS (X 1, Y 1 ) Y ENCy C 2 BS (X 2, Y 2 ) X ENCx C 3 BS (X 3, Y 3 ) Y ENCy C 4 BS (X 4, Y 4 ) 14

Related to successive refinement theory Single Encoder Case: Decoder gets better image quality step by step using successive descriptions D scene camera D 1 D 1 R 1 additional R 2 D 2 R 1 R 1 +R 2 R D 2 OR R 1 +R 2 For Gaussian source, the {rate, distortion} operates on the R(D) curve at each stage. [Equitz1991] 15

DiSAC2 with Gaussian model X R 1 additional R 3 broadcast Y additional R 2 scene two cameras X 1 Y 1 X 2 Y 2 X 3 Y 3 We prove: For Gaussian sources, the {sum-rate, distortion pair} operates on the Wagner Surface* at each stage, for any rate combinations (R 1, R 2, R 3, ). * A. B. Wagner, S. Tavildar, and P. Viswanath, Rate region of the quadratic gaussian two-encoder source-coding problem, IEEE Trans. Inf. Theory, vol. 54, no. 5, pp. 1938 1961, 2008. 16

DiSAC2 with real images We combine layer decomposition and linear prediction method Two dataset: Church, Park X: Y: X 1 Y 2 X 3 Y 4 Simulation results Surfaces: centralized coding as a bound of distributed coding Curves: operating points of DiSAC2 Within 3dB to the centralized coding bound Church Park 17

Video coding under interleaved sampling Videos from multiple cameras actually form a single video sequence. Camera 1 Camera 2 Independent video coding: each camera does its own job Merge 1/f Time 19

Video coding under interleaved sampling Videos from multiple cameras actually form a single video sequence. Camera 1 Camera 2 Each frame encoded indep. Independent video coding: each camera does its own job Merge 1/f Time Distributed video coding: DISCOVER [Girod2005] No comm. needed between two cameras Wyner-Ziv frame Quantizer Slepian- Wolf Encoder Slepian- Wolf Decoder Reconstruction Multiplexer Keyframe Encoder Intra-frame Encoder Intra-frame Decoder Decoder Side information 20

Video coding under interleaved sampling Videos from multiple cameras actually form a single video sequence. Camera 1 Camera 2 Independent video coding: each camera does its own job Merge 1/f Time Distributed video coding: DISCOVER [Girod2005] Joint video coding: overhearing is cheap in long range communication + _ Color Conv. DCT camera Q 1 IQ 2 Entropy encoder Entropy decoder Tx Rx antenna + IDCT + Switch to 1: Transmitting, Tx on, Rx off. Switch to 2: Overhearing, Tx off, Rx on. [Wang2006] P 21

Experimental setup Two regular cameras capture the datasets, and run algorithm on Sensorcam smart camera platform to get approximate energy profile facade of a building: planar scene suitable for image registration complex depth structure : mimicking two cameras deployed in a distributed manner Algorithm implementation written in C, based on OpenCV and x264 libraries Use computation time to estimate computation consumption Use compression ratio to estimate communication consumption 22

Exp.1: Which video coding scheme is better? Distributed coding and H.264 based joint coding Distributed coding fail to exploit correlation between cameras Sampling rate: <0.02 fps in environmental monitoring versus 30 fps in regular video Registration errors between cameras 23

Exp.1: Which video coding scheme is better? H.264 based joint coding and H.264 based independent coding Energy saving of joint coding in the 2Dparameter space of (GOP, sampling interval) Group of Pictures (GOP): T GOP=4 Error-resilience capability: GOP*T Time Joint coding works better when the sampling rate is low GOP is smaller than conventional videos a constant errorresilience capability of 60min Scene A Scene B 24

Exp.2: Per-camera consumption Solid lines: Per-camera real-time consumption of a two-camera system using joint coding Dashed line: Real-time consumption of a single-camera system with the same event detection probability Per-camera consumption is reduced by 30%-50%, consistent with a factor of camera number 2 Performance loss: Registration error and overhearing cost 25

What we really care in monitoring tasks? Relevant: people, cars, explosion Irrelevant: cloud, light, shadows Irrelevant Relevant Video codec like H.264 is inefficient as it ignore the ``meaning'' of video content! 27

To make camera smarter Wireless cameras nowadays are smart: Processing before transmitting When processing cost is smaller than the reduced communication cost Anomaly detection [Chandola2009] based on pattern recognition Does work 100%: missing many Need to collect events training data for each application 28

Our scheme: Event-driven video coding (EVC) Locate salient pixels in each image frame, and transmit the image fragments marked with saliency Refreshing: Group of pictures (GOP) Too much blocks detected as salient Decoding: an image blending problem Pyramid blending Poisson image editing 29

Simulation result: Compression rate Same quality on the detected events 70% energy saving over H.264! 30

Simulation result: Detection rate Compare with Anomaly detection based video coding (AnVC) Define an event as the appearance of one person 31

Implementation on a real wireless camera Monitor a parking lot using a smart camera Power management: wake-up; take an image and transmit; sleep until next schedule An energy meter connected in the power source to monitor the overall energy 40% energy saving over H.264! 32

EVC under interleaved sampling A large scale dataset with nine cameras 33

EVC under interleaved sampling Scaling behavior of per-camera communication cost Ideally 1/n Interleaved sampling configuration: 1 Camera 1 T s Time 2 Cameras Experiment result 1 2 2T s Time 3 Cameras 1 2 3T s Time 3 Due to increasing interframe variance Analysis with Gaussian model supports the loss term O(log n/n) 34

Multi-camera EVC 3 Cameras phantom frame regular frame 1 2 3 3T s Experiment result T s The loss term is eliminated Time Each camera takes a phantom frame Cost of taking an image is small Phantom frame s event is sent by other cameras Use it in saliency detection 35

Conclusions and future research Cooperative sampling framework for a multi-camera monitoring system Increase system robustness Reduce per-camera energy burden Coding methods for multiple cameras under the proposed sampling configuration Multi-view images Multi-view videos Adapt event-driven coding scheme Future research Fusion of information from many cameras with limited overlapping views MIMO can reduce energy consumption even further 37

Reference J. Hicks et al. (2008), An easily deployable wireless imaging system, Proceedings of the Workshop on Applications, Systems, and Algorithms for Image Sensing F. Ingelrest et al. (2010), SensorScope: Application-specific sensor network for environmental monitoring, ACM Transactions on Sensor Networks Ore, Ø. (1960), Note on Hamilton circuits, American Mathematical Monthly W. Equitz and T. Cover (1991), Successive refinement of information, IEEE Transactions on Information Theory B. Girod et al. (2005). Distributed video coding, Proc. IEEE A. Wang and C. Sodini (2006). On the energy efficiency of wireless transceivers, Proc. ICC V. Chandola et al. (2009), Anomaly detection: A survey, ACM Computing Surveys J. Degesys et al. (2007), DESYNC: Self-Organizing Desynchronization and TDMA on Wireless Sensor Networks, Proc. IPSN Levy et al. (2005), A distributed algorithm to find hamiltonian cycles in g(np) random graphs," Proc. CAAN 38

Thank you! Questions? Twelve Stereo Cameras - pottery figurines of Chinese zodiac animal (AD 618-907)