An adaptive hyperspectral imager: design, control, processing and applications

Transcription

1 An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix LAAS/Robotics Antoine Monmayrant LAAS/Photonics Hervé Carfantan IRAP/Signal & Image LAAS-CNRS / Laboratoire d analyse et d architecture des systèmes du CNRS

2 What is color? > Color: The property possessed by an object of producing different sensations on the eye as a result of the way it reflects or emits light 2

3 What is color? > The property possessed by an object of producing different sensations on the eye as a result of the way it reflects or emits light > Which eye? Human eye? Insect eye? Man/s shrimp eye? 3

4 What is color? > The property possessed by an object of producing different sensations on the eye as a result of the way it reflects or emits light > Light: Electromagnetic radiation within a certain portion of the electromagnetic spectrum An energy spectrum: The human por/on 4

5 What is color? > Color: The property possessed by an object of producing different sensations on the eye as a result of the way it reflects or emits light > Light: Electromagnetic radiation within a certain portion of the electromagnetic spectrum > Light through eyes 3 numbers 4 numbers 12 numbers! 5

6 Hyperspectral images: true colors > B&W image (aka intensity image): every pixel encodes the integral of energy over a given wavelength interval > Color image: every pixel encodes the energy captured along three wavelength intervals ( 3 image planes ) > Multi-spectral images: up to a dozen image planes > Hyperspectral images: hundreds of image planes 6

7 Hyperspectral images: true colors > B&W image (aka intensity image): every pixel encodes the integral of energy over a given wavelength interval > Color image: every pixel encodes the energy captured along three wavelength intervals ( 3 image planes ) > Multi-spectral images: up to a dozen image planes > Hyperspectral images: hundreds of image planes Hyperspectral cube 7

8 What for? > The reflected spectrum of a surface is characteristic of its physical nature Various materials Various minerals 8

9 What for? > The reflected spectrum of a surface is characteristic of its physical nature A wide spectrum of applications Astronomy Earth observation Agriculture Gaz detection Forensic Medicine Art paint analysis Microscopy... 9

10 What for? > The reflected spectrum of a surface is characteristic of its physical nature A wide spectrum of applications Astronomy Earth observation Agriculture Gaz detection Forensic Medicine Art paint analysis Microscopy... 10

11 Hyperspectral cameras > Core component: dispersing element Gra/ng (diffrac/on) Prism (refrac/on) > Main difficulty: recovering a 3D data cube with a 2D sensor 11

12 Hyperspectral cameras > Classic technology: imaging a slit through a disperser 12

13 Hyperspectral cameras > Classic technology: imaging a slit through a disperser > Alternative technologies: snapshot hyperspectral imaging Imaging several orders of diffrac/on + tomography-like reconstruc/on 13

14 Hyperspectral cameras > Classic technology: imaging a slit through a disperser > Alternative technologies: snapshot hyperspectral imaging Coded aperture snapshot spectral imaging (CASSI) Compressed sensing reconstruc/on 14

15 Hyperspectral cameras > Classic technology: imaging a slit through a disperser Main issue: sequential acquisition 15

16 Hyperspectral cameras > Classic technology: imaging a slit through a disperser Main issue: sequential acquisition > Alternative technologies: snapshot hyperspectral imaging Several issues High CPU load to produce the HS cube Performances fixed by design Tough calibration issues 16

17 Hyperspectral cameras > Classic technology: imaging a slit through a disperser Main issue: sequential acquisition > Alternative technologies: snapshot hyperspectral imaging Several issues High CPU load to produce the HS cube Performances fixed by design Tough calibration issues All approaches produce heavy data (~ 1 Gb): To acquire To process To transmit 17

18 An adaptive hyperspectral imager > Principle 18

19 An adaptive hyperspectral imager > Principle (illustration) A partial masking of the signal defines a spectral filter 19

20 An adaptive hyperspectral imager > Principle (illustration) A partial masking of the signal defines a spectral filter 20

21 An adaptive hyperspectral imager > Principle (illustration) A partial masking of the signal defines a spectral filter 21

22 An adaptive hyperspectral imager > Principle (illustration) A partial masking of the signal defines a spectral filter 22

23 An adaptive hyperspectral imager > Principle (illustration) A partial masking of the signal defines a spectral filter 23

24 An adaptive hyperspectral imager > Principle (illustration) A partial masking of the signal defines a spectral filter 24

25 An adaptive hyperspectral imager > Principle (illustration) A partial masking of the signal defines a spectral filter 25

26 An adaptive hyperspectral imager > Principle (illustration) A partial masking of the signal defines a spectral filter 26

27 An adaptive hyperspectral imager > Principle (illustration) For the same mask, different spatial rays are differently spectrally filtered 27

28 An adaptive hyperspectral imager > Principle (illustration) For the same mask, different spatial rays are differently spectrally filtered 28

29 An adaptive hyperspectral imager > Design: dual 4f-line "Single-shot compressive spectral imaging with a dual-disperser architecture", Ghem et al. Op/cs Express

30 An adaptive hyperspectral imager > Design: dual 4f-line Digital Micro-mirror Device 30

31 An adaptive hyperspectral imager > Design: dual 4f-line % :$:;<3<:=>8?<;4:3 B475<?<C8B43678%$6;<! #! # " " # $! %&#'()*+,-. & ) " * / 0#1 2# $ 1 31

32 An adaptive hyperspectral imager > Properties Possibility to acquire an intensity image 32

33 An adaptive hyperspectral imager > Properties Possibility to acquire an intensity image Colocation 33

34 An adaptive hyperspectral imager > Properties Possibility to acquire an intensity image Colocation Independence of lines 34

35 An adaptive hyperspectral imager > Properties Possibility to acquire an intensity image Colocation Independence of lines Simplicity of the model 35

36 An adaptive hyperspectral imager > Properties Possibility to acquire an intensity image Colocation Independence of lines Simplicity of the model Programmable acquisition Various acquisi/on schemes 36

37 First proof of concept > Design implementation 37

38 First proof of concept 38

39 First proof of concept > Scanning a slit on the DMD %)""" 9"" 7""!"# :)*+,-./0)12) "" $"" '"" #"" &""!"" %"" &'' Imaged object: known light source through a ()*+,-./0)12) $"" '"" #"" &""!"" %""!"" #"" $"" 7"" %)""" %)!""!$#!%# ()*+,-./0)12) &'' 7!" 7&" 7#" 7'" 7$" 78" 77" ;4<./.2=>?)*250 bi-λ illumination 39

40 Overview of the system possibilities > The control of the DMD allows to: Configure the system for a given purpose / characteristic Control the system to optimize its output (active perception paradigm) > Taxonomy of acquisition schemes, along 2 dimensions 1. Type of recovered information (defined by operational goals) - Full HS cube - Specific information (e.g. a set of given spectra) 2. Way to control the system - Pre-configured DMD patterns - On-line controlled / adapted patterns > Criteria to consider for application contexts: Number of acquisitions Computational load to recover the information Quantity/quality of recovered information 40

41 Overview of the system possibilities > Example 1: scanning slit Fills the whole cube 41

42 Overview of the system possibilities > Example 2: monochromatic imager Random access to any spectral plane in one acquistion 42

43 Overview of the system possibilities > Example 3: spectrum response image Exhibits the presence of a given spectrum in one acquisition Monochromatique Corrélation spectrale > V > > V V > V V > V > V V DMD DMD DMD DMD DMD > V > > VV V > V V > V > V V CMOS CMOS CMOS CMOS CMOS λ λ λ Monochroma/c image Spectrum correla/on 43

44 Overview of the system possibilities > Example 4: generalized bayer mosaics Numerous possible patterns DMD DMD DMD CMOS CMOS CMOS 44

45 Overview of the system possibilities > Example 5: near-snapshot partitioning 2 acquisitions: 1 panchro, 1 controlled 45

46 Overview of the system possibilities > Example 6: quadratic regularization Recovering the full cube from a small set of random DMD acquisitions (PhD of Ibrahim Ardi, IRAP/LAAS) Reconstruc/on of a 31 wavelengths cube from 5 images 46

47 Latest prototype > Really engineered > Images the visible nm > Integrated acquisition & DMD control 47

48 Earth observation applications? > Configurability yields a series of trade-offs in terms of: Number of acquisitions Computational load to recover the information Quantity/quality of recovered information 48

49 Earth observation applications? > Configurability yields a series of trade-offs in terms of: Number of acquisitions Computational load to recover the information Quantity/quality of recovered information > Matrix sensor in a push-broom configuration: Time delayed integration Real-time adaptive control of the acquisitions TDI (acquisi/on n) TDI (acquisi/on 2) TDI (acquisi/on 1) 49

50 As a conclusion > Co-design: interdisciplinary cross fertilization Op/c systems design Signal & image processing Adap*ve hyperspectral imager Ac/ve percep/on paradigm 50