Cameras and Image Sensors

Transcription

1 Intelligent Control Systems Cameras and Image Sensors Shingo Kagami Graduate School of Information Sciences, Tohoku University swk(at)ic.is.tohoku.ac.jp

2 Basic Motivation e.g. Vision-based Control of Robots image acquisition (today) image processing (from June 3 and on) robot control 3

3 Lens and Optical Parts Outline Image Sensors CCD / CMOS sensors Integration / Shutter Modes In-Camera Image Processing Image Data Transfer Dynamic Range Enhancement 4

4 Cameras and Image Sensors lens spatial low-pass filter wavelength filter color filter image sensor sensor controller in-camera processor bus interface camera 5

5 Examples USB camera with unremovable lens 6

6 Example 7

7 Example CMOS Imager 1280x1024 pixel 8

8 Camera and Lens Cameras with unremovable lens most of inexpensive web cameras Cameras with removable lens Nikon F-mount (large aperture size) C-mount (small aperture, long flange back) CS-mount (same aperture with C-mount, short flange back) The lens must be selected considering the imager size 1, 2/3 1/2, 1/3, 1/4 1 corresponds approx. to diagonal length D = 16 mm View angle θ determined by D and focal length f tan(θ/2) = D/2f F-number: f / A (A: aperture size) The smaller, the brighter but narrower depth of field 9

9 C-mount / F-mount Lenses C-mount lens F-mount lens 10

10 Pin-Hole Camera Model object image plane pin hole focal length: f image No restriction on the distance from camera to object Limited light amount available (dark image) 11

11 Lens formula More light is available (through finite lens aperture) Restricted distance from camera to object (Once f and s 1 are given, s 0 is uniquely determined) 12

12 Imager size, Aperture size and Focal length θ A D addmissible blur circle f tolerable errors in focal length 13

13 Lens and Optical Parts Outline Image Sensors CCD / CMOS sensors Integration / Shutter Modes In-Camera Image Processing Image Data Transfer Dynamic Range Enhancement 14

14 Solid-State Image Sensor Pixel E C h Electron Conductive Band E V Valence Band Hole 15

15 Minimal Knowledge of Semiconductor Devices S PolySi top view G D n + n + p SiO 2 side section view S G D MOS switch When V G is high, S and G are connected (switch on) V V 1 V 2 R Source Follower Amp. V 2 = V 1 + α Potential Well By applying locally high voltage, electronic charges can be collected 16

16 Photodiode h V > 0 (i.e. inversely biased) n p V = 0 An intuitive interpretation: A Ampere Meter here? photocurrent i: proportional to brightness 17

17 What if ampere meter is used Photocurrent is very weak order of pa ~ fa too susceptible to noise A Difficult to measure millions of pixels at the same time, so time division is mandatory for most of the time, photocurrent is just disposed 18

18 Photo Integration That is why we need integration: : charge i : voltage C: capacitance of the node where the charges are integrated Q C V Photocurrent is integrated over a certain integration time in a pixel while the other pixels are read out 19

19 Shot Noise Fundamental noise in optical measurement: fluctuation in the number of the particles such as electrons and photons Equivalently, 20

20 Noise and Integration time V integration time signal shot noise noise with constant variance (some introduced by integration circuitry) t int With N times longer t int, signal-to-noise ratio (SNR) is multiplied by: with respect to shot noise with respect to other noise 21

21 Effects of Integration The longer the integration time is, the brighter the image becomes (because more photo signal is collected) This is intuitive way of understanding; but it should be understood in terms of SNR Integration time Frame time: Thus high frame rate imaging makes images darker (or more correctly, noisier) Strong illumination may be needed Motion blur is caused when the scene moves fast 22

22 Integration-mode photodiode electrically floating grounded n p E c photo-generated charges E c V 23

23 Schematic Description of Integration V PD C (A) Reset (B) Integration V PD (A) (B) V DD i t 24

24 Potential Description of Integration V PD 0 V DD V PD potential well V PD 25

25 Lens and Optical Parts Outline Image Sensors CCD / CMOS sensors Integration / Shutter Modes In-Camera Image Processing Image Data Transfer Dynamic Range Enhancement 26

26 CCD and CMOS image sensors CCD sensors Special fabrication process Large power dissipation (multiple high voltage required) Difficult to be integrated with computational functionality CMOS sensors Standard CMOS process can be used (but special process is also used for high quality) Low power consumption (single CMOS level voltage) Easy to be integrated with CMOS processing circuits High image quality - high cost Varies from low quality low cost to high quality high cost 27

27 CCD and CMOS image sensors CCD: Charge-Coupled Devices CMOS: Complementary Metal-Oxide-Semiconductor These terms do not refer to photo detecting structures! Fundamental difference is how to readout the signal charge amount CCD sensor: h Q V within pixel CMOS sensor: h Q V within pixel 28

28 CCD (Charge-Coupled Device) PolySi (electrode) V > 0 SiO 2 (insulator) p V > 0 V > 0 V > 0 E c V > 0 E c V > 0 V > 0 V t 29

29 CCD Image Sensor Interline Transfer CCD (IT-CCD) Q vertical CCD horizontal CCD Q V Q-V converter 30

30 Signals in a CCD sensor photodiode CCD Q-V converter C amp. 31

31 Resetting in IT-CCD photodiode CCD reset 32

32 Electronic Shutters in CCD IT-CCD (w/o electronic shutter): integrated charge read pulse integration time IT-CCD (with electronic shutter): integrated charge integration time read pulse reset pulse 33

33 CMOS Image Sensor 3-transistor Active Pixel Sensor (3T-APS) reset switch amp reset signal select switch row select signal column circuit output 34

34 Signals in a CMOS sensor reset switch select switch V in C source follower amp V bias V out Signal Voltage source follower: V out = V in + reset integration reset t 35

35 Shutter Modes While IT-CCDs operate in the global shutter mode, 3T-APS CMOS sensors operate in the rolling shutter mode row1 row2 row3 row1 integration row2 integration row3 integration row1 readout row2 readout row3 readout 36

36 Rolling Shutter Example A spinning propeller taken by an iphone camera 37

37 Techniques for High-Speed Imaging Parallel readout / Parallel ADC column-parallel column-parallel x 2 (upper and lower) Readout Modes sub frame, sub sampling binning (neighbor pixels are concatenated) (semi-)random access Low-noise / High-sensitivity pixels micro lens back-illuminated sensor ADC ADC ADC 38

38 Lens and Optical Parts Outline Image Sensors CCD / CMOS sensors Integration / Shutter Modes In-Camera Image Processing Image Data Transfer Dynamic Range Enhancement 39

39 In-Camera Processing image sensor A/D color processing intensity curve correction data transfer may be in the sensor (chip level, or column level) may be done by software after the image data are transferred 40

40 Color Processing (demosaicing) Can be done by software; but it takes computation time Can be done in camera; but it consumes 3 times transfer bandwidth 41

41 Brightness, Contrast, and Gamma I: pixel value I I x: spatial position x x I I = I + a I = k I I = I I I x x x 42

42 Lens and Optical Parts Outline Image Sensors CCD / CMOS sensors Integration / Shutter Modes In-Camera Image Processing Image Data Transfer Dynamic Range Enhancement 43

43 Data Transfer 8 [bits/pixel] 1 M [pixels/frame] 30 [fps] = 240 M [bps] 8 [bits/pixel] 1 M [pixels/frame] 1000 [fps] = 8000 M [bps] interface max. bit rate IEEE 1394a 400 Mbps IEEE 1394b 800 Mbps USB Mbps USB Mbps Gigabit Ethernet 1000 Mbps PCI Express Mbps / lane Camera Link 2000 Mbps (base config.) 5440 Mbps (full config.) and more (extended config.) 44

44 Lens and Optical Parts Outline Image Sensors CCD / CMOS sensors Integration / Shutter Modes In-Camera Image Processing Image Data Transfer Dynamic Range Enhancement 45

45 Dynamic Range DR [db] 20 log i i upper lower V saturation level noise level integration time t 46

46 Dynamic range and Integration time V integration time Simply modifying the integration time will not contribute to dynamic range enhancement. Commonly used techniques utilize multiple integration times. t 47

47 Dynamic Range Enhancement Example MT9V403, Micron Technologies TX frame time reset/ readout TX Q max Q t1 Q1+Q2 t2 short integration Q1 Q2 i 48

48 References Textbooks on Cameras and Image Sensors: A. Hornberg eds.: Handbook of Machine Vision, Wiley-VCH, R. Szeliski: Computer Vision: Algorithms and Applications, Springer, J. Ohta: Smart CMOS Image Sensors and Applications, CRC Press, E. Hecht: Optics, Pearson Education, (in Japanese) 米本和也 : CCD/CMOS イメージ センサの基礎と応用, CQ 出版社, 相澤清晴, 浜本隆之 ( 編著 ): CMOS イメージセンサ, コロナ社, 黒田隆男 : イメージセンサの本質と基礎, コロナ社, ディジタル画像処理編集委員会, ディジタル画像処理, CG-ARTS 協会,