40K Television. David J. Brady. The Duke Imaging and Spectroscopy Program 1

Transcription

1 40K Television David J. Brady The Duke Imaging and Spectroscopy Program 1

2 The Duke Imaging and Spectroscopy Program 2

3 Traditional Television is a Porthole The displayed image is a literal pixel by pixel map of the captured image. The Duke Imaging and Spectroscopy Program 3

4 Television Can Be Interactive Virtual reality means that the viewer can pan, tilt, zoom, move forward and move side to side. The Duke Imaging and Spectroscopy Program 4

5 Television will be Personal and Panoramic The Duke Imaging and Spectroscopy Program 5

6 The Duke Imaging and Spectroscopy Program 6

7 The Duke Imaging and Spectroscopy Program 7

8 Network Architecture Supports Content Delivery The Duke Imaging and Spectroscopy Program 8

9 央视春晚 People like to watch live events. The Duke Imaging and Spectroscopy Program 9

10 We build cameras for interactive broadcasting The Duke Imaging and Spectroscopy Program 10

11 Duke Array Cameras Argus 360 array COMPI thin camera array 2012 AWARE gigapixel array 2015 hg 500 megapixel cameras The Duke Imaging and Spectroscopy Program 11

12 Camputer Enables Array Cameras to Change From Research System to Scalable Product < Camera Head with Raw Video Data Output + Computer Server Cluster For Storage, Processing, and Video Output Integrated Camera System with Capture, Storage, Processing, and Video Output Target Size 5 W x 5 H x 10 D 12k (100 Megapixel Video)@30 frames-persecond The Duke Imaging and Spectroscopy Program 12

13 Camputer Evolution Initial server-based platform System-on-a-chip (SOC) module-based platform Watts 12.5 Watts per 4k camera sensor $1500 $100 per 4k camera sensor The Duke Imaging and Spectroscopy Program 13

14 Camputer Architecture Acquisition System 1 Processing Engine Camera Interface Camera Interface Control Interface Acquisition System 2 Processing Engine Camera Interface Camera Interface Control Interface Acquisition System n Processing Engine Camera Interface Camera Interface Control Interface Memory Controller Render System 1 Data Predictor Analysis System 1 Data Predictor Rendering Engine Control Interface Rendering Engine Control Interface Video Out Control Interface Analysis Out The Duke Imaging and Spectroscopy Program 14

15 Acquisition System Acquisition System Processing Engine Camera Interface Camera Interface Control Interface Memory Controller The Control Interface manages resource utilization and defines the processing pipeline based on user command and control. Provides a single point of contact for multiple external applications. Manages data flow and processing pipeline based on external requests. Network interface in initial implementation The Camera Interface controls process maps incoming image data into memory. Controls camera settings and operating modes. Maps data from all supported cameras into common data structures for processing. The processing engine performs a sequence of transforms on the incoming data in preparation to handing the output to the Memory Controller. Optimized operation for CPU and GPU-based processing. Multiple processing steps implemented in a pipeline to optimize throughout and memory utilization. The Duke Imaging and Spectroscopy Program 15

16 Render System Render System Data Predictor Rendering Engine Control Interface Video Out Control Interface The Render System is a sequence of steps to combine data from multiple cameras into a common output video stream. The data predictor requests data from the Memory Controller based on requests from the Rendering Engine or Control interface. The complexity of the predictor is primarily a function of interactive latency. The Rendering Engine combines images from multiple cameras into a single image that is streamed out. The rendering engine may be a GPU resource or implemented as an OpenGL application. The Duke Imaging and Spectroscopy Program 16

17 2017 Aqueti Mantis Array Cameras The Duke Imaging and Spectroscopy Program 17

18 Mantis Specifications Total System Field-of-View Instantaneous field of view Total Pixel Count Over System Field-of- View # of 4k Microcameras Frame Rate Processor 70⁰x20⁰ Mantis 70 Mantis ⁰x22⁰ 64 mrad 100 mrad 107 Megapixels by 5600 pixels fps 30 fps 10x NVIDIA TX1 98 Megapixels by 3900 pixels 9x NVIDIA TX1 Bandwidth <800 Mbs H.264 <800 Mbs H.264 <8 Gbs MJEG <8 Gbs MJEG Image sensor 1/2.5 4K Progressive Scan CMOS 1/2.5 4K Progressive Scan CMOS Minimum illumination (lux) Latency SNR Power Operating Temperature Range Storage Temperature Range Humidity Range Moisture Resistance/Rating Color: 0.1LUX B/W: 0.01LUX 400ms Color: 0.1LUX B/W: 0.01LUX 400ms >120 > Watts 260 Watts %-90% 5%-90% IP66 Weight 11 kg 11 kg IP66 The Duke Imaging and Spectroscopy Program 18

19 The Duke Imaging and Spectroscopy Program 19

20 The Duke Imaging and Spectroscopy Program 20

21 The Duke Imaging and Spectroscopy Program 21

22 The Duke Imaging and Spectroscopy Program 22

23 The Duke Imaging and Spectroscopy Program 23

24 The Duke Imaging and Spectroscopy Program 24

25 The Duke Imaging and Spectroscopy Program 25

26 On Camera Processing The Duke Imaging and Spectroscopy Program 26

27 On Camera Processing The Duke Imaging and Spectroscopy Program 27

28 Discrete Luneberg lens Due to limited aperture size diffraction and Geometric aberration Geometric aberration free Luneberg lens The Duke Imaging and Spectroscopy Program 28

29 Discrete Luneberg lens Performance of two layered Luneberg lens approximation The Duke Imaging and Spectroscopy Program 29

30 Challenges for spherical focal surfaces Curved image sensors are not available Focus is over non-planar manifold Focusing motion The Duke Imaging and Spectroscopy Program 30

31 Discrete microcameras array Utilizing mass produced CMOS sensor Focusing independently Monocentric multiscale (MMS) architecture The Duke Imaging and Spectroscopy Program 31

32 System comparison The information density of the MMS architecture is ten times even more than hundreds times larger than traditional lens architecture. The Duke Imaging and Spectroscopy Program 32

33 Potential Improvemetns Too big and bulky. Far from the physical limit Vignetting over the overlapping field Microcamera with high complexity The Duke Imaging and Spectroscopy Program 33

34 Galilean multiscale design Eliminating vignetting Negative power group for aberration cancellation The Duke Imaging and Spectroscopy Program 34

35 Design example Specifications: Effective focal length: f = 25mm Aperture size: F/# = 3 Aptina MT9F002 CMOS sensor 4.25mm 13.10mm 11.20mm 47.64mm The Duke Imaging and Spectroscopy Program 35

36 Comparison of size of optics System MMS with 35% FOV gaps Objective radius Diameter (M) Elements number (M) Optical Volume 20mm 4.25mm L AWARE mm 11.20mm L More than 10 times smaller Note: The mark (M) used on the table represents microcamera The Duke Imaging and Spectroscopy Program 36

37 Thanks! Acknowledgements: David Brady is a partial owner and Employee of Aqueti, Inc. This work was partially supported by - Aqueti, Inc. - Duke Kunshan University - Kunshan Industrial Science Research Park - Intel Corporation The Duke Imaging and Spectroscopy Program 37