Adhocracy Innovation with Imaging technology Socionext Inc. Hiroyuki Komori July 5th, 2017
Video Steaming in the world Video uploading : 65,000 clip/min, 300 hour/min on YouTube Video viewing : billions clip/day on YouTube ref. Kevin Kelly. 2016. The Inevitable Viking Press 1
The world changing with imaging Imaging Eye 540 million years ago Cambrian Explosion Imaging Technology anyone anywhere any time 21st century Image Explosion 2
Challenges for "Image Explosion Age" Enormous amount of images Real time Power efficiency Dedicated Hardware with Flexibility 3
Adhocracy Innovation Computing Imaging Networking Wants Social problem Ideas Biz partner Academia 4
Socionext focus area 5
Topics 8K TV Solution Ultra Low Latency Codec/Transcoder Image Recognition Free Viewpoint Video Distribution 6
8K TV Solution "Art of Kyoto" Graceful & Delicate Picture Quality 7
4K 8K Broadcast Schedule in Japan CY 2015 2016 2017 2018 2019 2020 Events Rio Olympic Pyeong Chang Olympic Tokyo Olympic Satellite 4K 8K BS (Right Turned) BS (Left Turned) 110CS (Left Turned) 124/128 CS Cable TV 4K Practical Broadcast 4K Practical Broadcast 4K 8K Test Broadcast (BS17ch) 4K Test Broadcast 4K Practical Broadcast 4K 8K Practical Broadcast 4K Practical Broadcast Experimental Activity towards 8K 4K 8K Diffusion Expected Many People Enjoy Tokyo Olympics in 4K 8K IPTV etc. 4K Practical Broadcast Experimental Activity towards 8K Source: http://www.soumu.go.jp/menu_seisaku/ictseisaku/housou_suishin/4k8k_suishin.html 8
8K HEVC decoder SoC "SC1400A" World's First 8K TV Solution for Satellite Broadcasting PCI DUM32H HEVW HEVW BK01 Function Specification D UM64V HEVW HEVW D UM64V Product number Package Process SC1400A BGA (45 x 45mm) 28nm VOC VOC Main memory SiP DDR3-1866 (32bit x4ch) External DDR3-1866 (32bit x3ch) VOC BK02 VOC Video decoder HEVC 8K/60p 10bit (4:2:0) ITU-R BT.2073 (4 Slice Format Encoding) HDMI Rx Fuse HDMI-Tx x4 Picture quality Video output PCIe 8K/60p, HDR HDMI2.0 x4 Gen2(1lane) 9
Ultra Low Latency Codec/Transcoder For the increasing number of real time applications Electronic Mirror Telesurgery Teleoperation 10 Teleconference
Latency Requirement for Electronic Mirror Requirement of Glass-to-glass delay time delay time < 200 ms 11
Latency Requirement for Telesurgery Various experiments in telesurgery revealed an acceptable latency of less than 200 ms within 200 ms Internet Robotic surgical system 12
4K HEVC real-time codec "MB86M30" Ultra low latency HEVC encoding based on - Wave front parallel processing - Slice segment splitting SPI UART I2C GPIO Multi video format encoder (HEVC/H.264/MPEG2) Multi video format decoder (HEVC/H.264/MPEG2) ARM 946 x4 E-Fuse I2S x4 ARM Cortex A7 Quad core L1 I-Cache 32kB x4 L1 D-Cache 32kB x4 Video over lay,osd (2 layer) MMU x4 Power Management Unit Secure boot/ Program protection 4K scaler (Supports mutli channel) De-Interlacer/ (Supports mutli channel) LPDDR4 memory contoroller x6 (16bit data bus x 2) DRAM DRAM DRAM DRAM TS MUX/ DEMUX DMA 1G Ether PCIe gen2 4lane x3 SMPTE424M x4(input/output) (YUV 4:2:2 10bit up to 4K@60p) sflash Function Encoding/ Decoding Transcoding Multiple Channel Operation Power Consumption Specification H.265/HEVC Main, Main 10, Main 4:2:2 10, up to 4K/60p H.264/AVC BP, MP, HP, High 10, High 4:2:2, up to 4K/60p Ultra Low Latency Mode in HEVC HEVC to HEVC, H.264 to HEVC, MPEG2 to HEVC, etc. Up to 4ch @HEVC/H.265 encoding, 1080p60 6.3W (Typical) 13
Transcoder composition Real-time HEVC transcoder demand is increasing due to expanding live video viewing over the Internet Cloud HEVC/H.264 Video Stream Decode Transcoder Encode HEVC/H.264 Video Stream Software Hardware based on Multi-Core 64bit processor SC2A11 based on High efficiency multi-codec MB86M30 14
Image Recognition A solution suitable for Cognitive edge computing 15
Cognitive edge computing "Cognitive edge computing" will be installed in every market and device FA Delivery Agriculture Car Surveillance Robot Drone AR 16
Cognitive edge computing Recognition at edge device Camera Real-time Security & Privacy Communication Cost Communication Environment Power Efficiency Recognition in cloud Camera Cloud 17
Target position Both computing flexibility and power efficiency are necessary for wide application to many cognitive edge devices Power efficiency Dedicated HW Target GPU CPU Computing flexibility 18
Our approach: Vision Processor Unit (VPU) Hybrid of GPU and Dedicated HW is our answer GPU Hybrid Accelerator Flexibility, but High Cost High Power DPA: Data Parallel Accelerator HWA: Hardware Accelerator DPA HWA HWA HWA Dedicated HW Link Memory High performance Low Power, but Low Flexibility Flexibility & Power efficiency 19
VPU Programmability Khronos Group's computer vision API "OpenVX" Compliant Connected graph of visual node Node Link Node Link Node Node Link Node Link DPA HWA #0 HWA #1 HWA #2 Linkmem DDR 20
First VPU embedded into ADAS SoC "SC1810": GDC Integrated SoC solution for ADAS UART USART Gen. Purpose Bus 16bit 16ch DMA External Memory Interface ext. IRQ I²C PWM ADC I²S HS-SPI SPI Video Capture Video Capture Video Capture Vision Processor CAN Cortex -A9 L1-Cache 32/32kB NEON SIMD Cortex -A9 L1-Cache 32/32kB NEON SIMD Internal SRAM Video Capture Video Capture Video Capture L2 -Cache 1MB 3D Engine 2D Engine MediaLB 3-Pin Cortex -A9 L1-Cache 32/32kB NEON SIMD Cortex -A9 L1-Cache 32/32kB NEON SIMD PWR Mgmt Watchdog H.264 Codec Dithering Gamma SIG Unit Ethernet AVB USB 2.0 SD/MMC GPIO Motion JPEG Decode Display Ctrl Display Ctrl Display Ctrl Function CPU 3D Engine 2D Engine Image Recognition Engine Display Controller Video Capture Memories CODEC Specification ARM Cortex TM A9 Quad 1066MHz OpenGL ES PowerVR TM 8XE Series SEERIS Proprietary IP OpenVX TM Vison Processor Unit Proprietary IP 3 outputs 1920 x 1080p (maximum) RGB, YUV, FPD 6 inputs 1920 x 1080p (maximum) DDR3/3L 1866MHz (maximum) 32bit, 64bit H.264 CODEC, Motion-JPEG Decoder 21
Application for ADAS All Around View Monitor Approaching Object Detection by VPU Integrated Display Control 3D Sound Alert 22
Free Viewpoint Video Distribution Challenge for new quality of experience of live video viewing 23
Conclusion "Image Explosion age", in which images come into our life in extraordinary numbers, has come. Requirements for imaging solutions are as follows: Processing power for enormous amount of image data Real time processing Power efficiency These requirements are satisfied with: Dedicated hardware Combination of dedicated hardware and processor with software 24