Use cases. Faces tagging in photo and video, enabling: sharing media editing automatic media mashuping entertaining Augmented reality Games

Size: px

Start display at page:

Download "Use cases. Faces tagging in photo and video, enabling: sharing media editing automatic media mashuping entertaining Augmented reality Games"

Tobias Miller
6 years ago
Views:

1 Viewdle Inc. 1

2 Use cases Faces tagging in photo and video, enabling: sharing media editing automatic media mashuping entertaining Augmented reality Games 2

3 Why OpenCL matter? OpenCL is going to bring such use cases to new level of user experience: Faster Lower power Experience like never before 3

4 Typical face recognition application workflow User select photos Extract faces Group faces to clusters User tags clusters Suggest tag User uploads photos 4

5 Face detection pipeline Tracking Media input Face detection Feature detection Normalization Output * Tracking is a part of video pipeline only 5

6 Distribution of working time. Large photos 190 photos with 220 faces 8% 17% Input Face detection Feature detection Normalization Output 75% 6

7 Distribution of working time. Small photos 1440 photos with 2040 faces 3% 24% Input Face detection Feature detection Normalization Output 73% 7

8 Distribution of working time. Video 720P 1% 3% 3% video, duration ~ 2.5 min 8% 57% 28% Input Face detection Feature detection Tracking Normalization Output 8

9 We accelerate Face detection Video input (UVD) 9

10 Face detect problem statement Evaluate every rectangle inside image P for face presence Where evaluated rectangles set is gathered over number of downscaled copies of image P k with downscale factor α k (i.e α^k) So that in every image P k all rectangles MxN {R mnk } are evaluated 11

11 Scaling Scaling of image with step α=1.1 Usually about scales per image Linear interpolation Produce several scales per call 12

12 Image scales 13

13 One scale view Rectangle to cell mapping 14

14 Rectangles to check in all scales 15

15 Cascaded detection Output is an answer Face or not face? Yes Yes Face? Face? Face? Yes Face No No No Not a face Rectangle 16

16 Unbalancing due to cascades Detection is only in blue cells 17

17 Challenges Parallelization by scale does not work well on GPU because every scale contains different number of rectangles Rebalancing 18

Integral image scaling Detection cascades Detection

18 Face detection steps Integral image scaling Detection cascades Image integration Integral image scaling Integral image scaling Detection cascades Detection cascades Results merge Integral image scaling Detection cascades 19

19 Image integration 20

20 Detection Based on cascade scheme Input is from rectangle of constant size Every cascade try rejecting non-faces rectangles Output is an answer Face or not face? Yes Yes Face? Face? Face? Yes Face No No No Rectangle Not a face 21

21 Evaluation of rectangle We need to check does rectangle contains a face or not? Size of evaluated rectangle is equal to size of training sample image (for example, [20; 20]) Rectangle is checked by different cascades 22

22 What is not effective Parallel inside single scale or straight by scales Run all cascades till the end on all rectangles in parallel fashion 23

23 If parallel straight by scales Scales are stored one-by-one in linear chunk of memory Information about scales (width, height, stride, offset) is stored in additional structures 24

24 Parallel scales. Continued Scales are introduced by additional OpenCL working space dimension Better, because no overhead on call for every scale, but: 25

25 Challenges Cascade scheme brings a lot of unbalancing very bad for GPU Number of candidates-rectangles in scale can vary from 1 to (image.width trainingsample.width)*(image.height trainingsample.height) 26

26 What is effective Evaluate all rectangles from different scales in parallel (don t use scale number as OCL workgroup dimension) Split evaluation into stages, with rebalancing among OCL workitems after every or couple of stages 27

27 Our solution detail Process several scales of image per one iteration (all if possible, is limited by available memory size) Split work in scales by blocks Number of OpenCL workgroups does not depend on image sizes Forced rebalancing of working tasks every several stages based on queues 28

28 Queues Presented as linear chunk of memory Contain records with description of items to process: block of rectangles single rectangle Allows balancing work among GPU workitems efficiently 29

29 Structure of queue Header A several sub-queues(a number of sub-queues is equal to number of workgroups; one sub-queue ~ one workgroup) 30

30 Structure of queue. Continued All buffers for sub-queues have equal size Header contains a one record per sub-queue: a number of items in sub-queue 31

31 Number of workgroups Does not depends on image sizes, but only on hardware characteristics In our case, we found it experimentally, best performance is obtained with: Number of computing units * N, where N = 6,8,12 32

32 Sub-queues Sub-queues Queue rebalancing Separate kernel for unbalancing Balancing is performed every N (2, 3, 4) stages runs Formed sub-queues have almost equal length

33 GPU speedup Face detection relative performance The higher the better 6 GPU vs CPU Mpx 1MPx 4MPx 6MPx Size of input images 34

34 Modes of face detector Homogeneous Parallel (CPU only) Ocl (GPU only) Heterogeneous Balanced Split 35

35 Homogeneous modes Parallel multi-threaded mode, everything is processed on CPU Ocl - single-threaded mode, everything is processed on GPU using OpenCL 36

36 Heterogeneous modes Balanced mix of parallel and ocl modes. Split uses GPU for small scales (having many rectangles) and CPU for large. Goal: get maximum performance from using both CPU and GPU 37

37 Balanced heterogeneous face detection pipeline CPU face detection Tracking Media input CPU face detection Feature detection Normalization GPU face detection Output * Every Tracking is a part of video pipeline only 38

38 Heterogeneous face detect Detection filter runs in multiple threads. Every thread acquires detector from the pools. First pool contains GPU detector (usually one). Second pool has only CPU detectors. In balanced mode thread acquires one detector per thread (if GPU is not available then use CPU). Split mode acquires GPU detector and then CPU detector in thread sequentially. 39

39 Split heterogeneous face detection pipeline CPU + GPU face detection Tracking Media input CPU +GPU face detection CPU +GPU face detection Feature detection Normalization Output 40

40 Challenges Make sure GPU has work to do all the time TBB is not suitable for that TBB pushes number of items into pipelines and runs them almost simultaneously in the same filters Detection filter can be without work some time and then GPU is idle 41

41 Ways to minimize GPU being idle Make all filters parallel (feature detect example) Make CPU/GPU balancing via queuing Make sure there is always extra item in the queue for GPU 42

42 Profiling AMD APP Profiler 2.2 Parallel Path Analyzer: 43

43 Debugging Copy data back from GPU and compare it with golden results Printf via cl_amd_printf extension The OpenCL Emulator-Debugger from AMD 44

44 Processing time Performance results. APU 250 Photos set (190 images) Sabine(Torpedo) APU Engineering sample Quad Core 1.8GHz 1.88x 0 Parallel Balanced Splitted Mode of detector 45

45 Processing time Performance results. Mobile platform Photos set (190 images) Parallel Ocl Balanced Splitted Mode of detector AMD Phenom II N930 Quad Core 2.00 GHz ATI MOBILITY HD x 46

46 Processing time Performance results. Mobile platform. Continued 700 Photos set (1440 images) AMD Phenom II N930 Quad Core 2.00 GHz ATI MOBILITY HD x 0 Parallel Ocl Balanced Splitted Mode of detector 47

47 Processing time Performance results. Desktop system Photos set (190 images) Parallel Balanced Ocl Split Mode of detector Intel Core i x3GHz AMD Radeon HD x 48

48 Processing time Performance results. Desktop system. Continued 350 Photos set (1440 images) Intel Core i x3GHz AMD Radeon HD x 0 Parallel Balanced Ocl Split Mode of detector 49

49 Speedup on photos Platform Small photos Large photos Sabine(Torpedo) APU Engineering sample Quad Core 1.8GHz AMD Phenom II N930 Quad Core 2.00 GHz ATI MOBILITY HD5650 Intel Core i x3GHz AMD Radeon HD6970 *** times for full face detection pipeline! Platform: Windows 7 x64 50

50 FBUploader 51

51 FBUploader 52

52 CPU / GPU comparison 53

53 UVD UVD is a dedicated video decode processing unit Offloads CPU from the decoding process Reduces power usage 54

54 UVD. Details In Microsoft Windows works via DXVA(DirectX Video Acceleration) API We are using it to decode H.264 video 55

55 UVD. Performance w/o UVD w/i UVD FPS CPU load *Hardware: Sabine(Torpedo), Quad Core 1.8GHz Less FPS Less CPU load Offloads work to GPU 56

56 Processing time Video performance Video 720P CPU + no UVD CPU + UVD GPU + UVD Mode Sabine(Torpedo) APU Engineering sample Quad Core 1.8GHz 1.34x 57

57 Processing time Video performance 140 Video 720P Sabine (Armorhead)APU Engineering sample Quad Core 2.4GHz 1.29x 20 0 CPU + no UVD CPU + UVD GPU + UVD Mode 58

58 Speedup on video Platform Sabine(Torpedo) APU Engineering sample Quad Core 1.8GHz Sabine(Armorhead) APU Engineering sample Quad Core 2.4GHz Video 720P Results are obtained with Viewdle Video application Platform: Windows 7 x64 59

59 Viewdle Video application 60

60 Viewdle Video application. Details Analyzes and indexes video input automatically (based on face recognition) Organizes videos and sorts groups of frames (aka clusters ) and groups of clusters (aka clouds ) according to the faces in the video 61

61 User benefits Processing speedup Full utilization of user's hardware Longer battery life 62

62 OpenCL: why we love it Brings speed of processing to new level Best way to utilize current GPU hardware Protects investment Open standard Fast growing community 63

63 Conclusion OpenCL accelerates Viewdle face recognition solution by 1.6-3x in photos and by 1.3x in videos using AMD GPUs and APUs 64

64 Questions? 65

65 Disclaimer & Attribution The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. There is no obligation to update or otherwise correct or revise this information. However, we reserve the right to revise this information and to make changes from time to time to the content hereof without obligation to notify any person of such revisions or changes. NO REPRESENTATIONS OR WARRANTIES ARE MADE WITH RESPECT TO THE CONTENTS HEREOF AND NO RESPONSIBILITY IS ASSUMED FOR ANY INACCURACIES, ERRORS OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. ALL IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE ARE EXPRESSLY DISCLAIMED. IN NO EVENT WILL ANY LIABILITY TO ANY PERSON BE INCURRED FOR ANY DIRECT, INDIRECT, SPECIAL OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD arrow logo, and combinations thereof are trademarks of Advanced Micro Devices, Inc. All other names used in this presentation are for informational purposes only and may be trademarks of their respective owners. The contents of this presentation were provided by individual(s) and/or company listed on the title page. The information and opinions presented in this presentation may not represent AMD s positions, strategies or opinions. Unless explicitly stated, AMD is not responsible for the content herein and no endorsements are implied.

viewdle! - machine vision experts

viewdle! - machine vision experts topic using algorithmic metadata creation and heterogeneous computing to build the personal content management system of the future Page 2 Page 3 video of basic recognition