High Performance Video Artifact Detection Enhanced with CUDA. Atul Ravindran Digimetrics

Transcription

1 High Performance Video Artifact Detection Enhanced with CUDA Atul Ravindran Digimetrics

2 Goals & Challenges Provide automated QC for digital video files with accuracy and minimum false positives Provide real time or better speeds while detecting artifacts Video resolution is increasing at a very fast pace Files with 4K resolution are beginning to appear and they need to be Files with 4K resolution are beginning to appear and they need to be analyzed for errors

3 Current Video Artifact Detection Pattern Based Using Common Detection Methodologies Coded Pixel Values Edge Detection Missing or Unassigned Frame Information Grid Alignment Motion Compensation

4 Beyond Current Detection What happens when the artifact doesn t conform to a known pattern? Transcoding can obstruct exact patterns and modify pixel values Standards conversion and editing can lead to artifacts no longer being aligned to a detection grid

5 Human Beings Don t Miss This It troubled us that well-designed pattern-based algorithms would be thrown off by something as simple as a frame crop or transcode, when a human being could clearly see the visual obstruction still present in the frame.

6

7 So What Steps Are Being Taken? A B C D Pictures Related? (Same Scene?) Visual Obstruction? High Motion?

8 SSIM Structural Similarity Works by comparing structures in two pictures and providing an index value correlating the two pictures (0 is no correlation, while 1 is perfect correlation) Very accurate, but many calculations are required to get a result SSIM computation can be broken down into a series of matrix math calculations and blurring routines

9 Building a Metric Based on HVS 1. Full-Frame Comparisons Gross obstructions 2. Partial-Frame Comparisons Partial obstructions in one or more areas of the frame (most common) 3. Multiple Frame Comparisons Obstructions that continue for multiple frames could be masked otherwise

10 Scene Changes Here we have an obvious scene change Low SSIM between 2 nd and 3 rd frame, but this is normal, because the 1 st and 2 nd and 3 rd and 4 th frames are very highly related with high SSIM values.

11 Windowed Frame Comparison A1 A2 A3 B1 B2 B3 C1 C2 C3 D1 D2 D3 A4 A5 A6 B4 B5 B6 C4 C5 C6 D4 D5 D6 A7 A8 A9 B7 B8 B9 C7 C8 C9 D7 D8 D9 Pictures and Windows Related? (Same Scene?) Visual artifact in one or more Windows? High Motion causing the artifact?

12 Multiple Frame Artifacts Notice the artifacts in frames 2-4, which continue, and depending on the window size, could easily mask themselves, since the artifacts occur in the bookend frames. We need a method to detect this occurrence also, so we expand our reference window

13 Multiple Frame Artifacts We keep a running difference array of FrameA-FrameC similarity index and analyze the values to determine if there is a possible artifact. We look for changes which are greater than the average and less than the scene change values found in the array.

14 Contrast Maps Contrast map created in the pre scan pass by scanning the image pixel by pixel Routine can be configured to look for various kinds of pattern to create the required contrast map (Low, high, horizontal, vertical etc.)

15 CPU limitations 1920x1080 HD frames with has pixels HD frames are around 2MBs/ frame 3 such frames are compared / video frame On an average there are frames/sec frames on an average/second to obtain 3 SSIM values Achieving real time results becomes very difficult

16 Role of GPU Raw images are cached on the GPU, and gets swapped SSIM calculation is split into series of matrix math operations and natively implemented using CUDA c Pre-pass contrast map is parallelized by performing calculation on certain areas and combining the results Separate CUDA cores are used to perform calculation on each pixel of the frame Results can be combined on the fly to generate the final SSIM value

17 Results GPU gives between 30-40x speedup than the CPU counterpart of the same algorithm Can achieve real-time with one CUDA card Benchmarks

18 Key Points Observed Data transfer takes a big portion of the overall execution cycle Caching frames on the GPU increased the performance by a factor of 3X Occupying all the threads of a CPU adds to the gain All the calculations has to be performed on the GPU, only the All the calculations has to be performed on the GPU, only the absolutely required output is transferred back to the CPU

19 CPU to GPU algorithms Every algorithm that would eventually run on both the CPU and GPUs are designed together Key design goal is easy parallelization on the GPU and the CPU We design our CPU algorithms with data structures that can be easily mapped to CUDA C Tested on the CPUs first Ported to the GPUs after desired results are obtained

20 Summary SSIM based artifact detection gives us accurate results with noreference QC Contrast maps helps in finding errors where SSIM based detection is not very effective GPUs gives us the ability to achieve faster than real-time results GPUs gives us the ability to achieve faster than real-time results which is very critical in our line of business