methods for resizing medical images for tablets

Transcription

1 Retina Visual Interpretation Displays: Image with Three-Dimensional interpolation Annotations (VITA) : Open source automated 3D visual summary application using AIM (Annotation Imaging Markup) tablets enabled PACS based on radiologist annotations methods for resizing medical images for Sharmili Roy 1, Michael S. Brown 1, Yeang Chng, George L. Shih 2 1 School of Computing, National University of Singapore 2 Weill Cornell Medicine College (New York), Cornell University

2 Retina Displays Apple recently launched the new ipad4 named ipad with Retina Display Packs 3.1 million pixels in a 7.3 X 9.5 display Reaches visual acuity: Pixel density is so high that human eye cannot distinguish individual pixels

3 Image Resampling: Motivation Typical MR and CT Images are still at 256 X 256 to 1K X 1K pixels Retina display pixel density >> Medical image pixel density Need to fill around 2 million pixels to fit the retina display

4 Ipad 4 Retina Display Original Image 512 X 512 Resize

5 Image Resampling: Motivation Images are manipulated in 3 stages in radiology workflow Image Acquisition Converts proprietary image formats to DICOM-compatible bitmaps Typically involves no interpolation Quality Control Performed by technologist right after image acquisition Rarely deals with any data resampling if protocols are done right Radiological Viewing ( reading ) Most image interpolation done at this stage

6 Image Resampling: Motivation During radiological viewing, images are automatically zoomed to fit hanging protocols (view layouts) manually zoomed to magnify ROI (Region Of Interest) scrolled occasionally flipped and rotated in multiples of 90 degrees Each of these actions require image resampling/interpolation Can significantly degrade diagnostic quality if not done right

7 Image interpolation is extremely important in routine clinical practice

8 How best can we fill the missing pixels? Rescaled Image Resize Original Image 512 X 512

9 Nearest Neighbor Interpolation Bilinear Interpolation Bicubic Interpolation

10 Nearest Neighbor Interpolation 1 1 r c (ii,jj) ( j 1)*( c 1) jj round 1 ( c' 1) ( i 1)*( r 1) ii round 1 ( r' 1) c Map each output pixel location to the input image and use that pixel value r (i,j)

11 Nearest Neighbor Interpolation 1 1 r c (ii,jj) ( j 1)*( c 1) jj round 1 ( c' 1) ( i 1)*( r 1) ii round 1 ( r' 1) Map each output pixel location to the input image and use that pixel value 1 c r (i,j)

12 Nearest Neighbor Interpolation 1 r (ii,jj) Net effect is enlargement of the original pixels 1 c r (i,j)

13 Nearest Neighbor Interpolation Original Image 512 X 512 Rescaled Image Resize Nearest Neighbor

14 Nearest Neighbor Interpolation Simplest and fastest image resizing algorithm Low computation cost Does not alter the original data values Poor image quality due to pixelation/ blocky effect Stair stepped effect around diagonal lines and curves

15 Bilinear Interpolation Average Consider closest 2x2 neighbourhood of known pixels surrounding the unknown pixel Compute weighted average of these 4 pixels and assign to the unknown pixel Weights are proportional to the distances of the known pixels from the unknown pixel

16 Bilinear Interpolation Given pixel values at p, q, r and s we want to compute the value at c t s c p q

17 Bilinear Interpolation Given pixel values at p, q, r and s we want to compute the value at c t s First perform linear interpolation between p and q to obtain value at a a = (d x * q + d x * p)/(d x + d x ) d x c p d x a q

18 Bilinear Interpolation Given pixel values at p, q, r and s we want to compute the value at c t b s First perform linear interpolation between p and q to obtain value at a a = (d x * q + d x * p)/(d x + d x ) Perform another linear interpolation between t and s to obtain value at b b = (d x * s + d x * t)/(d x + d x ) p d x c d x a q

19 Bilinear Interpolation Given pixel values at p, q, r and s we want to compute the value at c First perform linear interpolation between p and q to obtain value at a a = (d x * q + d x * p)/(d x + d x ) Perform another linear interpolation between t and s to obtain value at b b = (d x * s + d x * t)/(d x + d x ) Finally interpolate between a and b to arrive at c c = (d y * b + d y * a)/(d y + d y ) t p d x b d y c d y dx a s q

20 Bilinear Interpolation Original Image 512 X 512 Rescaled Image Resize Bilinear

21 Bilinear Interpolation More smoother looking images than nearest neighbor interpolation Fast and simple No ringing effect Blurring effect caused by the averaging of the neighboring pixels

22 Bicubic Interpolation Average Average Consider closest 4x4 neighbourhood of known pixels surrounding the unknown pixel Compute weighted average of these 4 pixels and assign to the unknown pixel Weights are proportional to the distances of the known pixels from the unknown pixel

23 Given a 4X4 neighbourhood, Bicubic Interpolation

24 Bicubic Interpolation Given a 4X4 neighbourhood, we want to compute the value of the unknown pixel

25 Bicubic Interpolation Given a 4X4 neighbourhood, we want to compute the value of the unknown pixel Fit 4 cubic functions F(0), F(1), F(2) and F(3) in the horizontal direction F(0) F(1) F(2) F(3)

26 Bicubic Interpolation Given a 4X4 neighbourhood, we want to compute the value of the unknown pixel Fit 4 cubic functions F(0), F(1), F(2) and F(3) in the horizontal direction Using these functions compute values at a,b,c and d a b c d F(0) F(1) F(2) F(3)

27 Bicubic Interpolation Given a 4X4 neighbourhood, we want to compute the value of the unknown pixel Fit 4 cubic functions F(0), F(1), F(2) and F(3) in the horizontal direction Using these functions compute values at a,b,c and d Fit another cubic function through a,b,c and d a b c d F(0) F(1) F(2) F(3)

28 Bicubic Interpolation Given a 4X4 neighbourhood, we want to compute the value of the unknown pixel Fit 4 cubic functions F(0), F(1), F(2) and F(3) in the horizontal direction Using these functions compute values at a,b,c and d Fit another cubic function through a,b,c and d Compute unknown pixel value a b c d F(0) F(1) F(2) F(3)

29 Bicubic Interpolation: Fitting Cubic Function Given a 1, a 2, a 3, and a 4, what is the value of a x? a 1 a 2 a x a 3 a 4

30 Bicubic Interpolation: Fitting Cubic Function Given a 1, a 2, a 3, and a 4, what is the value of a x? Define: s k = a x a k s1 s 3 a 1 a 2 a x a 3 a 4 s 2 s 4

31 Bicubic Interpolation: Fitting Cubic Function Given a 1, a 2, a 3, and a 4, what is the value of a x? Define: s k = a x a k s1 s 3 a x = Ʃ k a k *u k a 1 a 2 a x a 3 a 4 s 2 s 4

32 Bicubic Interpolation: Fitting Cubic Function Given a 1, a 2, a 3, and a 4, what is the value of a x? Define: s k = a x a k s1 s 3 a x = Ʃ k a k *u k = { 1.5* s k 3 2.5* s k 2 + 1; 0<= s k < 1 u k = { -0.5* s k * s k 2 4* s k + 2; 1<= s k < 2 = { 0; 2 < s k a 1 a 2 a x a 3 a 4 s 2 s 4

33 Bicubic Interpolation Original Image 512 X 512 Rescaled Image Resize Bicubic

34 Bicubic Interpolation Generates sharper image compared to nearest neighbor and bilinear Ideal combination of computation cost and output quality Used in many image editing programs including Adobe Photoshop, GIMP, Printer drivers and in-camera interpolation Slower than nearest neighbor and bilinear interpolation May lead to ringing artifacts due to overshoot of pixel values caused by averaging

35 Comparison: Nearest Neighbor Interpolation Nearest Neighbor

36 Comparison: Bilinear Interpolation Bilinear

37 Comparison: Bicubic Interpolation Bicubic

38 Comparison: Bspline Interpolation Bspline

39 Comparison: Lanczos Interpolation Lanczos

40 Image Interpolation in ios Core Image : ios Application Developer Interface for Image Processing Core Image provides only Nearest Neighbor and Bilinear (default) image interpolation Developers may consider writing custom image interpolation routines for optimal results Graphics Video Image I/O Built-in Filters Custom Filters Core Image Run Time Input Image Core Image API Output Image

41 Core Image Custom Filter Attributes, Output Image and Kernel form the three methods of a filter Attributes: Returns a list of key-value pairs (parameters) that describe the filter Output Image: Produces the output image Attributes Output Image Kernel Methods of Typical Filter Kernel: Specifies computations to be performed in each pixel of the source image

42 Core Image Pixel Processing Path Input Image Sampler fetches pixels from the source image and provides to kernel Core Image runtime performs per-pixel image calculations as defined in the kernel Filter Sampler Kernel Output Image

43 Core Image Calculation Path Pixel Processing Path: Source Image -> Destination Image Image Calculation Path: Destination -> Source Image!! Computation in Core Image runtime begins at the destination and works its way back to the source pixels This actually minimizes the number of pixels used in any calculation

44 Core Image Calculation Path By looking ahead what the destination ought to be, Core Image determines which data from the source image affects the final image Restricts calculations only to these source pixels The sampler samples only the selected pixels from the input image for processing Sampler Kernel Filter

45 Custom Filters as Image Unit Custom filters can be made available to other applications by packaging them as a plug-in or a structure called image unit Image units can contain multiple filters Other applications can load the image unit using Core Image API and access the filters Description Property list Load Method Image Unit MyFilter1 MyFilter2 MyFilter3

46 Take Home Messages Developers APIs for ios (ipad and iphone) and Android tablets typically provide only nearest neighbor or bilinear interpolation Bicubic interpolation is better than nearest neighbor and bilinear interpolation More sophisticated interpolation algorithms available in literature Consider writing own interpolation algorithms for optimal viewing results