Automated Diagnosis of Lymphoma with Digital Pathology Images Using Deep Learning

Transcription

1 Automated Diagnosis of Lymphoma with Digital Pathology Images Using Deep Learning Andy Nguyen, M.D., M.S. Medical Director, Hematopathology, Hematology and Coagulation Laboratory, Memorial Hermann Laboratory Professor of Pathology and Laboratory Medicine, University of Texas-Houston, Medical School Grand Round-Dec 19, 2017

2 Outline of talk Introduction to lymphoma diagnosis Machine Learning method for analyzing digital images Our implementation of a Convolutional Neural Network as the machine learning method to build a lymphoma diagnostic model for four diagnostic categories Financial Disclosures: No relevant financial relationships with commercial interests to disclose

3 Diagnosis of Lymphoma Lymphoma is a clonal malignancy of lymphocytes (either T- or B cells) The WHO Classification of Lymphoid Malignancies includes at least 38 entities. Lymphoid malignancies were diagnosed in 280,000 people annually worldwide Lymphoma is typically first suspected by their pattern of growth and the cytologic features of the abnormal cells via light microscopy of hematoxylin-eosin stained tissue sections Immunophenotypes are required for diagnosis (by flow cytometry and/or immunohistochemical stains) In addition, genetic, molecular results, and clinical features are often needed in finalizing the diagnosis in certain lymphoma types.

4 Automated Diagnosis for Morphology Due to subtle difference in histologic findings between various types of lymphoma, histopathologic screen often presents a challenge to the pathologists. An automated diagnosis for digital images would be helpful to assist the pathologists in daily work. Previous attempts to classify histologic images were based on specific criteria (such as nuclear shape, nuclear size, texture, etc. obtained by edge detection, cell segmentation). They were not very successful. Attention has turned to machine learning

5 Machine Learning Machine learning: -A specialized area of computer science that is based on algorithms that help the computer to learn from training data and make predictions on new data -Useful in computing tasks where programming explicit algorithms is unfeasible; such as in optical character recognition and computer vision (including microscopic images). Many machine learning algorithms have been made available: Logistic Regression, Support Vector Machines, Random Forest, Naïve Bayes, Neural Network, etc. In recent years, deep learning techniques, especially convolutional neural network (CNN or ConvNet), have quickly become the state of the art in computer vision.

6 Early Generation Neural Network Neural network (inspired by biological neural networks): artificial nodes ("neurons ) are connected together to form a network for classification tasks Training a network with multiple hidden layers using supervised learning: (1) Parameters often do not converge; i.e. being stuck in local minimum, (2) Model not scaling well for large input such as images (diffusion of gradients causing poor learning in earlier hidden layers)

7 Deep Learning (3 rd Gen Neural Network) Deep Learning algorithms: (1) Unsupervised learning ->allows a network to be fed with raw data (no known outcomes) and to automatically discover the representations needed for detection or classification (2) Extract high-level & complex data representations through multiple layers; avoid problems of last-gen network Supporting hardware: multiple graphics processing units (GPU) for parallel computation (esp. matrix computation), critical to handle large amount of data (esp. images) (-3)x(3) +(-1)x(2) +(-1)x(-4)+ (-5)x(-3) +(1)x(0) =8

8 Recent Applications of Deep Learning Siri: voice recognition in iphone (Apple) Google Translate (Google) Self-driving cars (Uber, Google, Tesla, others) Echo: control home appliances (Google)

9 Types of Deep Learning Architectures 1 Convolutional neural networks <<<<<<<<<<<<<< 2 Recursive neural networks 3 Long short term memory (LSTM) 4 Deep belief networks 5 Convolutional deep belief networks 6 Deep Boltzmann machines 7 Stacked auto-encoders 8 Tensor deep stacking networks 9 Spike-and-slab RBMs 10 Compound hierarchical-deep models 11 Deep coding networks 12 Deep q-networks 13 Encoder decoder networks 14 Multilayer kernel machine etc. Convolutional neural network (CNN, or ConvNet): inspired by visual cortex

10 CNN: Inspiration from the primate visual cortex The ventral visual pathway is organized as a hierarchical series of interconnected visual areas. Neurons in early areas, such as area V1, respond to comparatively simple, spatially local features of the retinal image, while later areas, such as area V4, respond to increasingly complex visual features over larger regions of visual space The specialization of receptor cells are incorporated into the design of CNN A rough correspondence between the areas associated with the primary visual cortex and the layers in a convolutional network. (A) Four Brodmann areas associated with the ventral visual stream forward and backward projections between these areas. (B) A convolutional network with pairs of convolution operator followed by a pooling layer are roughly analogous to the hierarchy of the biological visual system.

11 Definition of Convolution Convolution: an operation in image processing using filters, to modify or detect certain characteristics of an image (Smooth, Sharpen, Intensify, Enhance). In CNN, it is used to extract features of images Mathematically, a convolution is done by multiplying the pixels value in image patch by a filter (kernel) matrix [dot product] This effectively calculates the value of an image patch by adding the weighted values of all the neighboring pixels together Moving the filter across input image-> the final output is a modified filtered image

12 Convolutional neural network: Convolutional Layer Image patch Filter (kernel, receptor field) Convolutional layer: consists of a set of learnable filters (or kernels), that scan throughout the input image Each filter is convolved across the input patch, computing the dot product between the entries of the filter and the input and producing a feature map of that filter. A filter can be applied to every pixel in a whole slide image in a sliding window - fashion (moving each time by stride length ) -> feature maps Different neurons in each feature map share the same weights, become activated in the presence of the same image feature, such as oriented edge ( translation invariance -> feature can be detected across the entire image)

13 Convolutional neural network: Pooling Layer Pooling layer: performs non-linear down-sampling (sub sampling). Max pooling is the most common. It partitions the input image into a set of nonoverlapping rectangles and, for each such sub-region, outputs the maximum. The pooling layer serves to progressively reduce the spatial size of the representation, to reduce the amount of computation in the network. The intuition is that the exact location of a feature is less important than its rough location relative to other features.

14 Convolutional neural network: All layers stacked together Processing pipeline of a convolutional neural network for the detection of visual categories in images: -The convolutional layers (indicated with C) perform feature extraction consecutively from the image patch to higher level features. -The max pooling layers (indicated with S) reduce image size -The last fully connected layers (indicated with F) : provide prediction based on the given features. Neurons in these layers have connections to all activations in the previous layer, as seen in traditional neural networks

15 Convolutional neural network: Input image-> Classified category Consecutive layers extracting key features such as edges, contours, etc.

16 Deep Learning and Breast Cancer Detection, the Camelyon Grand Challenge 2016 The International Symposium on Biomedical Imaging (ISBI) held a Grand Challenge to evaluate computational systems for the automated detection of metastatic breast cancer in whole slide images of sentinel lymph node biopsies. The Harvard & MIT team won the grand challenge: obtaining an area under the receiver operating curve (AUC) of for the task of whole slide image classification (pos vs. neg) Receiver operating characteristic (ROC) curve AUC=0.925

17 Stanford U. Study: DL to diagnose two critical binary classification types (biopsy-proven clinical images): (1) keratinocyte carcinomas versus benign seborrheic keratoses -> AUC=0.96 (2) malignant melanomas versus benign nevi -> AUC=0.94

18

19 The Goal of our Study Recent studies have shown promising results in using machine learning to detect malignancy in whole slide imaging. However, they were limited to just positive and negative finding for a particular neoplasm. We explore how Deep Learning can be used to accurately classify a test case as one of the 4 entities (Representative of various morphologic patterns in lymphoma): Benign lymph nodes Diffuse large B-cell lymphoma Burkitt lymphoma Small lymphocytic lymphoma

20 Digital Images of Lymphomas Data source: at Virtual Pathology at the University of Leeds (355,966 slides TB) Web browser-> SnagIt (TechSmith Corp, Okemos, Michigan, USA) ->save image Web browser (at 40x) SnagIt to capture and automatically save 40x40 image patch in a file xxx.jpg SLL (40x40)

21 Digital Images of Lymphomas Data source: also at Virtual Slide Box-Whole Slide Imaging Collection (1,000 slides from U of Iowa) on Biolucida Cloud Portal, hosted by MicroBrightField Bioscience (Williston, VT USA) Biolucida viewer-> SnagIt (TechSmith Corp, Okemos, Michigan, USA) ->save image Biolucida Cloud Viewer (at 40x) at 40x SnagIt to capture and automatically save 40x40 image patch in a file xxx.jpg DLBCL LN (40x40)

22 Whole Slide Imaging (WSI) of Lymphomas WSI s were typically obtained with Aperio (Aperio Technologies, San Diego, CA, USA) whole slide imaging systems (T3 or CS Series). Our study includes: 32 cases for each of the following: 1. Benign lymph nodes, 2. Diffuse large B-cell lymphoma, 3. Burkitt lymphoma, 4. Small lymphocytic lymphoma, 5 representative 40x40 images for each of the 32 cases -> a total of 5x32x4=640 images

23 Our Programming Platform We design a CNN model in R language. R is a programming language for statistical computing and graphics supported by the R Foundation for Statistical Computing, commonly used in deep learning (together with Python) We use many functions obtained from an R package called MXNET which allows for parallel computing using GPU Hardware: Intel i5-4590, 8GB RAM Windows 8-64 bit GPU: GTX745 (4 GB), 384 cores NVIDIA card supported by CUDA (Compute Unified Device Architecture) A small excerpt of R Coding for CNN Algorithm

24 Digital Image Format 40x40 pixel image: 40 rows x 40 columns -> 1600 pixels Each image: 1601 entries for each row in data set Col 1:DX Col 2-Col 1601: pixel property (location, RGB) DX code: Benign 0 DLBCL 1 BL 3 SLL 4

25 Validation Method Display of Analysis Results (60 images in test set) 580 images were used for training the model, the model was then used to test on 60 images (not used in training) For each test case, the predicted diagnosis is combined from the prediction for 5 images (at least 3 or more have to agree), a process known as majority voting Majority vote: ->0 ->1 ->3

26 RESULTS: ACCURACY (image-by-image) Observed DX Benign DLBCL BL SLL Predicted DX Benign Accuracy: 49/60=82% DLBCL BL SLL PC system run time: 2.45 min 49 images (out of 60) with correct prediction, 11 images with error When testing was performed for one image at a time, accuracy is only at 82%, indicating the need to pool results from 5 representative images for each case (next slide)

27 RESULTS: ACCURACY (case-by-case) Predicted DX Observed DX Benign DLBCL BL SLL Benign 3 DLBCL 3 Significantly better Accuracy: 12/12=100% BL 3 SLL 3

28 SUMMARY We explore how Deep Learning can be utilized for histologic diagnosis of lymphoma using whole slide imaging The study design includes significant number of histology types (1 benign, 3 lymphoma types) compared to other studies, getting closer to actual practice Deep learning with CNN algorithm yields an impressive result (an accuracy of 100%) Generic machine learning algorithm with CNN method-> no need for manual settings of morphology parameters (nuclear architecture, shape, and texture, etc.) -> presumably can be applied to other histologic pathologies (GI, GYN, etc) Current limitations include: (a) only 4 histologic categories were included, not yet practical for clinical use, (b) representative images require manual selection of suspected areas, (c) color variations in the tissue due to differences in slide preparation, staining, microscope and whole slide scanners [batch effect, could be alleviated with complex stain color-normalization techniques] Our preliminary study provided a proof of concept for incorporating automated lymphoma diagnosis using digital microscopic images into the pathology work flow to augment the pathologists productivity Future studies will need to include more histologic entities and many more cases for training, and validation