MEDICAL IMAGE COMPUTING (CAP 5937) LECTURE 20: Machine Learning in Medical Imaging II (deep learning and decision forests)

Transcription

1 SPRING MEDICAL IMAGE COMPUTING (CAP 5937) LECTURE 20: Machine Learning in Medical Imaging II (deep learning and decision forests) Dr. Ulas Bagci HEC 221, Center for Research in Computer Vision (CRCV), University of Central Florida (UCF), Orlando, FL bagci@ucf.edu or bagci@crcv.ucf.edu

2 2 Outline Image Segmentation using ML Image Registration using ML Deep Learning and its applications in radiology applications Decision Forests for Medical Image Analysis

3 3 Deep Learning? Reasonable Definition: Deep learning is a method that makes predictions using a sequence of non-linear processing stages The resulting intermediate representations can be interpreted as feature hierarchies and the whole system is jointly learned from data Deep Learning is a new way of fitting neural nets. Traditionally a neural net is fit into labeled data all in one operation. The weights are usually started at random values near zero. Due to the non-convexity of the objective function, the final solution can get caught in a poor local minimum. Deep Learning is a new way of fitting neural nets. Multiple layers are first fit in an unsupervised way, and then the values at the top layers are used as starting values for supervised learning. This two stage approach also allows the use of unlabeled data. Different learning models are possible: quantitative latent factors and discrete hidden factors.

4 Deep Learning 4

5

6 Deep Learning in Medical Imaging 6

7 Ex. Automatic LV Segmentation from US 7

8 Ex. Automatic LV Segmentation from US with Deep Belief Nets 8

9 Ex. Hippocampus Segmentation Using 7T MRI 9

10 Challenges in 7T 10

11 Hand-Crafted Features 11

12 Deep Learning Features 12

13 Hierarchical Feature Extraction 13

14 Multi-Atlas Segmentation 14

15 15 Qualitative Evaluations

16 Ex. Registration of Brain MR Images 16

17 Deep Learning for Image Registration 17

18 Deep Learning for Image Registration 18

19 Deep Learning for Image Registration 19

20 20 Deep Learning for Image Registration

21 21 Random Decision Forests Book: Decision Forests in Computer Vision and Medical Image Analysis. A. Criminisi and J. Shotton. Springer Code: The Microsoft Research Cambridge Sherwood Software Library

22 22 A Brief History of Random Forests [ L. Breiman, J. Friedman, C.J. Stone, and R.A. Olshen. Classification and Regression Trees (CART) ] [ Y. Amit and D. Geman. Randomized enquiries about shape; An application to handwritten digit recognition. Technical Report 1994] [ Y. Amit and D. Geman. Shape quantization and recognition with randomized trees ] [ L. Breiman. Random forests. 1999, 2001 ] [ V. Lepetit and P. Fua. Keypoint recognition using randomized trees. 2005, 2006 ] [ F. Moosman, B. Triggs, F. Jurie. Fast discriminative visual codebooks using randomized clustering forests ] [ G. Rogez, J. Rihan, S. Ramalingam, P. Orrite, C. Torr. Randomized trees for human pose detection ] [ C. Leistner, A. Saffari, J. Santner, H. Bischoff. Semi-supervised random forests ] [ A. Saffari, C. Leistner, J. Santner, M. Godec, H. Bischoff. On-line random forests ] [ S. Nowozin, C. Rother, S. Bagon, T. Sharp, B. Yao, and P. Kohli. Decision tree fields ]

23 23 What can decision forests do? Classification forests Density forests Regression forests Manifold forests Semi-supervised forests

24 24 What can decision forests do? Classification forests Regression forests e.g. semantic segmentation e.g. object localization Density forests Manifold forests Semi-supervised forests e.g. novelty detection e.g. dimensionality reduction e.g. semi-sup. semantic segmentation

25 25 Decision Trees/Forests A general tree structure A decision tree 0 root node Is top part blue? internal (split) node 1 2 Is bottom part green? Is bottom part blue? terminal (leaf) node A forest is an ensemble of trees. The trees are all slightly different from one another.

26 Decision Trees Training 26 How to split the data? Input data in feature space Binary tree? Ternary? How big a tree? What tree structure?

27 Decision forest model: training and information gain (for categorical, non-parametric distributions) Before split Information gain Split 1 Shannon s entropy Node training Split 2

28 Classification forest Training data in feature space? Classification tree training?? Model specialization for classification Input data point Output is categorical ( is feature response) (discrete set) Entropy of a discrete distribution Node weak learner Obj. funct. for node j (information gain) with Training node j Predictor model (class posterior)

29 Classification forest: the weak learner model Splitting data at node j Node weak learner Node test params Examples of weak learners Weak learner: axis aligned Weak learner: oriented line Weak learner: conic section Feature response for 2D example. Feature response for 2D example. Feature response for 2D example. With or With a generic line in homog. coordinates. With a matrix representing a conic. In general may select only a very small subset of features See Appendix C for relation with kernel trick.

30 Classification forest: the prediction model What do we do at the leaf? leaf leaf Prediction model: probabilistic leaf

31 Classification forest: the ensemble model Tree t=1 t=2 t=3 The ensemble model Forest output probability

32 32 Ex. Tissue Specific Segmentation of Brain Tumors Segmentation of tumorous tissues: T1-gad T1 T2 FLAIR DTI-p DTI-q Multi-channel 3D MRI input data ---- Active cells ---- Necrotic core ---- Edema ---- Background

33 Patient E Patient D Patient C Patient B Patient A Challenge: variability of input data Location Shape Intensity Texture 33

34 34 Context Sensitive Classification Forests Context-sensitivity? Making the decision based on the context (non-local neighbourhood). How? Integrate test-specific initial probabilities as additional input channels à related to Auto-Context / Entanglement ideas Use context-sensitive features (spatially non-local) à results in high-dimensional feature space, for which forests are a very suitable classifier

35 35 Context-sensitive classification forests The Approach Initial Probability Estimates (local, GMM-based on intensity features) input data MRI + DTI Classification Forests (context-sensitive through longrange spatial features) initial tissue probabilities (serve as additional input)

36 36 Training the classification forests test example: ø( ) - ø( ) > θ? - Spatial context, across channels - Test selection: optimization over randomized features B E AC NC AC NC B E

37 Testing the classification forest 37 Training the classification forests B E AC NC AC NC B E

38 Glioblastoma Segmentation 38

39 39 Glioblastoma Segmentation [D. Zikic, B. Glocker, E. Konukoglu, A. Criminisi, J. Shotton, C. Demiralp, O. M. Thomas, T. Das, R. Jena and S. J. Price. Decision Forests for Tissue-specific Segmentation of High-grade Gliomas in Multi-channel MR. in MICCAI, Springer, Oct 2012]

40 Regression forest Training data? Model specialization fore regression Input data point Output/labels are continuous Node weak learner Obj. funct. for node j Information gain defined on continuous variables. Training node j Leaf model

41 Regression forest: the node weak learner model Splitting data at node j Node weak learner Examples of node weak learners Node test params Weak learner: axis aligned Weak learner: oriented line Weak learner: conic section Feature response for 2D example. Feature response for 2D example. Feature response for 2D example. With or With a generic line in homog. coordinates. With a matrix representing a conic. In general may select only a very small subset of features See Appendix C for relation with kernel trick.

42 Regression forest: the predictor model What do we do at the leaf? Examples of leaf (predictor) models Predictor model: constant Predictor model: polynomial Predictor model: probabilistic-linear (note: linear for n=1, constant for n=0)

43 Regression forest: objective function At node j Computing the regression information gain at node j Regression information gain Differential entropy of Gaussian Our regression information gain

44 Regression forest: objective function At node j Comparison with Breiman s error of fit (CART) Our information gain Error of fit The error of fit objective is a special instance of our more general information-theoretical objective function.

45 Regression forest: the ensemble model Tree t=1 t=2 t=3 The ensemble model Forest output probability

46 46 Ex. Regression Forests for Anatomy Localization from CT Scans Applications: Selective retrieval of regions of interest from PACS databases. Efficient bandwidth use. Fast, semantic visualization and 3D image inspection. Linking of image portions with radiological reports. Initializing organ-specific image processing. FDA Approval Sep 2012

47 Regression forests for anatomy localization in CT images Each voxel in the image votes for the position of the 6 box sides We wish to learn a set of discriminative points (landmarks, clusters) which can predict the kidney position with high confidence. Regression forest Input data point Problem parametrization (for 1 organ only) (voxel position) (relative displacement) (absolute displacement) Output (continuous) (Gaussian representation) Objective function Node training Predictor model Leaf model = Multivariate, probabilistic constant

48 Regression forests for anatomy localization in CT images Visual features Regression forest Visual features Feature response Test parameters Node weak learner

49 Regression forests for anatomy localization in CT images Results in 2D Good behaviour despite anomaly (missing lung). Changes with forest size

50 Regression forests for anatomy localization in CT images Automatic, unsupervised discovery of landmark regions Here the system is trained to detect left and right kidneys. The system learns to use bottom of lung and top of pelvis to localize kidneys with highest confidence. Input 3D CT scan and detected landmark regions

51 Ex. Spine detection in arbitrary fov CT legend input output input output [B. Glocker, J. Feulner, A. Criminisi, D. Haynor and E. Konukoglu. Automatic Localization and Identification of Vertebrae in Arbitrary Field of View CT Scans. in MICCAI, Springer, Oct 2012]

52 Ex. Spine detection in arbitrary fov CT legend input output input output [B. Glocker, J. Feulner, A. Criminisi, D. Haynor and E. Konukoglu. Automatic Localization and Identification of Vertebrae in Arbitrary Field of View CT Scans. in MICCAI, Springer, Oct 2012]

53 53 Summary ML algorithms are key to detection and diagnosis problem Deep Learning is getting a lot of interests due to its strong features for CAD tasks Image analysis tasks can be enhanced via ML algorithms Decision forests algorithms are commonly used powerful algorithms for recognition and segmentation of organs/objects

54 54 Slide Credits and References Wernick et al., Signal Processing Magazine, Wang and Summers, Medical Image Analysis, Dinggang Shen, Deep Learning Talk at MLMI Workshop, MICCAI Wu, G., et al. MICCAI Kim, M., MLMI, MICCAI Decision Forests for Computer Vision and Medical Image Analysis, A. Criminisi, J. Shotton and E. Konukoglu