MedGIFT projects in medical imaging. Henning Müller

Transcription

1 MedGIFT projects in medical imaging Henning Müller

2 Where we are 2

3 Who I am Medical informatics studies in Heidelberg, Germany ( ) Exchange with Daimler Benz research, USA PhD in image processing, image retrieval, Geneva, Switzerland ( ) Exchange with Monash University, Melbourne, AUS Titular professor in radiology and medical informatics at the University of Geneva (2014-) Postdoc, assistant professor between Professor in Computer Science at the HES-SO, Sierre, Switzerland (2007-) 3

4 Why working on image retrieval? Much imaging data is produced Imaging data is very complex And getting more complex Imaging is essential in diagnosis and treatment planning Images out of their context loose most of their sense Clinical data is necessary Diagnoses are often not precise Evidence-based medicine & case-based reasoning 4

5 Khresmoi retrieval 5

6 The informed patient 6

7 Steps for our retrieval system Resource creation Image pretreatment Visual feature extraction Feature modeling Visualization, results presentation Classification, detection, retrieval Multimodal fusion 7

8 Diagnosis aid 8

9 Identifying user requirements Surveys among radiologists Also GPs and patients Observing diagnosis processes Analyzing search log files (Goldminer, PubMed, HON) Eye tracking on a radiology viewing station What are information needs and what are tasks that are hard and where help is needed? Test the developed systems in user studies Analyze feedback Record the system use for understanding problems 9

10 Eye tracking 10

11 Data used for ParaDISE Scientific data of the biomedical literature articles and 1.6 mio figures of the open access literature (>4 mio images if separating compound figures) Public data source but only 2D data Clinical data from the Vienna Medical University image archive 5TB of data of two consecutive months Radiology reports for each case (in German) Private data source, so access only with password Link medical cases with similar cases from the literature based on image data and text 11

12 Creation of the VISCERAL database 12

13 Annotations (20 organs, 55 landmarks) 13

14 Connecting different data levels EHR, PACS 14

15 Classification of journal figures Most figures in articles are not diagnostic imaging Captions do not always allow to identify the image type Visual information can help All these image types are mapped to RadLex and UMLS/MeSH H Müller, J Kalpathy-Cramer, D Demner-Fushman, S Antani, Creating a classification of image types in the medical literature for visual categorization, SPIE medical imaging, San Diego, USA, Allows reusing information and search in related terms 15

16 Context is important (25 yo vs. 88 yo)! 16

17 Visual feature extraction Colors & grey levels Shapes after segmentations Texture information In 2D, 3D, 4D In several scales and directions Local vs. global information extraction Finding interest points Finding regions or volumes of interest Combination of features is usually best 17

18 Visual feature modeling Visual words instead of raw visual features Reducing the curse of dimensionality Find models similar to text (synonyms, polysemy) A Foncubierta, AG Seco de Herrera, H Müller, Medical Image Retrieval using a Bag of Meaningful Visual Words, ACM MM workshop on medical multimedia retrieval, Barcelona, Spain,

19 Feature extraction and detection Learn combinations of Riesz wavelets as digital signatures using SVMs Create signatures to detect small local lesions and visualize them A Depeursinge, A Foncubierta Rodriguez, D Van de Ville, H Müller, Rotation covariant feature learning using steerable Riesz wavelets, IEEE Transactions on Image Processing,

20 Information fusion Combine information from text or structured data with visual information Text data can be mapped to semantics to understand links Also language-independent Early fusion Late fusion Rank-based vs. score-based t1 M t M c1 M c N SVM SVM q t q c p ( w) t i p ( w) c i N mod t1 M tm c1 M cn SVM 20

21 Detection and retrieval of similar cases 21 A Depeursinge, D Van de Ville, A Platon, A Geissbuhler, PA Poletti, H Müller, Near-Affine-Invariant Texture Learning for Lung Tissue Analysis Using Isotropic Wavelet Frames, IEEE Transactions on Information Technology in Biomedicine, 16(4), 2012.

22 Khresmoi4radiology interface 22

23 Khresmoi4professionals interface 23

24 Semantic search, also for images 24

25 Khresmoi4everyone interface 25

26 Shambala a simple web interface 26

27 Much involvement in benchmarking ImageCLEF Has had a medical task since : modality classification, compound figure separation, image-based and case-based retrieval 2014: liver annotation VISCERAL Organ segmentation and landmark detection (ISBI) Lesion detection and retrieval task Khresmoi LinkedIn group, 27

28 Cloud-based evaluation in VISCERAL Test 28

29 VISCERAL data 29

30 4D data analysis m(e) (cm2/ mg) Iodine Water Material Attenuation Coefficient vs kev 80 kev 140 kev Photon Energy (kev) Dual Energy CT for perfusion analysis in pulmonary embolism Collaboration with emergency radiology Epileptogenic lesion detection in several MRI image series (T1, T2, DTI) OA Jimenez del Toro, A Foncubierta-Rodriguez, MI Vargas Gomez, H Müller, A Depeursinge, Epileptogenic lesion quantification in MRI using contralateral 3D texture comparisons, MICCAI 2013, Springer LNCS, Nagoya, Japan, A Depeursinge, A Foncubierta-Rodriguez, A Vargas, D Van de Ville, A Platon, PA Poletti, H Müller, Rotation-covariant texture analysis of 4D dual-energy CT as an indicator of local pulmonary perfusion, ISBI 2013, San Francisco, USA,

31 4D visualization Visualization of two (min and max) energy levels to visualize pulmonary embolisms 31

32 Another view on 4D 32

33 An infrastructure supporting the load Small, fixed experiments are easy, large routine updates and use are difficult!! Big data is hard! Workflow for data re-indexation, maximum automation Khresmoi: Private cloud All components in virtual machines connected with a SOA infrastructure, reattribution of resources possible Local computation Hadoop/MapReduce to distribute the computation Needs some optimization Cloud use when local resources are not sufficient 33

34 System overview 34

35 Infostructure in MD-Paedigree 35

36 Conclusions Visual information retrieval has many interesting challenges in the medical field Many supporting techniques are required Treating big data is a challenge and digital medicine is really big data Many techniques can and need to be used with image analysis and machine learning as the basis Digital medicine is a reality and more is yet to come genetics, molecular imaging, We also need corresponding infrastructures 36

37 Contact and more information More information can be found at Contact: 37