Chapter 6. The Interpretation Process. (c) 2008 Prof. Dr. Michael M. Richter, Universität Kaiserslautern

Transcription

1 Chapter 6 The Interpretation Process

2 The Semantic Function Syntactically an image is simply a matrix of pixels with gray or color values. The intention is that the image mirrors the visual impression of the human viewer. This is, however, often not the case. In magnetic resonance e.g. one can generate data invisible to a viewer. In any case, the human associates a meaning when looking at the image and we want to formalize this process. Hence we have to compute a function SF: image SF is called a semantic function. meaning

3 Computing the Semantic Function (1) The computation is a complex process which we define as a configuration process. We construct the semantic function using a local-global principle: There are possible objects that can occur on the image, e.g. bones, parts of the brain etc. These are objects of the real world in some scenario S build up in a component oriented way: We assume atomic components (C i, i I) and a constructor function C such that the whole scenario S is given by S = C(C i, i I). An essential part of the constructor function C uses the part-of-relation; the atomic components are given as values of attributes.

4 Computing the Semantic Function (2) This real world scenario S is represented on the image by a mapping Im. The semantic function has to identify it, i.e. to reconstruct a description of S from the image = Im(S) by analyzing the representation on the image. It is assumed that Im(S) is also composed from atomic parts, Im(S) = C im (Im(C i ), i I); the essential parts of the constructor function are concerned with the geometric relations of the atomic parts.

5 Computing the Semantic Function (3) This identification proceeds in the following steps: (i) Identification of the atomic geometric objects (ii) Assigning to certain geometric objects G some object C i = SF(G) such that G = Im(C i ). This is called the local semantics because it assigns a meaning to a part of the image. (iii) If several of such objects occur on the image, the semantic function has to analyze their relative geometric positions and topological relationships in order to produce the position description PD. (iv)from the steps above the semantic function constructs the meaning as a global semantics of the image. Hence the semantic function SF is decomposed in the following way: SF(image) = SF({Im(C i ) i I}, PD) where the C i are the identified objects and PD describes their positions.

6 A Top-Level Diagram Representation Real World Object S = C(C i, i I) Object Description The interpretation in the real world will be neglected here Technical process Image as Pixel matrix SF Image Description Interpretation in the Real World

7 Discussion A real world object is something physical. An object description is a (formal) abstraction. It has necessarily to omit or to simplify some aspects of the real world. An image codes the object into a pixel matrix. Certain aspects will be lost, others can possibly reconstructed. The only impression and actual information a viewer has is the image. The task is to reconstruct the object descriptions (not the physical object!). For this a second information source is used: The knowledge base. This restricts the set of possible object descriptions to a small sets compatible with the given descriptions.

8 Example We look at the area marked with a red boundary in the figure: there is a horseshoe-shaped darker structure, which is clearly a candidate for representing some anatomical structure in the brain. As a human expert we expect at this place the fourth ventricle. In fact, the medical knowledge supports this. In an automated image understanding task, this conclusion has to be achieved through a number of reasoning steps.

9 Difficulties (1) (i) A real world object can be represented in various ways. The variations are provided e.g. by views from different positions or different brightness. (ii) The definition of the objects and their relations may be vague. (iii) In medicine there are two kinds of objects, non-pathological and pathological ones. The former ones are (although in a fuzzy way) described in a standard way. Pathological objects are not so standard and can in particular be confused with errors in the image analysis. The description of pathological objects uses in addition, vague notions as strong version of XY or advanced tumor of type Z etc. It is important to point out that it is not sufficient simply to use fuzzy logic because the fuzzy values need an explanation of what their meaning is and how they can contribute to express subjective opinions.

10 Difficulties (2) (i) There are different types of image generation techniques like X-rays, ultra sound or NMR which have different characteristics. (ii) The semantic functions will operate stepwise and possibly interactive and some image processing operations have to be carried out. The result of these operations is not always predictable and the operations may have to be repeated with different parameters, e.g. what are the steps necessary to identify the fourth ventricle? (iii) The semantic functions need access to domain knowledge which is often incomplete and imprecise, e.g., what are descriptions of the fourth ventricle that allow the identification of this structure?

11 Non-Pathological Objects In case of a known non-pathological object the interpretation reduces to a simplified identification procedure. In this case all objects and their positions are known at least in an ideal sense, e.g. through a brain atlas. If there would be no uncertainties then the identification process identifies all representations of objects piece by piece according to the ideal description of the scenario. In case of uncertaincies tolerances have to be respected.

12 Pathological Objects If pathologies are present they may have different effects: - An object is missing, e.g., the fourth ventricle is not clearly identifiable due to a brain mass dislocation caused by some accident, such as an oedema. - An additional object is present, e.g. a tumor. - An object does not have the correct form, texture, boundary etc. - An object does not present the ideal or correct geometric and topological features, e.g., the position of the fourth ventricle relative to the general distribution of brain parenchima and also its relative size.

13 Configuration as a Recursive Process Actual state New state Configuration action Validation Requirements satisfied? No Select operator or backtracking Yes Termination

14 The Configuration Process in General Types of steps (configuration actions): decompose a component into subcomponents (part-of relation) specialize a component (class-of relation) parametrize a component (determine attribute values) realize a component Corresponding types of operators: decompose, specialize, parametrize realize

15 Use of the Language Hierarchy Repetition: Level of overall description The image processing level Selection, Parametrization The geometric level Execution The pixel level

16 Configuration in Image Processing The main object which has to be configured is a sequence of image processing actions. The purpose of this sequence is to allow the generation of the meanings, i.e. of the semantics. Configuration states: Information state about the possible meaning The currently performed configuration steps Steps: Generating a hypothesis Selecting and parametrizing an image processing action (decompose, specialize andparametrize) Realize :Execute

17 Realization (1) In the configuration of technical objects realization assigns a technical object to the component. The image interpretation assigns an image processing method to the component. In addition, the method has to be executed (on the pixel level). This is a major difference to the classical configuration. The determination of the realization uses knowledge from geometric and image processing levels. The realization generates a new (possibly partial) image. This extends the information state and is up to further analysis (on the geometric and overall level). This analysis uses knowledge from those levels.

18 Realization (2) In the configuration we perform a split of tasks: 1. Selection, specification and parametrization of an imageprocessing method 2. Execution of the method The control is done by a virtual machine: request virtual machine geometric level return call return The virtual machines allows an independent management of the pool. Image processing pool

19 Validation (1) The validation of a configuration action always checks the satisfaction of constraints. A constraint violation generates a backtracking step. Origin of the constraints: Classical configuration: Requirements (from customer, technical ones etc.) Image interpretation: From the generated hypotheses, technical ones. While the requirements are assumed to be static (i.e. not up to change) the hypotheses are dynamically generated during the configuration process. Backtracking means to revise a hypothesis but does not necessitate the withdrawal of the results of the image processing method.

20 Validation (2) A hypothesis generates an expectation. There are two types: Expectation about an geometric object. This is a consequence of a hypothesis concerning the meaning of an image object: If the object is hypothetically of type X the it has to have the geometric property Y (e.g. to be an oval). The selected method has to check Y. Expectation about the result of the executed method. This is necessary because the methods do often not have an exact pre-post condition description, the outcome of the execution cannot always be predicted. The validation can be supported by the system but may also involve the human in an interactive process.

21 The Configuration State At each time the system is in some information state: a) Which properties are known about possible objects of the image: Inferred using the knowledge bases. b) Which image processing algorithms have been applied Successfully Unsuccessfully c) Which expectations have been Generated Withdrawn Both are determined by the results of the validation. The state has except at the end an incomplete information. The expectation generation produces only a hypothesis about a possible object but deduces at the same time from the knowledge base all properties of this object.

22 Revised Configuration: Top-Level Actual state New state Configuration action Execution (Intermediate-) results Validation Expectations satisfied? No Yes Consequences of not satisfied expectations

23 Cyclops: An Example System User Interface Domain Knowledge Image Storage Image Archives Inference Engine CS Redux TMS Expectatio n evaluation Algorithmic Knowled ge Executio n Control Algorithm Pool

24 Segmentation Errors There are two ways in which a segmentation algorithm can fail: Undersegmentation: Two objects are incorrectly merged into one object. Oversegmentation: An object is incorrectly divided into parts. There is no absolute definition of a segmentation being correct. It refers to the fact that the resulting image reflects the real object correctly. This is beyond the scope of image processing methods and requires access to a knowledge base. Medical objects do have a definition but admit tolerances. The problem is to decide whether there is an image processing error or a pathological situation. The decision can often done best by a human.

25 Segmentation Errors : Example Image a): Correctly segmented Image b): Undersegmentation: Two objects are incorrectly merged into one object. Image c): Oversegmentation: An object is incorrectly divided into parts.

26 Discussion The image shows the abdominal aorta. The aorta is identified by their size and position. There is, however, a dark shade, which is not admissible for the aorta and is therefore marked as a region of interest (Aneurisma). The discovery that over or under segmentation is present uses the knowledge that the objects in reality look like in image a). It has also be guaranteed that the wrong segmentation is not that consequence of a pathological situation (which would mean that there is no segmantation error). In image b) the aneurysmatic tissue is segmented together with adjacent structures. In image c) we encounter an oversegmentation. We show the segment boundaries for this purpose. We will now describe how to correct oversegmentation using the knowledge bases.

27 Dedection and Correction of Segmentation Errors The principle is that a human (or equally intelligent agent) looks at the image. The visual impression shows that the there was an oversegmentation took place: To separate segments are joint into one segment, e.g. by applying a closing operator. This insight is the consequence of some complex background knowledge. It may be difficult to formalize this knowledge appropriately. Hence the interference of a human agent may be useful or necessary as in the following example.

28 Knowledge Based Correction of Oversegmentation (1) Parameter Description Type Bounding Box Area Gray value Rectangle parallel to the xy axes, describing the simplest possible convex hull for the image. Area of the pixels representing the anatomy. Mean gray value of the anatomy. Can be either in HU Values or in absolute pixel values. Neighbors List of all other anatomies adjacent to this one. Anatomy Type Anatomy of the application domain represented by the image. A structure can be optional or obligatory. Optional structures always refer to pathologies. Pair of xy coordinates Real Real List of anatomies. Domain specific symbol. Symbol

29 Knowledge Based Correction of Oversegmentation (2) The upper half of the attributes refers to the geometric level and the lower half to the domain specific level. In addition, there are special parameters describing the segmentation: Parameter Description Type Gravity Center Preclassification Reliability Center of mass of the segment; used by the distance computation: if the center lies outside of the bounding box of a label, this label cannot be given to it. Anatomy of the application domain most likely to be represented by this segment. Value stating how reliable is the preclassification symbol. Based on the neural network error. xy-point Domain specific symbol. Real

30 Knowledge Based Correction of Oversegmentation (3) In the knowledge base (the data of) a prototype for this specific image and for segmentations are stored. A similarity function computes the distance between actual parameter values and those from the prototype. If distance exceeds some threshold, the segmentation is considered as failed. For correcting the oversegmentation two adjacent segments are hypothetically merged. If now the similarity to the prototype increases the merge is kept.

31 General Critisism (1) In this approach we have reduced step by step a complex image to elementary parts. For these parts we could give a precise definition because the reduction was to geometric objects. From the elementary parts the meaning of the whole object was constructed bottom-up using declarative techniques. This, however, does not always (or mostly) correspond to the way humans interpretate images.

32 General Critisism (2) Humans usually have more complex elementary images that they consider as basic, e.g. glasses of a human; these images are not anymore decomposed. Such basic images have been identified in art for describing pictures of Picasso or ornaments in ancient greece. This gives rise to other types of syntax of visual information. In art this is quite popular but has not been recognized in computer science.