On Generating Templates for Hypothesis in Inductive Logic Programming

Transcription

1 On Generating Templates for Hypothesis in Inductive Logic Programming Andrej Chovanec and Roman Barták Charles University in Prague, Faculty of Mathematics and Physics Malostranské nám. 25, Praha 1, Czech Republic Abstract. Inductive logic programming is a subfield of machine learning that uses first-order logic as a uniform representation for examples and hypothesis. In its core form, it deals with the problem of finding a hypothesis that covers all positive examples and excludes all negative examples. The coverage test and the method to obtain a hypothesis from a given template have been efficiently implemented using constraint satisfaction techniques. In this paper we suggest a method how to efficiently generate the template by remembering a history of generated templates and using this history when adding predicates to a new candidate template. This method significantly outperforms the existing method based on brute-force incremental extension of the template. Keywords: inductive logic programming, template generation, constraint satisfaction. 1 Introduction Inductive logic programming (ILP) is a discipline investigating invention of clausal theories from observed examples and additional background knowledge. Formally, for a given set of positive examples E + and a set of negative examples E - we are finding a hypothesis H such that H entails all examples from E + and does not entail any example from E - (a so-called consistent hypothesis). Background knowledge is a domain-specific knowledge which is available prior to learning process. Through this paper we suppose without loss of generality that background knowledge is empty. Since logical entailment is not complete in first-order logic θ-subsumption is used as a decidable restriction of logical entailment [8]. In our work, examples are represented as sets of fully instantiated atoms and hypothesis is a set of atoms with variables. In such setting, the process of inventing the consistent hypothesis H consists of (1) determining its structure, that is which atoms and how many of them are in the hypothesis, and (2) finding the unifications between variables in the atoms in such way that we obtain a consistent hypothesis. Hypothesis structure is called a template and hypothesis H is formed from template T by applying a substitution θ of variables such that Tθ = H. For instance, T = {arc(x 1,X 2 ), arc(x 3,X 4 )} is a template consisting of two atoms and four variables and H = {arc(x 1,X 2 ), arc(x 2,X 1 )} is one I. Batyrshin and G. Sidorov (Eds.): MICAI 2011, Part I, LNAI 7094, pp , Springer-Verlag Berlin Heidelberg 2011

2 On Generating Templates for Hypothesis in Inductive Logic Programming 163 particular hypothesis obtained from the template T by applying substitution θ = {X 3 /X 2, X 4 /X 1 }. Depending on the fact whether we do or do not know the template a priori, we speak about template ILP consistency in case we do know the template, and about general ILP consistency in case we do not know the template. Although both consistency problems belong to the same complexity class [5], the general task deals with an extra problem of determining the structure of the template, which has a radical impact on the overall performance of ILP system. Based on this fact, there are huge differences in running time between algorithms solving general and template ILP consistency problems. This situation motivated us to study the problem of general ILP consistency and we developed a new algorithm which should make solving general consistency more efficient. In this paper we present a new approach to generate templates for the general ILP consistency problem. Our work focuses on a method how the initial template is generated. The method uses existing algorithms to transform the template into a consistent hypothesis by finding certain unifications between variables. We propose a generate-and-test algorithm which successively generates templates until it is possible to generate a consistent hypothesis from the template. The algorithm remembers the previously generated templates and their properties and takes them into account when generating a next template. In other words, the algorithm is continuously learning itself from the rejected templates in order to improve templates generated in later iterations. The paper is organized as follows. In section 2 we will introduce existing algorithms used for the template ILP consistency problem. These algorithms are used as subroutines in our template generating algorithm. Simultaneously we will describe a basic version of the template generating algorithm. Section 3 will give the main result of the paper we will propose how to extend the basic template generating algorithm with several improvements and extensions yielding a self-learning template generating algorithm which is significantly more efficient. In section 4 we will further enhance the algorithm and finally in section 5 we will present experimental results comparing the effectiveness of the proposed methods. In the whole paper we will work with examples of identifying common structure (i.e. hypothesis) in random graphs (i.e. positive and negative examples). 2 Background As we have mentioned in the introduction, ILP deals with the problem of finding a hypothesis that covers a set of positive examples and excludes a set of negative examples. Hypothesis and examples are supposed to be clauses and the hypothesis is required to logically entail all positive examples and no negative example. Entailment is checked using θ-subsumption [8] which is a decidable restriction of logical entailment. We will assume the clauses to be expressed as sets of literals, and, without loss of generality, we will only work with positive literals, that is, non-negated atoms. All terms in learning examples (hypotheses, respectively) will be constants (variables) written in lower (upper) cases. For instance, E = {arc(a,b), arc(b,c), arc(c,a)} is an

3 164 A. Chovanec and R. Barták example and H = {arc(x,y), arc(y,z)} is a hypothesis. Hypothesis H subsumes example E, if there exists a substitution θ of variables such that Hθ E. In the above example, substitution θ = {X/a, Y/b, Z/c} implies that H subsumes E. In order to develop a general ILP system based on the generate-and-test approach, we must specify algorithms for three major components of the system. These components include generating the template, testing θ-subsumption and finally deciding template consistency. In this case, the ILP algorithm usually repeats the following three steps: generate a template, find unifications of variables in the template to obtain a hypothesis, test consistency of a hypothesis using θ-subsumption check. If the hypothesis is not consistent, the algorithm tries other unifications and if all unifications are exhausted, it generates a new template. The θ-subsumption check and deciding template consistency have already been addressed by the utilization of constraint satisfaction (CSP) techniques. θ- subsumption check is a crucial part of ILP systems and thus it naturally motivated new efficient approaches to be developed. Maloberti and Sebag proposed an efficient CSP algorithm Django [6] that significantly outperformed existing systems. Dramatic speed-up brought by Django encouraged exploitation of CSP techniques in other parts of ILP systems. In [2] Barták suggested a novel approach utilizing CSP towards deciding template consistency. His work was based on finding unifications between variables in the template to obtain a consistent hypothesis. The idea is as follows. We start with a hypothesis consisting of mutually different variables (the template) and then we systematically search the space of possible unifications until we obtain a consistent hypothesis or the search exhausts all possibilities. Barták proposed a sophisticated CSP model that extensively prunes the space of all unifications and thus keeps runtime in very sensible bounds. For generating the template, Barták used a simple algorithm based on the generateand-test method that works as follows: first we recall that each variable in the template appears exactly once (it is the goal of template consistency to decide which variables in atoms should be unified to obtain a consistent hypothesis). Hence template generation is about deciding how many atoms of each predicate symbol will appear in the template. In [2] this is done by exploring systematically all possible templates with the increasing length of template. Let k denote the number of different predicate symbols in all examples (in case the background knowledge is not empty, also predicate symbols from the background knowledge should be considered), for example {arc(a,b), arc(b,c), arc(c,a)} contains one predicate symbol (arc) and three atoms for this predicate symbol. To simplify notation, we will write predicate p instead of writing the atom of predicate symbol p. For example, adding predicate p to template means adding a new atom of the predicate symbol p with fresh variables to the template. The algorithm starts with a template of length k containing one atom for each predicate symbol and as soon as the constraint model for template consistency rejects the template, a new template is generated. A template space is searched systematically by generating all possible templates of length k, k+1, k+2,

4 On Generating Templates for Hypothesis in Inductive Logic Programming 165 The complete algorithm solving the general ILP consistency problem may look like in Figure 1. As the stop criterion we can use either reaching of the maximal k or reaching the maximal runtime. 1: k number of all different predicates in examples 2: repeat 3: S T all templates of length k 4: for each T S T do 5: H Decide-Consistency(T) 6: if H is consistent then return H 7: end for 8: k k + 1 9: until stop criterion satisfied Fig. 1. A basic generate-and-test template generating algorithm (IDS) We can notice easily that algorithm in Figure 1 is performing iterative deepening search (IDS). An advantage of this approach is that it guarantees finding the shortest consistent hypothesis. However, we are not always interested in finding the shortest possible solution. In fact, the trade-off between finding the optimal hypothesis and the cost of this search is so cumbersome that this method becomes impractical, especially for longer templates (see the section with experimental results). Hence we suggest a different method for exploring the space of templates based on incremental extension of the template by adding new predicates. 3 Incremental Template Generation When we analyzed reasons why the IDS algorithm was not capable to find more complicated (longer) hypotheses, we find out it is because of its leaps between two distinct parts of the template space. Think of the following example: we are solving a problem having the final template equal to 5 a, 5 b, 5 c (5 predicates a, 5 predicates b, and 5 predicates c) and we have examined all templates of length 14. Let us suppose without loss of generality, that the last examined template of the length 14 was 4 a, 5 b, 5 c. We can see that it is sufficient to add predicate a to the template. Despite, the basic systematic algorithm starts blindly generating all templates of the length 15 from the beginning until it reaches the correct template. Based on the above observation we suggest searching the space of templates only one way new predicates can be added to the template and once they are added, they cannot be removed. Further, we need to determine which predicate is due to be added to the template in each step of the algorithm. We can use systematic search that adds different predicates one by one in cycles add predicate a, followed by predicate b, followed by predicate c, and again predicate a etc. Or we can add each predicate with certain probability. If the probability of adding predicate to the template is distributed among all predicates uniformly, the earlier and the latter approaches are identical in

5 166 A. Chovanec and R. Barták terms of developing the template. Both approaches are identical because the probability of the case when the probabilistic approach would significantly prefer adding one particular predicate (the number of its occurrences would be k-times of occurrences of other predicates for any constant k > 0) diminishes exponentially (a socalled Chernoff bound [7]). However, the probabilistic approach is still preferable to the systematic one as it allows us to guide the search algorithm according to custom conditions which may even evolve in time. In Figure 2 we show the basic incremental probabilistic approach (IPS) to template generation. Procedure Decide-Consistency does the actual template validation. The stop criterion is met when either Decide-Consistency succeeded in validating the template or execution time expired. It may sometimes happen that the hypothesis created from the template contains some isolated atoms, i.e. atoms with variables that are not unified with other variables. These atoms are obviously redundant and hence they are removed from the final hypothesis (Remove-Isolated-Atoms procedure in the algorithm). A drawback of the whole algorithm is that it can negatively affect the performance of Decide-Consistency function as the function has to consider unifications between variables in atoms that are eventually identified as redundant. 1: P all different predicate symbols in examples 2: T empty template 3: repeat 4: Generate predicate p P with probability distributed uniformly 5: T T {p} 6: H Decide-Consistency(T) 7: until stop criterion satisfied 8: H Remove-Isolated-Atoms(H) Fig. 2. An incremental probabilistic template generation (IPS) 4 History-Driven Tabu Template Generation The key drawback of the pure incremental approach as described in the previous section is that it generates new atoms without any respect to the previous work it did. In particular, we noticed that if the added atom contributes to better consistency of the hypothesis, it might be useful to add atom of the same predicate symbol again. We formalize this idea in another template generation algorithm based on two techniques: (1) For a given template, we check the maximal number B of broken negative examples achieved by some hypothesis generated from the template (hypothesis breaks the negative example if it does not subsume it and subsumes all positive examples at the same time). Then we check the value of B between two subsequent templates. Based on information whether B increases or decreases between the iterations we will decide about the next predicate to be added to the template. (2) We maintain a tabu list [4] of predicates such that these predicates cannot be added to the template in next steps of the algorithm.

6 On Generating Templates for Hypothesis in Inductive Logic Programming 167 Firstly, let us discuss in detail the point (1). After we launch the formerly described algorithms (either IDS or IPS), we analyze the maximal numbers of the negative examples which are broken by any hypothesis formed from the last two templates. In particular, if we added predicate p to the last template and the maximal number B of the broken negative examples increased in comparison to the number B of the last but one template, then we studied what happens with the maximal number of the broken negative examples for the next template if we add the same predicate p again. Empirical results showed that if a predicate added to the template increased the number of broken negative examples in one iteration then it is very likely to increase the number B again if we add the same predicate in the next iteration again. Particularly, in our experiments the average number of such iterations, where the number of broken negative examples was increased obeying the suggested rule, was 69.3% (the base was computed as the number of all subsequent pairs of templates, where the first template of the pair increased the maximal number of the broken negative examples). In contrast, we also averaged the number of iterations where the number of broken negative examples was increased by adding a different predicate than the one added last time. The number of such iterations was 26.19%, which is much less than the number of the improving iterations in the first case. Thus adding the same predicate seems to be beneficial. Finally we should note that the fact that in the remaining 30.7% of the first case when the rule did not increase the maximal number of the negative examples does not mean the heuristic went always wrong. It might also happen that there was no predicate at all that would increase the maximal number of the broken negative examples for a given template structure (no matter which one would be added). Now let us continue with the point (2) that resembles the idea of Tabu Search [4] though there are some differences in handling the tabu list. If we add some predicate that did not cause the maximal number of broken negative examples to increase, then we forbid adding this predicate to the template in the following iterations of the algorithm until some condition (a so-called aspiration criterion) is met and the predicate is allowed to be used again. For this purpose we maintain a tabu list consisting of all predicates that cannot be added to the template in next iterations. As soon as the tabu list contains all predicates or some predicate increased the maximal number of broken negative examples, the tabu list is emptied and the process continues. In the second case, we empty the tabu list because of the fact that if the last predicate increased the maximal number of broken negative examples, then the template was changed relatively significantly and all predicates (also those in the tabu list) are now more likely to break some new negative examples than they were before. Now if we put these two concepts together, we get a basic history-driven tabu template generation algorithm which is illustrated in Figure 3. In each iteration of the algorithm, we try to validate the current template (line 6) and if we succeed, we are done. Otherwise we store the maximal number of broken negative examples during the validation (line 9) and then we proceed with the next steps depending on the maximal number of broken negative examples in the previous iteration. If the number

7 168 A. Chovanec and R. Barták increased, we add the same predicate to the template as we did the last time (line 11). In other case we generate a new predicate randomly such that it is not in the tabu list and we add it to the template (lines 16, 17). 1: Initialize template T 2: B last 0 {max. number of broken neg. examples in the last iteration} 3: P last any predicate {a predicate appended into template in the last iteration} 4: Tabu {tabu list} 5: repeat 6: H consistent Decide-Consistency(T) 7: if H consistent exists then return H consistent 8: else 9: B current Max-Broken-Neg-Examples(T) 10: if B current > B last then 11: T T {P last } 12: Tabu 13: else 14: if all predicates in Tabu then Tabu 15: P current random predicate not in Tabu 16: T T {P current } 17: Tabu Tabu {P current } 18: P last = P current 19: end if 20: B last = B current 21: end if 22: until stop criterion satisfied Fig. 3. A history-driven template generation The complexity of the above algorithm is strongly dependent on the complexity of Decide-Consistency function. It has been proved that the problem of deciding template consistency is Σ 2 P -complete [5], hence the whole algorithm belongs at least to this complexity class. If we want to determine the upper bound, it is necessary to identify the number of calls of Decide-Consistency function in the repeat loop. In general, the loop does not have to finish at all. However, we are usually not interested in finding arbitrary solution for the ILP problem and rather it makes sense to impose an upper bound on the size of the desired solution. Furthermore, if the bound is polynomially related to the size of the evidence, we get a so-called bounded ILP problem which is Σ 2 P -complete [5]. 4.1 Stochastic Extension of History-Driven Tabu Template Generation The algorithm in Figure 3 behaves like a modification of the well known hill-climbing algorithm and it is fully deterministic except for the step at line 15. We can further

8 On Generating Templates for Hypothesis in Inductive Logic Programming 169 improve its performance by modifying the probability to select a predicate at line 15. We developed a stochastic model as an extension of the basic history-driven algorithm yielding the stochastic history-driven tabu algorithm. The idea is that the predicate that behaved well in previous iterations is preferred to be added to the template. First, besides the tabu list we introduce a candidate list which is the list of all predicates not in the tabu list. Furthermore, each predicate in the candidate list is assigned a probability such that all probabilities in the candidate list sum to one. In each iteration of the algorithm we pick a predicate from the candidate list according to its probability and add it to the template. The candidate list is a dynamic structure. In addition to adding and removing predicates into or from it, the probabilities may change as well. There are two situations when the probabilities in the candidate list are recomputed. The first situation arises when the last predicate added to the template increased the maximal number of broken negative examples. In that case we (1) move all predicates from the tabu list back to the candidate list and set their probabilities to some low value p, (2) set the probability of the last added predicate to some high value p high in the candidate list. After adding the predicates from the tabu list back to the candidate list, we should still prefer those predicates originally in the candidate list to those recently added from the tabu list when selecting the next predicate to be added to the template. The reason is that the predicates originally in the tabu list behaved worse than those in the candidate list (that is the reason why they were tabu). The intended effect can be achieved by modifying the probabilities of predicates in the candidate list in the following way. We find a predicate with the minimal probability p min among all predicates in the candidate list and then we assign the probability p = p min. p tabu to all predicates moved from the tabu list. In our implementation we use p tabu = 0.1 so the probability of selecting some predicate originally from the tabu list is at least ten times smaller than the probability of selecting any predicate originally in the candidate list. Setting the probability of the last successful predicate to p high means that we prefer appending those literals that increased the maximal number of broken negative examples in the last iteration (as we have already proposed). In our algorithm we use p high = We should remark that after every change of the probabilities in the candidate list we normalize them to yield the sum equal to one. The second situation when the probabilities of the predicates in the candidate list are adjusted is when the candidate list becomes empty (it means that all predicates are in the tabu list). In this case we put all predicates back to the candidate list and we distribute the probability among them uniformly. In Figure 4 we give a pseudo code for the stochastic history-driven tabu template generation. This algorithm is very similar to the algorithm in Figure 3 except it extends it with the stochastic steps. We use a few new procedures in the algorithm. Procedure Distribute-Probability- Uniformly returns the set of all predicates with identical probabilities. Procedure Predicate-With-Min-Prob returns the predicate with the minimal probability. Procedure Update-Probability takes the list provided in the first argument and updates the probability of the predicates in the second argument to the value in the third argument. Finally procedure Normalize-Probabilities adjusts the probabilities such that they sum to one. As the stop criterion we can use expiration timeout.

9 170 A. Chovanec and R. Barták 1: Initialize template T 2: B last 0 {maximal number of broken negative examples in the last iteration} 3: P last any predicate {a predicate appended to the template in the last iteration} 4: Tabu {tabu list} 5: Cand Distribute-Probability-Uniformly {initialization of candidate list} 6: repeat 7: H consistent Decide-Consistency(T) 8: if H consistent exists then return H consistent 9: else 10: B current Max-Broken-Neg-Examples(T) 11: if B current > B last then 12: p min = Predicate-With-Min-Prob(Cand) 13: Update-Probability(Cand, {P last }, p high ) 14: Update-Probability(Cand, Tabu, p min. p tabu ) 15: Normalize-Probabilities(Cand) 16: Tabu 17: else 18: if all predicates in Tabu then 19: Tabu 20: Cand Distribute-Probability-Uniformly 21: end if 22: end if 23: P current predicate from Cand with corresponding probability 24: T T {P current } 25: Tabu Tabu {P current } 26: Cand Cand \ {P current } 27: Normalize-Probabilities(Cand) 28: B last = B current 29: P last = P current 30: end if 31: until stop criterion satisfied Fig. 4. A stochastic history-driven tabu template generation 5 Experimental Results In order to compare effectiveness of the proposed methods, we implemented and benchmarked the algorithms in SICStus Prolog on 2.0 GHz Intel Xeon processor with 12 GB RAM under Gentoo Linux. The first set of experiments was executed on ten instances of identifying common structures in random graphs generated according to Barabási-Réka model [1]. We used graphs consisting of 20 nodes that were constructed by incrementally adding new nodes and connecting them with three arcs

10 On Generating Templates for Hypothesis in Inductive Logic Programming 171 to existing nodes in the graph. The hidden structure that we were looking for (the consistent hypothesis) consisted of five nodes. Both positive and negative evidence contained ten instances of the graphs. In Table 1 we show a comprehensive comparison of all methods described in the paper the naïve iterative deepening search (IDS) from [2], the incremental probabilistic search (IPS) and finally the stochastic history-driven tabu search (SHDTS). For each method we show the overall running time and the length of the found template. The runtime limit was set to 600 seconds. Since IPS and SHDTS are randomized algorithms, the results were obtained by averaging values through their five runs. In case any of these runs exceeded the maximal time limit, the final values were averaged only from the successful runs and the number of these unfinished runs is showed in the corresponding column. Table 1. A comparison of the naïve iterative deepening search (IDS), the incremental probabilistic search (IPS) and the stochastic history-driven tabu search (SHDTS) IDS IPS SHDTS time[s] length time[s] length #unfinished time[s] length #unfinished > First let us compare the results between IDS and IPS. If we look at the hypotheses lengths 1, we can see that they are almost identical for both searches, thus IPS finds almost always the optimal solution (we know that the solution given by the IDS is optimal). Further let us compare the runtimes. Except for one dataset, IPS clearly outperforms IDS. On the other hand, we have to realize that these results are not guaranteed as IPS is a randomized algorithm. In fact, in seven of ten cases the algorithm did not finish in one of its five runs. The reason why the search algorithm ran so long is that the random generator generated repeatedly wrong predicates until the template was so long that its validation exceeded the time limit. Now let us focus on the result of SHDTS. We can see that the length of the final hypothesis is almost identical to the optimal solution again and thus templates generated by SHDTS are appropriate. Actual significance of this method is evident when we analyze the runtimes. The method not only beats IPS for almost every case 1 When comparing the lengths of the hypotheses, we only compare the number of all atoms in the hypothesis but we do not consider the discrepancies between actual predicate symbols of the atoms and their arities in the hypotheses.

11 172 A. Chovanec and R. Barták but it also finished for all instances. The worse result in the ninth case is due to the randomized nature of the algorithms. On the other hand, IPS did not finish once on this instance. Hence the experiments proved that stochastic history-driven tabu search noticeably outperforms the former two methods. To support further that the proposed algorithms contribute to better performance of whole ILP system, we evaluated them on another set of input instances generated according to different model than in the first case. In this case we were again identifying common structures in random graphs, however new nodes were connected to the existing ones obeying Erdös-Rényi [3] model. In particular, each input dataset consists of ten positive and ten negative examples (graphs), where the graphs contain 20 nodes with density of arcs 0.2. In Table 2 we present results for finding implanted hidden structure consisting of 5, 6 and 7 nodes with arc density of 0.4. In the first column there is a number of nodes of the hidden common structure and in other columns there are runtimes and hypothesis lengths of each algorithm. The overall runtime limit was set to 1200 seconds. From Table 2 it clearly follows that SHDTS is working best for most of the datasets. Table 2. A comparison of the three methods for identifying common structures (subgraphs) of various number of nodes IDS IPS SHDS nodes time[s] length time[s] length time[s] length > > > > > > > > > > > > > Conclusions In this paper we addressed the problem of generating templates for inductive logic programming. We presented a novel approach that uses existing CSP algorithms as its subroutines. Specifically we use subroutines for θ-subsumption check [6] and for deciding template consistency [2]. We started with a simple algorithm from [2] performing iterative deepening search that is showed to be inefficient. The main result of the paper is a history-driven tabu template generating algorithm which is guided by evaluating how the negative evidence is covered by the generated template. Furthermore, we suggested a stochastic extension of this algorithm yielding a stochastic history-driven

12 On Generating Templates for Hypothesis in Inductive Logic Programming 173 tabu search. The efficiency of the last algorithm is demonstrated experimentally on the problem of identifying common structures in randomly generated graphs according to two different models. Stochastic history-driven search is doing better on almost all input instances and it significantly decreases the template generation time in comparison to former algorithms. An interesting feature of this algorithm is that it uses some form of learning so the algorithm learns itself during the learning process. Future work should mainly deal with testing the performance of the algorithms on instances of domains other than those of the random graphs, for example those of the bioinformatics. Real-life problems have often a particular structure and thus it is challenging to examine our methods in these fields. Acknowledgments. The authors would like to thank Filip Železný and Ondřej Kuželka for useful discussions on ILP techniques and for providing a generator of random problems for experiments. References 1. Barabási, A.-L., Réka, A.: Emergence of scaling in random networks. Science 286, (1999) 2. Barták, R.: Constraint Models for Reasoning on Unification in Inductive Logic Programming. In: Dicheva, D., Dochev, D. (eds.) AIMSA LNCS (LNAI), vol. 6304, pp Springer, Heidelberg (2010) 3. Erdös, P., Rényi, A.: On Random Graphs I. Publicationes Mathematicae Debrecen 6, (1959) 4. Glover, F., Laguna, M.: Tabu Search. Kluwer, Norwell (1997) 5. Gottlob, G., Leone, N., Scarcello, F.: On the complexity of some inductive logic programming problems. New Generation Computing 17, (1999) 6. Maloberti, J., Sebag, M.: Fast Theta-Subsumption with Constraint Satisfaction Algorithms. Machine Learning 55, (2004) 7. Mitzenmacher, M., Upfal, E.: Probability and Computing: Randomized Algorithms and Probabilistic Analysis. Cambridge University Press (2005) 8. Plotkin, G.: A note on inductive generalization. In: Meltzer, B., Michie, D. (eds.) Machine Intelligence, vol. 5, pp Edinburgh University Press (1970)