Learning with the Aid of an Oracle

Transcription

1 ' Learning with the Aid of an Oracle (1996; Bshouty, Cleve, Gavaldà, Kannan, Tamon) CHRISTINO TAMON, Computer Science, Clarkson University, tino Exact Learning, Boolean Circuits, Disjunctive Normal Form, Com- Synonyms and Index Terms plexity Oracles 1 Problem Definition In the exact learning model of Angluin [1], a learning algorithm function must discover an unknown that is a member of a known class of Boolean functions. The learning algorithm can make at least one of the following types of queries about : Equivalence query, for a candidate function : The reply is either yes, if!, or a counterexample " with #$"&% Membership query () $", for some "+*, : The reply is the Boolean value $". Subset query -/.102, for a candidate function : The reply is yes, if 435, or a counterexample " with $"768#9", otherwise. Superset query -/.1:2 9, for a candidate function : The reply is yes, if ;35, or a counterexample " with #9"<6= >9", otherwise. $", otherwise. A Disjunctive Normal Formula ) is a depth-2 BDC circuit whose size is given by the number of its BDC gates. Likewise, a Conjunctive Normal Formula (E@7A ) is a - BDC circuit whose size is given by the number of gates. Any Boolean function can be represented as both or a G@FA formula. A H is where gate has a fan-in of at most H ; similarly, we may define a H -G@FA. Problem For a given class of Boolean functions, such as polynomial-size Boolean circuits or Disjunctive Normal Form ) formulas, the goal is to design polynomial-time learning algorithms for any unknown I*J and ask a polynomial number of queries. The output of the learning algorithm should be a function of polynomial size satisfying KL M. The polynomial functions bounding the running time, query complexity, and output size are defined in terms of the number of inputs N and the size of the smallest representation (Boolean circuit ) of the unknown function 1

2 2 Key Results One of the main results proved in [4] is that Boolean circuits and Disjunctive Normal Formulas are exactly learnable using equivalence queries and access to oracle. Theorem 1 The following tasks can be accomplished with probabilistic polynomial-time algorithms that have access to oracle and make polynomially many equivalence queries: The idea behind this result is simple. Any class of Boolean functions is exactly learnable with equivalence queries using the Halving algorithm of Littlestone [10]. This algorithm asks equivalence queries that are the majority of candidate functions from. These are functions in that are consistent with the counterexamples obtained so far by the learning algorithm. Since each such majority query eliminates at least half of the candidate functions, equivalence queries are sufficient to learn any function in. A problem with using the Halving algorithm here is that the majority query has exponential size. But, it can be shown that a majority of a polynomial number of uniformly random candidate functions is a good enough approximator to the majority of all candidate functions. Moreover, with access to oracle, there is a randomized polynomial time algorithm for generating random uniform candidate functions due to Jerrum, Valiant, and Vazirani [6]. This yields the result. The next observation is that subset and superset queries are apparently powerful enough to simulate both equivalence queries and oracle. This is easy to see since the tautology test! is equivalent to -/.102 I-/.102 9, for any unknown function ; and, 9 is equivalent to -/.102 -/.1:2. Thus, the following generalization of Theorem 1 is obtained. Theorem 2 The following tasks can be accomplished with probabilistic polynomial-time algorithms that make polynomially many subset and superset queries: Stronger deterministic results are obtained by allowing more powerful complexity-theoretic oracles. The first of these results employ techniques developed by Sipser and Stockmeyer [11, 12]. Theorem 3 The following tasks can be accomplished with deterministic polynomial-time algorithms that have access to an oracle and make polynomially many equivalence queries: 2

3 In the following result, is an infinite class of functions containing functions of the form 2 J. The class is -evaluatable if the following tasks can be performed in polynomial time: Given, is a valid representation for any function &*? Given a valid representation and *, 2, is / '? Theorem 4 Let be any -evaluatable class. The following statements are equivalent: is learnable from polynomially many equivalence queries of polynomial size (and unlimited computational power). is learnable in deterministic polynomial time with equivalence queries and access to a oracle. For exact learning with membership queries, the following results are proved. Theorem 5 The following tasks can be accomplished with deterministic polynomial-time algorithms that have access to oracle and make polynomially many membership queries (in and G@FA sizes of, where is the unknown function): Learning monotone Boolean functions. Learning L N -E@7A GN The ideas behind the above result use techniques from [1, 3]. For a monotone Boolean function, the standard closure algorithm uses both equivalence and membership queries to learn using candidate functions satisfying 3. The need for membership can be removed using the following observation. Viewing G as a monotone function on the inverted lattice, we can learn and G simultaneously using candidate functions, respectively, that satisfy 3. oracle is used to obtain an example " that either helps in learning or in learning E ; when no such example can be found, we have learned. Theorem 6 Any class of Boolean functions that is exactly learnable using a polynomial number of membership queries (and unlimited computational power) is exactly learnable in expected polynomial time using a polynomial number of membership queries and access to oracle. Moreover, any -evaluatable class that is exactly learnable from a polynomially number membership queries (and unlimited computational power), is also learnable in deterministic polynomial time using a polynomial number of membership queries and access to a oracle. Theorems 4 and 6 showed that information-theoretic learnability using equivalence and membership queries can be transformed into computational learnability at the expense of using the oracles, respectively. 3

4 3 Applications The learning algorithm for Boolean circuits using equivalence queries and access to oracle has found an application in complexity theory. Watanabe (see [9]) showed an improvement on a known theorem of Karp and Lipton [7]: has polynomial-size circuits, then the polynomialtime hierarchy collapses to. Some techniques developed in Theorem 5 for exact learning using membership queries of monotone Boolean functions have found applications in data mining [5]. 4 Open Problems It is unknown if there are polynomial-time learning algorithms for Boolean circuits formulas using equivalence queries (without complexity-theoretic oracles). There are strong cryptographic evidence that Boolean circuits are not learnable in polynomial-time (see [2] and the references therein). The best running time for formulas is as given by Klivans and Servedio [8]. It is unclear if membership queries help in this case. 5 Experimental Results 6 Data Sets 7 URL to Code 8 Cross References For related learning results, see Learning DNF Formulas (1997; Jackson) and Learning Automata (2000; Beimel, Bergadano, Bshouty, Kushilevitz, and Varricchio) in this encyclopedia. 9 Recommended Reading [1] D. ANGLUIN, Queries and Concept Learning, Machine Learning, 2 (1988), pp [2] D. ANGLUIN AND M. KHARITONOV, When Won t Membership Queries Help?, Journal of Computer and System Sciences, 50 (1995), pp

5 [3] N. H. BSHOUTY, Exact Learning Boolean Function via the Monotone Theory, Information and Computation, 123 (1995), pp [4] N. H. BSHOUTY, R. CLEVE, R. GAVALDÀ, S. KANNAN, AND C. TAMON, Oracles and Queries That Are Sufficient for Exact Learning, Journal of Computer and System Sciences, 52 (1996), pp [5] D. GUNOPOLOUS, R. KHARDON, H. MANNILA, S. SALUJA, H. TOIVONEN, AND R. S. SHARMA, Discovering All Most Specific Sentences, ACM Transaction on Database Systems, 28 (2003), pp [6] M. R. JERRUM, L. G. VALIANT, AND V. V. VAZIRANI, Random Generation of Combinatorial Structures from a Uniform Distribution, Theoretical Computer Science, 43 (1986), pp [7] R. M. KARP AND R. J. LIPTON, Some Connections Between Nonuniform and Uniform Complexity Classes, in Proceedings of the 12th Ann. ACM Symposium on Theory of Computing, ACM Press, 1980, pp , Journal of Com- [8] A. R. KLIVANS AND R. A. SERVEDIO, Learning DNF in Time puter and System Sciences, 68 (2004), pp [9] J. KÖBLER AND O. WATANABE, New Collapse Consequences of NP Having Small Circuits, SIAM Journal of Computing, 28 (1998), pp [10] N. LITTLESTONE, Learning Quickly When Irrelevant Attributes Abound: A New Linear- Threshold Algorithm, Machine Learning, 2 (1987), pp [11] M. SIPSER, A complexity theoretic approach to randomness, in Proceedings of the 15th Annual ACM Symposium on Theory of Computing, ACM Press, 1983, pp [12] L. J. STOCKMEYER, On approximation algorithms for, SIAM Journal of Computing, 14 (1985), pp