Learning convex bodies is hard

Transcription

1 Learning convex boies is har Navin Goyal Microsoft Research Inia Luis Raemacher Georgia Tech Abstract We show that learning a convex boy in R, given ranom samples from the boy, requires 2 Ω /ɛ) samples By learning a convex boy we mean fining a set having at most ɛ relative symmetric ifference with the input boy To prove the lower boun we construct a har to learn family of convex boies Our construction of this family is very simple an base on error correcting coes 1 Introuction We consier the following problem: Given uniformly ranom points from a convex boy in R, we woul like to approximately learn the boy with as few samples as possible In this question, an throughout this paper, we are intereste in the number of samples but not in the computational requirements for constructing such an approximation Our main result will show that this nees about 2 Ω ) samples This problem is a special case of the statistical problem of inferring information about a probability istribution from samples For example, one can approximate the centroi of the boy with a sample of size roughly linear in On the other han, a sample of size polynomial in is not enough to approximate the volume of a convex boy within a constant factor [3], an see Section 5 here for a iscussion) Note that known approximation algorithms for the volume eg, [2]) o not work in this setting as they nee a membership oracle an ranom points from various carefully chosen subsets of the input boy Our problem also relates to work in learning theory eg, [12, 8]), where one is given samples generate accoring to say) the Gaussian istribution an each sample is labele positive or negative epening on whether it belongs to the boy Asie from ifferent istributions, another ifference between the learning setting of [8] an ours is that in ours one gets only positive examples Klivans et al [8] give an algorithm an a nearly matching lower boun for learning convex boies with labele samples chosen accoring to Supporte by ARC ThinkTank Supporte by ARC ThinkTank an NSF the Gaussian istribution Their algorithm takes time 2Õ ) an they also show a lower boun of 2 Ω ) Note that our lower boun is not just a simplification of theirs: Neither lower boun seems to imply the other in a simple way The problem of learning convex sets from uniformly ranom samples from them was raise by Frieze et al [4] They gave a polynomial time algorithm for learning parallelopipes Another somewhat relate irection is the work on the learnability of iscrete istributions by Kearns et al [7] We remark that in the traitional istribution-inepenent PAClearning setting, it is not possible to learn convex boies efficiently because their VC-imension is infinite Our lower boun result like that of [8]) also allows for membership oracle queries Note that it is known that estimating the volume of convex boies requires an exponential number of membership queries if the algorithm is eterministic [1], which implies that learning boies requires an exponential number of membership queries because if an algorithm can learn the boy then it can also estimate its volume To formally efine the notion of learning we nee to specify a istance, ) between boies A natural choice in our setting is to consier the total variation istance of the uniform istribution on each boy see Section 2) We will use the term ranom oracle of a convex boy K for a black box that when querie outputs a uniformly ranom point from K Theorem 1 There exists a istribution D on the set of convex boies in R satisfying the following: Let ALG be a ranomize algorithm that, given a ranom convex boy K accoring to D, makes at most q total queries to ranom an membership oracles of K an outputs a set C such that, for 8/ ɛ 1/8, PrC, K) ɛ) 1/2 where the probability is over K, the ranom sample an any ranomization by ALG Then q 2 Ω /ɛ) Remarkably, the lower boun of Klivans et al [8] is numerically essentially ientical to ours 2 Ω ) for ɛ constant) Constructions similar to theirs are possible for our particular scenario [5] We believe that our argument is consierably simpler, an elementary compare to that of [8] Furthermore, our construction of the har to learn family

2 is explicit Our construction makes use of error correcting coes To our knowlege, this connection with error correcting coes is new in such contexts an may fin further applications See Section 5 for some further comparison An informal outline of the proof The iea of the proof is to fin a large family of convex boies in R n satisfying two conflicting goals: 1) Any two boies in the family are almost isjoint; 2) an yet they look alike in the sense that a small sample of ranom points from any such boy is insufficient for etermining which one it is because there are many boies in the family consistent with the sample Since any two boies are almost isjoint, even approximating a boy woul allow one to etermine it exactly This will imply that it is also har to approximate We first construct a family of boies that although not almost isjoint, have sufficiently large symmetric ifference We will then be able to construct a family with almost isjoint boies by taking proucts of boies in the first family The first family is quite natural it is escribe formally in Sec 31) Consier the cross polytope O n in R n generalization of the octaheron to n imensions: convex hull of the vectors {±e i : i [n]}, where e i is the unit vector in R n with the ith coorinate 1 an the rest 0) A peak attache to a facet F of O n is a pyrami that has F as its base an has its other vertex outsie O n on the normal to F going through its centroi If the height of the peak is sufficiently small then attaching peaks to any subset of the 2 n facets will result in a convex polytope We will show later that we can choose the height so that the volume of all the 2 n peaks is Ω1/n) fraction of the volume of O n We call this family of boies P [We remark that our construction of cross-polytopes with peaks has resemblance to a construction in [10] with ifferent parameters, but there oes not seem to be any connection between the problem stuie there an the problem we are intereste in] Intuitively, a ranom point in a boy from this family tells one that if the point is in one of the peaks then that peak is present, otherwise one learns nothing Therefore if the number of queries is at most a polynomial in n, then one learns nothing about most of the peaks an so the algorithm cannot tell which boy it got But these boies o not have large symmetric ifference can be as small as a O1/n2 n )) fraction of the cross polytope if the two boies iffer in just one peak) but we can pick a subfamily of them having pairwise symmetric ifference at least Ω1/n) by picking a large ranom subfamily We will o it slightly ifferently which will be more convenient for the proof: Boies in P have one-to-one corresponence with binary strings of length 2 n : each facet correspons to a coorinate of the string which takes value 1 if that facet has a peak attache, else it has value 0 To ensure that any two boies in our family iffer in many peaks it suffices to ensure that their corresponing strings have large Hamming istance Large sets of such strings are of course furnishe by goo error correcting coes From this family we can obtain another family of almost isjoint boies by taking proucts, while preserving the property that polynomially many ranom samples o not tell the boies apart This prouct trick also known as tensoring) has been use many times before, in particular for amplifying harness, but we are not aware of its use in a setting similar to ours Our construction of the prouct family also resembles the operation of concatenation in coing theory Acknowlegments We are grateful to Aam Kalai an Santosh Vempala for useful iscussions 2 Preliminaries Let K, L R n be boune an measurable We efine a istance istk, L) as the total variation istance between the uniform istributions in K an L, that is, istk, L) = { K\L K L\K L if K L if L > K We will use to enote the volume of sets A R n, an also to enote the carinality of finite sets A; which one is meant in a particular case will be clear from the context Let 1 enote the vector 1,, 1) log enotes logarithm with base 2 We will nee some basic efinitions an facts from coing theory; see, eg, [11] For a finite alphabet Σ, an wor length n, a coe C is a subset of Σ n For any two coewors x, y C, istance istx, y) between them is efine by istx, y) := {i [n] : x i y i } The relative minimum istance for coe C is min x,y C,x y istx, y)/n For Σ = {0, 1}, the weight of a coewor x is {i [n] : x i 0} Define V q n, r) := r n i=0 i) q 1) i The following is well-known an easy to prove: Theorem 2 Gilbert Varshamov) For alphabet size q, coe length n, an minimum istance, there exists a coe of size at least q n /V q n, 1) When the alphabet is Σ = {0, 1}, we efine the complement c of a coewor c C as c i := 1 c i 3 A har to learn family of convex boies The construction procees in two steps In the first step we construct a large subfamily of P such that the relative pairwise symmetric ifference between the boies is Ω1/n) This symmetric ifference is however not sufficiently large for our lower boun The secon step of the construction amplifies the symmetric ifference by consiering proucts of the boies from the first family 31 The inner family: Cross-polytope with peaks We first construct a family with slightly weaker properties The family consists of what we call cross polytope with peaks The n-imensional cross polytope O n is the convex hull of the 2n points {±e i, i [n]} Let F be a facet of O n, an let c F be the center of F The peak associate to F is the convex hull of F an the point αc F, where α > 1 is a positive scalar efine as follows: α is picke as large as possible so that the union of the cross polytope an all 2 n peaks is a convex boy A cross polytope with peaks will then be the union of the cross polytope an any subfamily of the 2 n possible peaks The set of all 2 2n boies of this type will be enote P By fixing of an orering of the facets of

3 the cross polytope, there is a one-to-one corresponence between the cross polytope with peaks an 0 1 vectors with 2 n coorinates Let P enote the cross polytope with all 2 n peaks We will initially choose α as large as possible so that the following conition necessary for convexity of P but not clearly sufficient is satisfie: for every pair of ajacent facets F, G of O n, the vertex of each peak is in the following halfspace: the halfspace containing O n an whose bounary is the hyperplane orthogonal to the vector connecting the origin to the) center of F G, an containing F G A straightforwar computation shows that α = n/n 1) for this conition This implies by another easy computation that the volume of all the peaks is O n /n 1) We will now show that this weaker conition on α is actually sufficient for the convexity of P an any cross polytope with peaks Proposition 3 Every set in P is convex Proof Let Q be the intersection of all halfspaces of the form {x R n : a x 1} where a R n is a vector having entries in { 1, 0, 1} an exactly one zero entry Equivalently, the bounary of each such halfspace is a hyperplane orthogonal to the center of some n 2)-imensional face of O n an containing that face In the rest of the proof we will show that P = Q, which gives the convexity of P This equality implies that a cross polytope with only some peaks is also convex: any such boy can be obtaine from P by intersecting P with the hafspaces inuce by the facets of O n associate to the missing peaks, an it is easy to see from the efinition of the peaks that each such intersection removes exactly one peak It is clear that P Q For the other inclusion, let x Q By symmetry we can assume x 0 If x i 1, then x O n P If x i > 1, we will show that x is in the peak of the positive orthant We woul like to write x as a convex combination of e 1,, e n an the extra vertex of the peak, v = 1/n 1) Let µ = n 1) x i ) 1) > 0 We want a vector λ = λ 1,, λ n ) such that x is a convex combination of the vertices of the peak: x = µv + λ i e i = µv + λ, that is, λ = x x i It satisfies µ + λ i = 1 an λ i = x i + 1 x i, an this is non-negative: By efinition of Q we have for all j [n] xi 1 + x j This shows that x belongs to the peak in the positive orthant For notational convenience we let N := 2 n Recall that we ientify boies in P with binary strings in {0, 1} N Let C {0, 1} N be a coe with relative minimum istance at least 1/4 To simplify computations involving istance ist between boies, it will be convenient to have the property that all coewors in C have weight N/2 We can ensure this easily as follows Let C {0, 1} N/2 be a coe with relative minimum istance at least 1/4, then set C := {c, c) : c C} Clearly C = C By Theorem 2 we can choose C such that C 2 c1n, for a positive constant c 1 We fix C to be such a coe, ie a coe with relative minimum istance at least 1/4, size 2 c1n, an all coewors with weight N/2 We efine the family P C as the family consisting of boies in P corresponing to coewors in C As all coewors in C have the same weight, we have that all boies in P C have the same volume Recall that the volume of each peak O is n 2 n n 1) Therefore for istinct P, Q P C the volume of the symmetric ifference of P an Q is at least O n 4n 1) 32 The outer family: The prouct construction Let C be a coe with coewors of length k an minimum istance at least k/2 on the alphabet P C That is, coewors in C can be represente as B 1,, B k ), where B i P C for i = 1,, k The prouct family PC C corresponing to coe C, has C boies in R kn, one for each coewor The boy for coewor B 1,, B k ) C is simply B 1 B k Clearly PC C = C Using Theorem 2 we can choose C such that C q k /V q k, k/2) Now note that k/2 V q k, k/2) = i=0 k/2 k )q 1) i q 1) k/2 i < 2 k q 1) k/2 < 4q) k/2 i=0 ) k i Therefore q k /V q k, k/2) > q k /4q) k/2 = q/4) k/2 Setting q = 2 c1n, we get C > 2 c1n 2)k/2 > 2 c2kn, for constant c 2 > 0, assuming N is sufficiently large We just showe: Lemma 4 P C C > 2c2k2n The following lemma shows that the boies in PC C are almost pairwise isjoint Lemma 5 For istinct A, B P C C ista, B) = 1 Proof We constructe PC C same volume This implies ista, B) = 1 we have > 1 e k/16n) so that all boies in it have the Let A = A 1 A k an B = B 1 B k Then = A 1 B 1 A k B k A 1 A k Since the minimum relative istance in C is at least 1/4 an the weight of each coewor is N/2, we have that for

4 A i B i the number of peaks in A i B i is at most 2 n 3/8 Hence A i B i 1 + 3/8n 1)) A i 1 + 1/2n 1)) Since the minimum istance of C is at least k/2, we have A i B i for at least k/2 values of i in [k] Therefore we get ) k/ /8n 1)) 1 + 1/2n 1)) 1 1 ) k/2 < e k/16n) 8n 4 Proof of the lower boun Proof of Theorem 1 We will make use of the family PC C that we constructe in the previous section Recall that the boies in this family live in R, for := kn For this proof we will think of as fixe an we will choose n appropriately for the lower boun proof By a straightforwar but teious argument it is enough to prove the theorem assuming that is a power of 2 We will use Yao s principle see, eg, [9]) To this en, we will first show that the interaction between an algorithm an the oracles can be assume to be iscrete, which in turn will imply that effectively there is only a finite number of eterministic algorithms that make at most q queries The iscretization of the oracles also serves a secon purpose: that we can see eterministic algorithms as finite ecision trees an use counting arguments to show a lower boun on the query complexity Fix a boy K from PC C Suppose that a ranomize algorithm has access to the following iscretizations of the oracles: A iscrete ranom oracle that generates a ranom point X = X 1,, X k ) from K = i K i an, for each i [k] outputs whether X i lies in the corresponing cross-polytope or in which peak it lies A iscrete membership oracle that when given a sequence of inices of peaks I = i 1,, i k ) outputs, for each i [k], whether peak i is present in K i Claim: A ranomize algorithm with access to iscrete versions of the oracles can simulate a ranomize algorithm with access to continuous oracles with the same number of queries Proof of claim: We will show it for boies in P, ie cross polytopes with peaks; the generalization of this argument to prouct boies in PC C is straightforwar Let A an B be the algorithm with access to the continuous an iscrete oracles respectively Algorithm B acts as A, except when A invokes an oracle, where B will o as follows: When A makes a query p to the continuous membership oracle, B will query the peak that contains p we can assume that p lies in a peak, as otherwise the query provie no new information) Now suppose that A makes a query to the continuous ranom oracle an gets a point p Then B makes a query to the iscrete ranom oracle B then generates a uniformly ranom point p in the region that it got from the oracle Clearly p has the same istribution as p, namely uniform istribution on the boy If we see eterministic algorithms as ecision trees, it is clear that there are only a finite number of eterministic algorithms that make at most q queries to the iscrete oracles of K Thus, by Yao s principle, for any istribution D on inputs, the probability of error of any ranomize algorithm against D is at least the probability of error of the best eterministic algorithm against D Our har input istribution D is the uniform istribution over PC C Now, in the ecision tree associate to a eterministic algorithm, each noe associate to a membership query has two chilren either the query point is in the boy or not), while a noe associate to a ranom sample has at most 2 n + 1) k chilren the ranom sample can lie in one of the 2 n peaks or in O n, for each factor in the prouct boy) Thus, if the algorithm makes at most q queries in total, then the ecision tree has at most 2 n + 1) kq leaves These leaves inuce a partition of the family of inputs PC C By Lemma 5, the istance between any pair of boies is at least 1 e k/16n = 1 e /16n2, where n is chosen so that the output of the algorithm can be within ɛ of at most one boy in each part of the partition That is, 2ɛ < 1 e k/16n, which implies that we shoul take n < 4 ln 1 As = kn is a power of 2, we can satisfy the previous inequality an the integrality constraints of k an n by using our assumption that 8/ ɛ 1/8 an letting n be a power of 2 such that 2 ln 1 n < 4 ln 1 By Lemma 4, the total number of boies is P C C 2 c2k2n This implies that the probability of error is at least 1 2n + 1) kq P C C 1 2n + 1)kq 2 c2k2n If we want this error to be less than a given δ, then for some c 3, c 4 > 0 we nee ) log1 δ) q c 3 + 2n kn n ) log1 δ) c log 1 2ɛ 2 log 1 log1 δ) c 3 + ɛ 2 For δ = 1/2 an ɛ 1/4 this implies q 2 Ω /ɛ) log 1 )

5 5 Discussion Informally, our construction of P C C can be thought of as coes in R, namely sets in R that are far from each other; the ifficulty in the construction of such coes comes from the requirements of convexity an that the istributions of polynomially many ranom samples look alike By using slightly more involve arguments we can hanle ɛ arbitrarily close to 1 an prove a similar lower boun It is not clear if such a lower boun is possible for other learning settings that have been stuie in the past, eg labele samples from Gaussian istribution Unlike that setting, we o not know a matching upper boun for learning convex boies in our moel Our construction of the har family is more explicit than that of [8]: The har family they construct is obtaine by a probabilistic argument; our construction can be mae explicit by using goo error correcting coes We mention here a somewhat surprising corollary of our result, without etaile proof or precise numerical constants Informally, it shows instability of the reconstruction of a convex boy as a function of the volume of its intersection with halfspaces, relative to its volume It is an exercise to see that knowlege of K H / K for every halfspace H uniquely etermines K Moreover, given c for some fixe constant c > 0) ranom samples from a convex boy K R, with high probability we can estimate K H / K for all halfspaces H within aitive error of O1/ c ), where c is a positive constant epening on c This can be prove using stanar arguments about ɛ-approximations an the fact that the VC-imension of halfspaces in R is + 1 We say that two convex boies K an L are α-halfspacefar if there is a halfspace H such that K H / K L H / L > α Thus if we choose some t < c an K an L are 1/ t -halfspace-far, then we can etect this using c ranom points, with high probability Now, we claim that there is a pair of boies in PC C that is not far For otherwise, all pairs woul be far an we woul be able to istinguish every boy in PC C from every other boy in PC C with a sample of size c, an thus learn it But as we have prove, this is impossible So we can conclue that there are two boies in PC C that are not 1/t -halfspace-far, ie they are 1/ t - halfspace-close This gives: Corollary 6 For any constant t > 0 an sufficiently large there exist two convex boies K, L R such that K, L are 1/ t -halfspace-close: for every halfspace H K H K but istk, L) > 1/8 L H L 1 t, An earlier version of this manuscript mentione the problem of whether the volume of a convex boy in R can be estimate from poly) uniformly ranom samples Very recently, Ronen Elan [3] has answere this in the negative His result provies a probabilistic construction of a family of convex boies such that the volume a ranom boy from this family is har to estimate from ranom samples His result oes not supersee ours in the sense that our lower boun of 2 Ω ) is stronger, an perhaps optimal, an our construction of the har family is explicit It is known that if the convex boy is a polytope with poly) facets, then it can be learne with poly) uniformly ranom samples [6] in an information-theoretical sense However, whether this can be one efficiently remains open: Problem Can one learn polytopes with poly) facets from poly) uniformly ranom over the polytope) samples in poly) time? References [1] I Bárány an Z Fürei Computing the volume is ifficult Discrete & Computational Geometry, 2: , 1987 [2] M E Dyer, A M Frieze, an R Kannan A ranom polynomial time algorithm for approximating the volume of convex boies J ACM, 381):1 17, 1991 [3] R Elan A polynomial number of ranom points oes not etermine the volume of a convex boy Manuscript, 2009 [4] A M Frieze, M Jerrum, an R Kannan Learning linear transformations In FOCS, pages , 1996 [5] A Kalai Personal communication 2006 [6] A Kalai an S Vempala Personal communication 2007 [7] M J Kearns, Y Mansour, D Ron, R Rubinfel, R E Schapire, an L Sellie On the learnability of iscrete istributions In STOC, pages , 1994 [8] A Klivans, R O Donnell, an R Serveio Learning geometric concepts via gaussian surface area In FOCS, 2008 [9] R Motwani an P Raghavan Ranomize Algorithms Cambrige University Press, 1995 [10] L Raemacher an S Vempala Testing geometric convexity In FSTTCS, pages , 2004 [11] J van Lint Introuction to Coing Theory Springer, thir eition, 1998 [12] S Vempala The Ranom Projection Metho American Mathematical Society, DIMACS, 2004