Classifier Evasion: Models and Open Problems

Transcription

1 Classiier Evasion: Models and Open Problems Blaine Nelson 1, Benjamin I. P. Rubinstein 2, Ling Huang 3, Anthony D. Joseph 1,3, and J. D. Tygar 1 1 UC Berkeley 2 Microsot Research 3 Intel Labs Berkeley Abstract. As a growing number o sotware developers apply machine learning to make key decisions in their systems, adversaries are adapting and launching ever more sophisticated attacks against these systems. The near-optimal evasion problem considers an adversary that searches or a low-cost negative instance by submitting a minimal number o queries to a classiier, in order to eectively evade the classiier. In this position paper, we posit several open problems and variants o the near-optimal evasion problem. Solutions to these problems would signiicantly advance the state o the art in secure machine learning. Keywords: Query Algorithms, Evasion, Reverse Engineering, Adversarial Learning 1 Introduction A number o systems and security engineers have proposed the use o machine learning techniques or detecting or iltering miscreant activities in a variety o applications, e.g., spam, intrusion, virus, and raud detection. All known detection algorithms have blind spots: classes o miscreant activity that ail to be detected. While learning enables the detector to adapt, adversaries can still exploit blind spots to evade detection. A signiicant challenge is to quantiy how eectively an adversary can discover blind spots by querying the detector. Consider, or example, a spammer who wishes to minimally modiy a spam message so it is not classiied as spam by a public webmail system s spam classiier (here cost is change in the value o the message ater modiication). The spammer can observe the classiier s behavior by creating a dummy account on the webmail system. By observing the responses o the spam detector to queries, the spammer can search or a minimal modiication. Similarly or host-based intrusion detection, an intruder may be orced to obuscate an attack to avoid detection. The attacker can alter their exploit by inserting no-ops, using synonymous system calls, or even choosing between one o several possible exploits. While instances have associated costs to the adversary, a second type o cost suered by the attacker is the number o queries needed to search or evading instances. Larger numbers require additional resources and arouse increasing suspicion o malicious behavior. In practice, adversaries employ a number o techniques to mitigate this second cost, such as opening multiple webmail accounts or rate-limiting spam or worm probing attacks; these strategies require the attacker use additional sophistication and computational resources.

2 Here we revisit the near-optimal evasion problem a theoretical ormulation that quantiies how eectively a classiier can be evaded through querying. We outline open problems and introduce novel variants o the original ormulation. 1.1 The Near-Optimal Evasion Problem The Near-Optimal Evasion Problem is a ormulation o adversarial evasion that quantiies the hardness o search or low-cost negative instances in terms o the number o queries used. As irst posed by Lowd and Meek [6] under the name adversarial classiier reverse engineering, the near-optimal evasion problem is to quantiy the query complexity required by an adversary to ind a near-minimal cost negative instance in terms o the size o the eature space and the desired accuracy. By analyzing the query complexity or a amily o classiiers, nearoptimal evasion provides a notion o how hard that amily is to evade; e.g., i a spammer must send exponentially many queries per dimension o the eature space, it is diicult or the spammer to successully improve their spam. In this setting, we assume instances are represented in a D-dimensional eature space (e.g., X R D ) and the target classiier belongs to a amily F o binary classiiers where each classiier F is a mapping rom eature space X to a label in {' ', '+'}. A deterministic F partitions X into two sets the positive class X + = {x X (x) = '+'} and the analogous negative class X which we take to be the normal instances. We assume that the adversary is aware o at least one instance in each class, x X and x A X +, knows F and not, but can observe (x) or any x X by issuing a membership query. Adversarial Cost We assume the adversary has a cost unction A : X R 0+ ; e.g., or a spammer this could be edit distance on messages. The adversary wishes to optimize A over the negative class X ; e.g., the spammer wants to send spam that will be classiied as normal (' '). Typically the cost unction is a distance to a target instance x A X + that is most desirable to the adversary; e.g., an l p distance induces cost A p (x) = x x A p. Lowd and Meek [6] deine minimal adversarial cost (MAC ) o a classiier to be the best lower bound on the cost that any negative instance obtains MAC (, A) in x X [A (x)]. (1) They urther deine a data point to be an ɛ-approximate instance o minimal adversarial cost (ɛ-imac ) i it is a negative instance with cost no more than a actor o (1 + ɛ) times the MAC ; i.e., every ɛ-imac is a member o the set 1 { } ɛ-imac (, A) x X A (x) (1 + ɛ) MAC (, A). (2) The goal o the Near-Optimal Evasion Problem is to quantiy the worst-case query complexity required or an adversary to iner an ɛ-imac, depending on 1 We use ɛ-imac to reer both to this set and its members.

3 the richness o the amily o classiiers. Finding an ɛ-imac or a singleton amily can be achieved oline without any queries, whereas i the amily is large, many more queries will be necessary to ind an ɛ-imac. Formally, A amily o classiiers F is ɛ-imac searchable under a amily o cost unctions A i or all F and A A, there is an algorithm that inds x ɛ-imac (, A) using polynomially many membership queries in D and L ɛ = log 1 ɛ. We will reer to such an algorithm as eicient. 1.2 Security and the Near-Optimal Evasion Problem The near-optimal evasion problem sheds light on several real security issues. The problem abstracts the scenario o an adversary who wishes to launch a speciic attack that is blocked by a classiier-based deense. The attacker has a limited number o probing opportunities ater which she must send an attack as close as possible to his originally intended attack a near-optimal attack. In the case o spam, the spammer may originally have a message that will be detected as spam. She probes, inds a near-optimal message that evades the ilter, and sends this message instead. In the case o an intruder, she has a preerred sequence o system calls that will be detected as intrusions. She probes, inds and executes a near-optimal sequence that evades the detector. With this ramework in mind, we now clearly see the role o a deender: to provide a classiier that resists near-optimal evasion. Practical implementation requires careul selection o costs and realistic bounds on the number o probes an adversary can perorm. Resulting lower-bounds on the number o probes required or near-optimal evasion provide signiicant evidence o eective security. 1.3 Previous Work Lowd and Meek [6] irst introduced near-optimal evasion, and developed eicient methods that reverse-engineer linear classiiers in both real-valued and Boolean eature spaces or l 1 costs. Further, Nelson et al. [7] generalized their result rom linear classiiers to the amily o convex-inducing classiiers that partition the space o instances into two sets one o which is convex. In generalizing to this amily, they showed that near-optimal evasion does not require an estimate o the classiier s decision boundary or state. Nelson et al. [8] urther explored general l p costs and ound not all are ɛ-imac searchable or convex-inducing classiiers. Dalvi et al. use a cost-sensitive game theoretic approach to preemptively patch a classiier s blind spots [5]. They construct a modiied classiier designed to detect optimally modiied instances. Biggio et al. [3] extend this game theoretic approach and propose hiding inormation or randomization as additional deense mechanisms or this setting. However, they do not explore near-optimal evasion or randomized classiiers as we propose in Section 3.3.

4 2 Open Problems in the Theory o Near-Optimal Evasion A number o unanswered questions remain about the near-optimal evasion problem. Here we motivate these problems and suggest potential directions. Question 1: Can we ind matching upper and lower bounds or evasion algorithms? Is there a deterministic strategy with polynomial query complexity or all convex-inducing classiiers? In previous work, linear and convex-inducing classiiers were shown to be ɛ-imac searchable or l 1 costs by demonstrating algorithms with polynomial query complexity. Currently, it is known that or convex positive class, O ( L ɛ + L ɛ D ) queries are suicient to ind a near-optimal instance, although the tightest known lower bound in this case is O (L ɛ + D). In the case o convex X, the best known algorithm is a randomized ellipsoid approach (c. [2]) that inds a near-optimal instance with high probability using O ( D 5) queries (ignoring logarithmic terms). Question 2: Is there some amily larger than the convex-inducing classiiers that is ɛ-imac searchable? Are there amilies outside o the convex-inducing classiiers or which near-optimal evasion is eicient? Existing approaches to near-optimality have built on the machinery o convex optimization. However, many interesting classiiers are not convex-inducing classiiers. Currently, the only known result due to Lowd and Meek is that linear classiiers on Boolean instance space are 2-IMAC searchable. Question 3: Is some amily o SVMs ( e.g., with a known kernel) ɛ-imac searchable or some ɛ? Can an adversary incorporate the structure o a nonconvex classiier into the ɛ-imac search? Consider SVMs with non-linear kernels. The classiier is non-convex in the original eature space, while in the Reproducing Kernel Hilbert Space the classiier is linear but the cost unction may no longer be easy to minimize. However SVMs have other properties that may acilitate near-optimal evasion. For instance, in cases where there are ew support vectors, one only needs to ind these instances to reconstruct the classiier. Question 4: Are there characteristics o non-convex, contiguous bodies that are indicative o the hardness o the body or near-optimal evasion? What about non-contiguous bodies? It appears that the amily o contiguous bodies (i.e., the set o all classiiers or which either X + or X is a contiguous set) cannot be generally ɛ-imac searchable since this amily includes members with many locally minimal cost regions which are hard or local search or binary search procedures to avoid, but perhaps some subsets o this amily are ɛ-imac searchable. For amilies o non-contiguous bodies, ɛ-imac searchability seems impossible to achieve (disconnected components could be arbitrarily close to x A ) unless the classiiers structure can be exploited; e.g., as we discuss or SVMs above. Question 5: For what classes o classiiers is reverse-engineering as easy as evasion? Reverse-engineering is the process o querying to learn the decision boundary, and is suicient or solving the evasion problem. It is now known that the query complexity o reverse-engineering linear classiiers is identical to that o evasion, while reverse-engineering is strictly more diicult or general convexinducing classiiers [8]. It is unknown whether there exists a class in between linear and convex-inducing classiiers on which the two tasks are equivalent.

5 3 Alternative Models or Evasion Here we suggest a number o variants o near-optimal evasion that generalize or reormulate the original problem to capture new aspects o the overall challenge. 3.1 Additional Inormation about Training Data Distribution Consider an adversary that knows the training algorithm and obtains samples drawn rom a natural distribution. A ew interesting settings include: 1. The adversary s samples are a subset o the training data. 2. The adversary s samples are rom the same distribution as the training data. 3. The adversary s samples are rom a perturbation o the training distribution. With this additional inormation, the adversary may estimate their own classiier and analyze it oline. Some open questions include: Question 6: What can be learned rom about? How can best be used to guide search? Can the sample data be directly incorporated into ɛ-imac - search? Relationships between between and can build on existing results in learning theory. A possibility is to bound the dierence between MAC (, A) and MAC (, A) in one o the above settings. I the dierence is suiciently small with high probability, then a search or an ɛ-imac could use MAC (, A) to initially lower bound MAC (, A). This should reduce search complexity since lower bounds on the MAC are typically harder to obtain than upper bounds. 3.2 Beyond the Membership Oracle Question 7: What types o additional eedback may be available to the adversary and how do they impact the query complexity o ɛ-imac -search? In this scenario, the adversary receives more rom the classiier than just a '+'/' ' label. For instance, suppose the classiier is deined as (x) = I {g (x) > 0} or some realvalued unction g (as is the case or SVMs) and the adversary receives g (x) or every query instead o (x). I g is linear, the adversary can use D + 1 queries and solve a linear regression problem to reverse engineer g. This additional inormation may also be useul or approximating the support o an SVM. 3.3 Evading Randomized Classiiers In this variant o near-optimal evasion, we consider randomized classiiers that generate random responses rom a distribution conditioned on the query x. To analyze the query complexity o such a classiier, we must irst generalize our concept o the MAC. We propose the ollowing candidate generalization: RMAC (, A) = in x X {A (x) + λp ( (x) = ' ')}. I is deterministic, we need λ MAC (, A) or this deinition to be equivalent to Eq. (1) (e.g., λ = A ( x A) + 1 is suicient); otherwise, a trivial minimizer is x A. For a randomized classiier, λ balances cost with probability o success.

6 Question 8: Given access to the membership oracle only, how diicult is near-optimal evasion o randomized classiiers? Are there amilies o randomized classiiers that are ɛ-imac searchable? Potential randomized amilies include: 1. Classiiers with uzzy boundary o width δ around a deterministic boundary 2. Classiiers based on the class-conditional densities or a pair o Gaussians, a logistic regression model, or other members o the exponential amily. Evasion o randomized classiiers seems to be more diicult than or deterministic classiiers as each query provides limited inormation about the query probabilities. Based on this argument, Biggio et al. promote randomized classiiers as a deense against evasion [3]. However, it is not known i randomized classiiers have provable worse query complexities. 3.4 Querying with Real-World Objects Question 9: How can the eature mapping be inverted to design real-world instances to map to desired queries? How can query algorithms be adapted or approximate querying? In the original model o evasion, it was assumed that the attacker could observe (x) or any x X. Implicit in this capability is the assumption that the attacker has knowledge o the eature mapping rom realworld objects such as raw s or network packets, into the eature space in which the deender learns. Even when this is true, the mapping may not be oneto-one or onto: multiple s may map to the same bag-o-words vector, and some instances in eature space may not correspond to any real-world object. 3.5 Evading an Adaptive Classiier Finally we consider a classiier that periodically retrains on queries. This variant is a multi-old game between the attacker and learner, with the adversary now able to issue queries that degrade the learner s perormance. Techniques rom game-theoretic online learning should be well-suited to this setting [4]. Question 10: Given a set o adversarial queries (and possibly additional innocuous data) will the learning algorithm converge to the true boundary or can the adversary deceive the learner and evade it simultaneously? I the algorithm does converge, at what rate? To properly analyze retraining, it is important to have an oracle that labels the points sent by the adversary. I all points sent by the adversary are labeled '+', the classiier may prevent eective evasion, but with a large numbers o alse positives due to the adversary queries in X ; this itsel constitutes an attack against the learner [1]. 4 Conclusion The intersection o security, systems, and machine learning research has yielded signiicant advances in decision-making or complex systems, but has also introduced new challenges in protecting against malicious users. While earlier research

7 laid the groundwork or understanding the near-optimal evasion problem, undamental problems remain unaddressed. In this paper, we discussed several o these problems and variants, and proposed potential avenues or uture research. Reerences 1. M. Barreno, B. Nelson, A. D. Joseph, and J. D. Tygar. The security o machine learning. Machine Learning, To appear. 2. D. Bertsimas and S. Vempala. Solving convex programs by random walks. J. ACM, 51(4): , B. Biggio, G. Fumera, and F. Roli. Multiple classiier systems under attack. In MCS, pages 74 83, N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, N. Dalvi, P. Domingos, Mausam, S. Sanghai, and D. Verma. Adversarial classiication. In Proc. KDD 04, pages , D. Lowd and C. Meek. Adversarial learning. In Pro. KDD 05, pages , B. Nelson, B. I. P. Rubinstein, L. Huang, A. D. Joseph, S. Lau, S. Lee, S. Rao, A. Tran, and J. D. Tygar. Near-optimal evasion o convex-inducing classiiers. In Proc. AISTATS 2010, pages , B. Nelson, B. I. P. Rubinstein, L. Huang, A. D. Joseph, S. J. Lee, S. Rao, and J. D. Tygar. Query strategies or evading convex-inducing classiiers, Report arxiv: v1 available at