Adversarial Examples in Deep Learning. Cho-Jui Hsieh Computer Science & Statistics UC Davis

Transcription

1 Adversarial Examples in Deep Learning Cho-Jui Hsieh Computer Science & Statistics UC Davis

2 Adversarial Example

3 Adversarial Example

4 More Examples Robustness is critical in real systems

5 More Examples Robustness is critical in real systems Real world adversarial examples: Attack stop sign in real world (Evtimov et al., 2017) Adversarial turtle (Athalye et al., 2017)

6 Research Questions Questions: Attack: How to generate adversarial example? (AISec 17, Arxiv 17) Verification: evaluate the robustness of deep networks (ICLR 18, Arxiv 18) Defense: how to improve robustness? (Arxiv 17)

7 Notations and Attack Procedure

8 Attack as an optimization problem Input image: x 0 R d Adversarial image: x R d x = arg min x Dis(x, x 0 ) + c g(x)

9 Attack as an optimization problem Input image: x 0 R d Adversarial image: x R d arg min x Dis(x, x 0 ) + c g(x) Dis(x, x 0 ): the distortion, e.g., x x 0 2 2

10 Attack as an optimization problem Input image: x 0 R d Adversarial image: x R d arg min x x x c g(x) Dis(x, x 0 ): the distortion g(x): loss to measure the successfulness of attack

11 Attack as an optimization problem Input image: x 0 R d Adversarial image: x R d arg min x x x c g(x) Dis(x, x 0 ): the distortion g(x): loss to measure the successfulness of attack c 0 controls the trade-off See (Carlini & Wagner, 2017)

12 Define successfulness of attack

13 Define successfulness of attack Untargeted attack: success if f j (x) y 0 g(x) = max{f y0 (x) max j y 0 f j (x), 0}

14 Define successfulness of attack Targeted attack: success if f j (x ) = t g(x) = max{max j t f j(x) f t (x), 0}

15 White box setting x = arg min x Dis(x, x 0 ) + c g(x) (1) Model f ( ) is revealed to attacker gradient of g(x) can be computed by back-propagation attacker minimizes (1) by gradient descent

16 White box setting x = arg min x Dis(x, x 0 ) + c g(x) (1) Model f ( ) is revealed to attacker gradient of g(x) can be computed by back-propagation attacker minimizes (1) by gradient descent Can be generalized to other definition of Dis(x, x 0 ) (e.g., l 1 penalty, Elastic net, ) EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples. [C, S, Z, Y, H], AAAI, 2018.

17 Attack Image Captioning Targeted attack Output a specified sentence Keywords attack output contains specified keywords Show-and-Fool: Crafting Adversarial Examples for Neural Image Captioning. [C, Z, C, Y, H, 2017]

18 Attack Image Captioning Usually use CNN+RNN network (Figure from Show and Tell, Vinyals et al, 2014)

19 Attack Image Captioning Usually use CNN+RNN network (Figure from Show and Tell, Vinyals et al, 2014) Define the loss on sequence output for different attacks

20 Black Box Setting In practice, the deep network structure/parameters are not revealed to attackers cannot compute gradient f (x)

21 Black Box Setting In practice, the deep network structure/parameters are not revealed to attackers cannot compute gradient f (x) Attacker can query the ML model and get the probability output

22 Black Box Setting In practice, the deep network structure/parameters are not revealed to attackers cannot compute gradient f (x) Attacker can query the ML model and get the probability output Previous approach (Papernot et al., 2016): Train a substitute model using (x 1, f (x 1 )),, (x q, f (x q )) Attack this substitute model

23 Black Box Setting arg min x Dis(x, x 0 ) + c g(x) We can actually solve the same optimization problem! ZOO: Zeroth Order Optimization based Black-box Attacks. [C, Z, S, Y, H], CCS AI-Security 2017

24 Black Box Setting arg min x Dis(x, x 0 ) + c g(x) We can actually solve the same optimization problem! Symmetric difference quotient to estimate gradient: ĝ i F (x) F (x + ɛe i) F (x ɛe i ) x i 2ɛ ZOO: Zeroth Order Optimization based Black-box Attacks. [C, Z, S, Y, H], CCS AI-Security 2017

25 Black Box Setting arg min x Dis(x, x 0 ) + c g(x) We can actually solve the same optimization problem! Symmetric difference quotient to estimate gradient: Zeroth order gradient descent ĝ i i g(x) g(x + ɛe i) g(x ɛe i ) 2ɛ ĝ 1 x x η. ĝ d ZOO: Zeroth Order Optimization based Black-box Attacks. [C, Z, S, Y, H], CCS AI-Security 2017

26 Black Box Setting However, Need O(d) queries to estimate gradient

27 Black Box Setting However, Need O(d) queries to estimate gradient ImageNet: d = > 268K 100 iterations 26 million queries

28 Black Box Setting However, Need O(d) queries to estimate gradient ImageNet: d = > 268K 100 iterations 26 million queries Reduce number of queries: Stochastic Coordinate descent: update a small set of coordinates at each time Greedy approach: select important coordinates first Attack-space dimension reduction + hierarchical attack y pixel coordinate y pixel coordinate y pixel coordinate x pixel coordinate x pixel coordinate x pixel coordinate 0.3

29 Black-box Attack on MNIST & CIFAR10 100% success rate, similar distortion with white box attack. MNIST Success Rate Avg. L 2 Avg. Time (per attack) White-box 100 % min Black-box (Substitute Model) % min ( min) Proposed black-box (ZOO) 98.9 % min CIFAR10 Success Rate Avg. L 2 Avg. Time (per attack) White-box 100 % min Black-box (Substitute Model) 5.3 % min ( min) Proposed Black-box (ZOO) 96.8 % min

30 Black-box Attack on ImageNet (Inception-v3)

31 Research Questions Questions: Attack: How to attack? Verification: Evaluate the robustness of your model Defense: how to improve robustness?

32 Measuring Robustness How to measure the robustness of a neural network?

33 Measuring Robustness How to measure the robustness of a neural network? The measurement should be independent to attack algorithms!

34 Measuring Robustness How to measure the robustness of a neural network? The measurement should be independent to attack algorithms! Robustness: For an example x 0, find r such that f (x 0 + ) is correct for all < r

35 Measuring Robustness Theorem: For a model f ( ) and an example x 0, f (x 0 + ) = y for all < min j c f y (x 0 ) f j (x 0 ) L j, where L j = max{ f y (x) f j (x) : x S} Evaluating the robustness of neural network: an extreme value theory approach. [W, Z, Y, G, H, D], ICLR, 2018.

36 How to evaluate L j? L j = max{ h(x) : x S} First attempt: Solve as an optimization problem

37 How to evaluate L j? L j = max{ h(x) : x S} First attempt: Solve as an optimization problem Doesn t work : f ( ) is non-convex and often non-smooth

38 Sampling Approach L j = max{ h(x) : x S} Naive Sampling Algorithm Sample x 1, x 2,, x T S and select max

39 Sampling Approach L j = max{ h(x) : x S} Naive Sampling Algorithm Sample x 1, x 2,, x T S and select max Dimension is too high Non-meaningful solution even with > 1 million samples

40 Extreme Value Theory Extreme Value Theorem (Fisher-Tippett-Gnedenko Theorem) Let X 1,, X n be iid random variables and M n = max{x 1,, X n }. Define the cdf of M n : P(M n x) = G n (x). When n, G n belongs to either the Gumbel, Frechet, or Reverse Weibull family.

41 Evaluate robustness Computing CLEVER score P {ϕ} For i = 1, 2,, M Randomly sample x 1,, x N S Compute b max i=1,,n h(x i ) P P b â MLE of location parameter of reverse Weibull distribution on S r h(x 0) h(x 0) L j â (CLEVER score)

42 Experiments CW Attack: upper bound of r CLEVER: estimated lower-bound of r SLOPE: previous algorithm for estimating r CW Attack CLEVER SLOPE MNIST-CNN MNIST-DD MNIST-BReLU CIFAR-CNN CIFAR-DD CIFAR-BReLU Resnet MobileNet DD, BReLU: defensive algorithms

43 Not a certificated bound Estimated lower bound can be sometimes larger than r A safe zone only in probability

44 Can we do better for ReLU Network? State-of-the-art networks use ReLU activation: θ(x) = max(0, x) Goal: provide a certified, meaningful and computational efficient lower bound of r

45 Previous Work ReLUplex (Katz et al., 2017): combinatorial optimization, Network with 300 neurons need 3 hours Linear Programming (Kolter & Wong, 2017): Network with 5120 neurons need 8 hours State-of-the-art networks have millions of nuorons

46 A linear time algorithm Assume perturbation is bounded: x i = x i + δ i, 0.1 δ 0.1 f 1 (x) = W 4 θ(w 3 θ(w 2 θ(w 1 (x + δ))))

47 A linear time algorithm Activated neuron: input 0 (for all perturbations) output = max(0, input) = input All neurons activated Linear neural network f 1 (x + δ) = W 4 W 3 W 2 W 1 (x + δ)

48 A linear time algorithm Inactivated neuron: input<0 for all perturbations Remove those nodes, still linear output = max(0, input) = 0

49 A linear time algorithm Unsure neuron: input can be positive or negative Deal with both cases exponential time algorithms (previous approaches)

50 A linear time algorithm Assume we know l input u, how to approximate?

51 A linear time algorithm Assume we know l input u, how to approximate? u input max(0, input) u u l (input l) u l

52 A linear time algorithm Replace by a linear function with an added bias term

53 A linear time algorithm Form the linear approximation layer by layer Now Upper/lower bounds of f j (x) can be easily computed

54 Moreover Same time complexity with forward-propagation!

55 Moreover Same time complexity with forward-propagation! We have another algorithm to compute Lipchitz constant (gradient upperbound) Similar to back-propagation!

56 Experiments Fast-Lin: Our method (python) ReluPlex: Exponential time algorithm for computing exact bound (Katz et al., 2017) LP: Linear programming for lower bound (Kolter and Wong, 2017) Network Method Time δ MNIST 2 20 Fast-Lin ReluPlex LP MNIST 3 20 Fast-Lin ReluPlex LP MNIST Fast-Lin LP CIFAR Fast-Lin

57 Research Questions Questions: Attack: How to attack? Verification: Evaluate the robustness of your model Defense: how to improve robustness?

58 Defense from Adversarial Attacks Adversarial retraining (Goodfellow et al., 2015): Adding adversarial examples into training set Robust optimization + BReLu (Zantedeschi et al., 2017) min w E (x,y) DE x N(0,σ 2 )loss(f w (x + x), y) Security community: detect adversarial examples MagNet: (Meng and Chen, 2017), (Li and Li, 2017),

59 Defense from Adversarial Attacks Adversarial retraining (Goodfellow et al., 2015): Adding adversarial examples into training set Robust optimization + BReLu (Zantedeschi et al., 2017) min w E (x,y) DE x N(0,σ 2 )loss(f w (x + x), y) Security community: detect adversarial examples MagNet: (Meng and Chen, 2017), (Li and Li, 2017), However, they are still vulnerable if we choose a better way to attack (Carlini and Wagner, 2017, MagNet and Efficient Defenses Against Adversarial Attacks are Not Robust to Adversarial Examples)

60 Adding randomness to the model All the attacks are based on gradient computation

61 Adding randomness to the model All the attacks are based on gradient computation Idea I: Adding randomness to fool the attackers f (x) f ɛ (x) where ɛ is random Resistant to Black-box attack: gradient cannot be estimated f ɛ1 (x + he i ) f ɛ2 (x he i ) 2h i f (x)

62 Adding randomness to the model Naive approach: adding randomness only in the testing phase prediction accuracy 87% 20% (Cifar-10 with VGG)

63 Ensemble How to boost the accuracy?

64 Ensemble How to boost the accuracy? Idea II: Ensemble can improve the robustness of model: ensemble T models f ɛ1 (x),, f ɛt (x) Our approach: Random + Ensemble

65 Our approach (RSE) Prediction: Ensemble of random models p = T f ɛj (w, x), j=1 and predict ŷ = arg max p k k

66 Our approach (RSE) Prediction: Ensemble of random models p = T f ɛj (w, x), j=1 and predict ŷ = arg max p k k Training: minimizing an upper bound E (x,y) D E ɛ N loss(f ɛ (x), y) E (x,y) D loss(e ɛ N f ɛ (x), y) Training by SGD: sample (x, y), ɛ and conduct w w η w loss(f ɛ (x), y)

67 Our approach (RSE) Prediction: Ensemble of random models p = T f ɛj (w, x), j=1 and predict ŷ = arg max p k k Training: minimizing an upper bound E (x,y) D E ɛ N loss(f ɛ (x), y) E (x,y) D loss(e ɛ N f ɛ (x), y) Training by SGD: sample (x, y), ɛ and conduct w w η w loss(f ɛ (x), y) Implicit regularizing Lipchitz constant: E ɛ N(0,σ 2 )loss(f ɛ (w, x i ), y i ) loss(f 0 (w, x i ), y i ) + σ2 2 L l f 0,

68 Experiments We fix σ init = 0.4 and σ inner = 0.1 for all the datasets/networks Accuracy (%) STL-10 + ModelA No defense Adv retraining Robust Opt+BReLU RSE Distill CIFAR10 + VGG16 No defense Adv retraining Robust Opt+BReLU RSE Distill CIFAR10 + ResNeXt No defense Adv retraining Robust Opt+BReLU RSE Distill Avg. Distortion C No defense Adv retraining Robust Opt+BReLU RSE Distill C C 1.2 No defense Adv retraining 1.0 Robust Opt+BReLU 0.8 RSE Distill C C 1.0 No defense Adv retraining 0.8 Robust Opt+BReLU RSE 0.6 Distill C

69 Targeted attack Original image: Targeted attack

70 References [1] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, C.-J. Hsieh. ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models. AISec, [2] H. Chen, H. Zhang, P.-Y. Chen, J. Yi, C.-J. Hsieh. Show-and-Fool: Crafting Adversarial Examples for Neural Image Captioning. Arxiv [3] X. Liu, M. Cheng, H. Zhang, C.-J. Hsieh. Towards Robust Neural Networks via Random Self-ensemble. Arxiv, [4] P.-Y. Chen, Y. Sharma, H. Zhang, J. Yi, C.-J. Hsieh. EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples. AAAI [5] T.-W. Weng, H. Zhang, P.-Y. Chen, D. Su, Y. Gao, J. Yi, C.-J. Hsieh, L. Daniel. Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach. ICLR, [6] N. Carlini, D. Wagner.Towards evaluating the robustness of neural networks. SC [7] I. Goodfellow, J. Shlens, C. Szegedy. Explaining and Harnessing Adversarial Examples. ICLR [8] V. Zantedeschi, M. Nicolae, A. Rawat. Efficient Defenses Against Adversarial Attacks. Arxiv, [9] D. Meng, H. Chen. MagNet: a Two-Pronged Defense against Adversarial Examples. CCS, [10] N. Carlini, D. Wagner. MagNet and Efficient Defenses Against Adversarial Attacks are Not Robust to Adversarial Examples. Arxiv, 2017.