Adapting Deep Networks Across Domains, Modalities, and Tasks Judy Hoffman!

Transcription

1 Adapting Deep Networks Across Domains, Modalities, and Tasks Judy Hoffman! Eric Tzeng Saurabh Gupta Kate Saenko Trevor Darrell

2 Recent Visual Recognition Progress ImageNet Performance 100 Accuracy

3 Recent Visual Recognition Progress ImageNet Performance Accuracy Deep models

4 Deep Visual Models [LeCuN 89, 98] [Hubel and Wisel 59] [Krizhevsky 2012] [Szegedy 2014] [Simonyan 2014]

5 Deep Visual Models [LeCuN 89, 98] [Hubel and Wisel 59] [Krizhevsky 2012] [Szegedy 2014] [Simonyan 2014]

6 Domain Adaptation: Train on source adapt to target backpack chair bike Source Domain PS (X, Y ) lots of labeled data DS = {(xi, yi ), 8i 2 {1,..., N }}

7 Domain Adaptation: Train on source adapt to target backpack chair bike Source Domain PS (X, Y ) lots of labeled data DS = {(xi, yi ), 8i 2 {1,..., N }}?? bike Target Domain PT (Z, H) unlabeled or limited labels DT = {(zj,?), 8j 2 {1,..., M }}

8 Domain Adaptation: Train on source adapt to target Adapt backpack chair bike Source Domain PS (X, Y ) lots of labeled data DS = {(xi, yi ), 8i 2 {1,..., N }}?? bike Target Domain PT (Z, H) unlabeled or limited labels DT = {(zj,?), 8j 2 {1,..., M }}

9 Prior work: domain adaptation Minimizing distribution distance! Borgwardt`06, Mansour`09, Pan`09, Fernando`13 Deep model adaptation Chopra`13, Tzeng`14, Long`15, Ganin`15

10 Adapting across domains minimize discrepancy min repr L conf (x, z, D ; repr )= X x i 2S H(U(D),q s i )+ X z j 2T H(U(D),q t j) [ICCV 2015]

11 Adapting across domains minimize discrepancy min repr L conf (x, z, D ; repr )= X x i 2S H(U(D),q s i )+ X z j 2T H(U(D),q t j) x i [ICCV 2015]

12 Adapting across domains minimize discrepancy min repr L conf (x, z, D ; repr )= X x i 2S H(U(D),q s i )+ X z j 2T H(U(D),q t j) c object classifier x i [ICCV 2015]

13 Adapting across domains minimize discrepancy min repr L conf (x, z, D ; repr )= X x i 2S H(U(D),q s i )+ X z j 2T H(U(D),q t j) z j c object classifier x i [ICCV 2015]

14 Adapting across domains minimize discrepancy min repr L conf (x, z, D ; repr )= X x i 2S H(U(D),q s i )+ X z j 2T H(U(D),q t j) z j Discrepency c object classifier x i [ICCV 2015]

15 Adapting across domains minimize discrepancy min repr L conf (x, z, D ; repr )= X x i 2S H(U(D),q s i )+ X z j 2T H(U(D),q t j) z j c object classifier x i D domain classifier [ICCV 2015]

16 Adapting across domains minimize discrepancy min repr L conf (x, z, D ; repr )= X x i 2S H(U(D),q s i )+ X z j 2T H(U(D),q t j) z j q s i =[1, 0] q t j =[0, 1] x i D domain classifier [ICCV 2015]

17 Adapting across domains minimize discrepancy min repr L conf (x, z, D ; repr )= X x i 2S H(U(D),q s i )+ X z j 2T H(U(D),q t j) Cross-entropy with uniform distribution z j x i D domain classifier [ICCV 2015]

18 Adapting across domains minimize discrepancy min repr L conf (x, z, D ; repr )= X x i 2S z j H(U(D),q s i )+ X z j 2T H(U(D),q t j) x i D domain classifier [ICCV 2015]

19 Adapting across domains minimize discrepancy min repr L conf (x, z, D ; repr )= X x i 2S z j H(U(D),q s i )+ X z j 2T H(U(D),q t j) c object classifier x i [ICCV 2015]

20 Adapting across domains minimize discrepancy backpack chair bike conv1 conv5 fc6 fc7 source data fc8 classification loss Source Data

21 Adapting across domains minimize discrepancy backpack chair bike conv1 conv5 fc6 fc7 source data fc8 Source Data shared shared shared shared shared classification loss backpack Target Data conv1 conv5 fc6 fc7 fc8 labeled target data?

22 Adapting across domains minimize discrepancy backpack chair bike conv1 conv5 fc6 fc7 source data fc8 backpack Source Data Target Data shared shared shared shared conv1 conv5 fc6 fc7 source data all target data fcd domain confusion loss domain classifier loss shared fc8 classification loss labeled target data?

23 Verify confusion

24 Domain Adaptation: Train on source adapt to target Adapt backpack chair bike Source Domain PS (X, Y ) lots of labeled data DS = {(xi, yi ), 8i 2 {1,..., N }}?? bike Target Domain PT (Z, H) unlabeled or limited labels DT = {(zj,?), 8j 2 {1,..., M }}

25 Domain Adaptation: Train on source adapt to target Adapt backpack chair bike Source Domain PS (X, Y ) lots of labeled data DS = {(xi, yi ), 8i 2 {1,..., N }}?? bike Target Domain PT (Z, H) unlabeled or limited labels DT = {(zj,?), 8j 2 {1,..., M }}

26 Standard supervised deep loss H(h, p) =E h [ log p] [ICCV 2015]

27 Standard supervised deep loss H(h, p) =E h [ log p] z j p j conv1 conv5 fc6 fc7 fc8 Bottle MugChair Laptop Keyboard [ICCV 2015]

28 Standard supervised deep loss H(h, p) =E h [ log p] z j p j conv1 conv5 fc6 fc7 fc8 Bottle MugChair Laptop Keyboard bottle h j Bottle MugChair Laptop Keyboard [ICCV 2015]

29 Source Softlabels Source CNN softmax high temp Bottle MugChair Laptop Keyboard Source CNN softmax high temp Bottle MugChair Laptop Keyboard + Bottle MugChair Laptop Keyboard source bottle examples Source CNN softmax high temp Bottle MugChair Laptop Keyboard

30 Standard supervised deep loss H(h, p) =E h [ log p] z j p j conv1 conv5 fc6 fc7 fc8 Bottle MugChair Laptop Keyboard bottle Bottle MugChair h j Laptop Keyboard [ICCV 2015]

31 Class correlation transfer loss H(h, p) =E h [ log p] z j p j conv1 conv5 fc6 fc7 fc8 Bottle MugChair Laptop Keyboard bottle h j Bottle MugChair Laptop Keyboard source softlabel [ICCV 2015]

32 Class correlation transfer loss backpack chair bike conv1 conv5 fc6 fc7 source data fc8 source data Source Data shared shared shared shared fcd domain confusion loss shared classification loss backpack Target Data conv1 conv5 fc6 fc7 all target data domain classifier loss fc8 labeled target data?

33 Class correlation transfer loss backpack chair bike conv1 conv5 fc6 fc7 source data fc8 source data Source Data shared shared shared shared fcd domain confusion loss shared classification loss backpack Target Data conv1 conv5 fc6 fc7 all target data labeled target data domain classifier loss fc8 softmax high temp softlabel loss Source softlabels?

34 Office dataset Experiment Adapting Visual Category Models to New Domains 9 31 categories laptop instance 2 instance 1 letter tray... webcam instance 5 keyboard dslr instance 5 all classes have source labeled examples 15 classes have target labeled examples evaluate on remaining 16 classes headphones amazon... file cabinet instance 2 instance domains Fig. 4. New dataset for investigating domain shifts in visual category recognition tasks. Images of objects from 31 categories are downloaded from the web as well as captured by a high definition and a low definition camera. [saenko`10] popular way to acquire data, as it allows for easy access to large amounts of

35 Office dataset Experiment A! W A! D D! A D! W W! A W! D Average MMDT [18] 44.6 ± ± 0.5 Source CNN 54.2 ± ± ± ± ± ± Ours: dom confusion only 55.2 ± ± ± ± ± ± Ours: soft labels only 56.8 ± ± ± ± ± ± Ours: dom confusion+soft labels 59.3 ± ± ± ± ± ± Table 2. Multi-class accuracy evaluation on the standard semi-supervised adaptation setting with the Office dataset. We evaluate on 16 Multiclass accuracy over 16 classes which lack target labels

36 Baseline soft label Baseline soft activation ring binder monitor Target test image back pack bike bike helmet bookcase bottle calculator desk chair desk lamp desktop computer file cabinet headphones keyboard laptop computer letter tray mobile phone monitor mouse mug paper notebook pen Ours soft label Our soft activation phone printer projector punchers ring binder ruler scissors speaker stapler tape dispenser trash can back pack bike bike helmet bookcase bottle calculator desk chair desk lamp desktop computer file cabinet headphones keyboard laptop computer letter tray mobile phone monitor mouse mug paper notebook pen phone printer projector punchers ring binder ruler scissors speaker stapler tape dispenser trash can

37 Source soft labels back pack bike bike helmet bookcase bottle calculator desk chair desk lamp desktop computer file cabinet headphones keyboard laptop computer letter tray mobile phone

38 Cross-dataset Experiment Setup Source: ImageNet! Target: Caltech256! 40 categories! Evaluate adaptation performance with 0,1,3,5 target labeled examples per class [tommasi`14]

39 ImageNet adapted to Caltech Multi-class Multiclass Accuracy Source+Target CNN Ours: softlabels only Ours: dom confusion+softlabels Number Number Labeled of Target labeled Examples target examples per Category [ICCV 2015]

40 Summary: simultaneous transfer across domains and tasks Domain confusion aligns the distributions Softlabels transfer class correlations Paper presented in poster session Wednesday 12/16 4B

41 Discrepancy due to modality shift Lots of data to train models RGB System uses model

42 Label space discrepancy lamp night-stand lamp bed bed pillow night-stand Current output System uses model [NIPS 2014, CVPR 2015]

43 Label space discrepancy lamp night-stand lamp bed bed pillow night-stand Desired output System uses model [NIPS 2014, CVPR 2015]

44 Adapting Deep Visual Models Adapting across domains Adapting across tasks lamp bed pillow night-stand Adapting across modalities Error bounds on adapted deep models Generally applicable to adaptation with deep learning in AI

45 Thank you.