Unsupervised Domain Adaptation by Backpropagation. Chih-Hui Ho, Xingyu Gu, Yuan Qi

Transcription

1 Unsupervised Domain Adaptation by Backpropagation Chih-Hui Ho, Xingyu Gu, Yuan Qi

2 Problems Deep network: requires massive labeled training data. Labeled data: Available sometimes: Image recognition Speech recognition Recommendation Difficult to collect sometimes: Robotics Disaster Bioinformatics Medical diagnosis

3 Problems Test time failure: distribution of actual data is different from training data. Example: Model is Trained on synthetic data (abundant and fully labeled), but Tested on real data. MJSynth (synthetic) IIIT5K (real)

4 Results Source datapoint Target datapoint MNIST & MNIST-M (extracted features) Adaptation SYN NUMBERS & SVHN (label classifier s last hidden layer) Adaptation

5 Objective Given: Lots of labeled data in the source domain (e.g. synthetic images) Lots of unlabeled data in the target domain (e.g. real images) Domain Adaptation (DA): In the presence of a shift between source and target domain, Train a network on source domain that performs good on target domain.

6 Objective Example: Office dataset Source: Amazon photos of office objects (on white background) Amazon Target: Consumer photos of office objects (taken by DSLR camera / webcam) DSLR Webcam

7 Previous Approaches - MMD Maximum Mean Discrepancy (measures domain-distance) Reweighing target domain images. Distance between source and target domains. Explicit distance measurement (e.g. kernel Hilbert space).

8 Previous Approaches - DLID Deep Learning by Interpolating between Domains Feature transformation mapping source into target. Train feature extractor layer-wise. Gradually replacing source samples with target samples. Train classifier on features.

9 Proposed Solution - Deep Domain Adaptation (DDA) Standard CNN + domain classifier. An implicit way to measure similarity between source and target. If domain classifier performs good: dissimilar features. If domain classifier performs bad: similar features. Objective: feature is best for label classifier (discriminative) worst for domain classifier (invariant)

10 Improvement Previous approaches (DLID, MMD, etc.) Proposed solution (DDA) Measurement of similarity between domains Explicit (distance in Hilbert space) Implicit (performance of domain classifier) Training steps Separate feature extractor and label classifier Jointly trained by backpropagation Architecture Complicated Simple (standard CNN + domain classifier)

11 Proposed Solution

12 Proposed Solution

13 Proposed Solution Label predictor

14 Proposed Solution

15 Proposed Solution

16 Proposed Solution Consider an image from source domain

17 Proposed Solution Consider an image from target domain ---

18 Proposed Solution

19 Proposed Solution

20 Proposed Solution How to backpropagate the label classifier loss? Consider only the upper architecture This is typical backpropagation

21 Proposed Solution How to backpropagate the domain classifier loss? Consider only the lower architecture Define gradient reversal layer (GRL) +

22 Proposed Solution Forward Backward

23 Proposed Solution After training, the label predictor can be used to predict labels for samples from either source or target domain Experiment results

24 Source & Target Datasets Perform on large-scale datasets of small images, i.e.. MNIST, SVNH, GTSRB. Perform on small-scale dataset, i.e. the OFFICE datasets (Saenko et al., 2010), which are a standard for domain adaptation in computer vision.

25 Table Explanation Source Only: the lower performance bound (i.e. if no adaptation is performed) SA: The DA method proposed in FERNANDO ET AL., 2013 Train on Target: training on the target domain data with known class labels (upper bound on the DA performance) Data in the bracket: show how much of the gap between the lower and the upper bounds was covered, e.g. 7.9% = ( lower_bound)/(upper_bound - lower_bound)

26 MNIST MNIST-M

27 MNIST MNIST-M Blue points correspond to the source domain examples, while red ones correspond to the target domain. Utilize t-sne (van der Maaten, 2013) visualizations of the CNN s activations. The adaptation makes the two distributions of features much closer.

28 Synthetic numbers SVHN

29 Synthetic numbers SVHN

30 MNIST SVHN The two directions (MNIST SVHN and SVHN MNIST) are not equally difficult. SVHN is more diverse, a model trained on SVHN is expected to be more generic and to perform reasonably on the MNIST dataset. Unsupervised adaptation from MNIST to SVHN gives a failure example for this approach.

31 SVHN MNIST (Judy Hoffman et al.2017) classification accuracy for SVHN->MNIST 90.4% ± 0.4%

32 Synthetic Signs GTSRB

33 Synthetic Signs GTSRB Semi-supervised domain adaptation, i.e. when one is additionally provided with a small amount of labeled target data. (Future work) 100,000 synthetic and 430 real images respectively

34 Office dataset Amazon DSLR Webcam

35 Office dataset

36 Summary Proposed a new approach to unsupervised domain adaptation; Unlike previous approaches, this approach can be implemented to any deep learning models; Improve the state-of-art results at that time.