Word-Conditioned Image Style Transfer. Yu Sugiyama and Keiji Yanai The University of Electro-Communications, Tokyo

Transcription

1 Word-Conditioned Image Style Transfer Yu Sugiyama and Keiji Yanai The University of Electro-Communications, Tokyo 1

2 Introduction Neural Style Transfer, Image style transfer 2018/11/27 UEC Yanai Lab. Tokyo. 2

3 Introduction Style Transfer requires CONTENT and STYLE image User need to find good image 2018/12/03 UEC Yanai Lab. Tokyo. 3

4 Objective Words condition makes easy to find good style "Leaves 4

5 Related Work 1. A Neural Algorithm of Artistic Style Leon A. Gatys, CVPR, 2016 Image style synthesis for artistic style 2. Perceptual Losses for Real-Time Style Transfer and Super-Resolution Jastin Johnson, ECCV, 2016 Pre-training stylization network to fast image transfer 5

6 Related Work 3. Unseen Style Transfer Keiji Yanai, ICLR WS, 2017 improvement of fast style transfer Stylize image with un-trained images 4. Arbitrary Style Transfer Golnaz Ghiasi, Honglak Lee, et al. BMVC, 2017 Different method of Unseen Style Transfer Synthesis images with Conditional Instance Normalization 6

7 Method Arbitrary Style Transfer Network 2018/12/03 UEC Yanai Lab. Tokyo. 7

8 Method SSN(Style Selector Network) predicts Style Vector for transfer network 2018/12/03 UEC Yanai Lab. Tokyo. 8

9 Train Word2Vec Generate 200-dim vector from words English Wikipedia Corpus Reduce row corpus Remove all Stopping words, low frequency word(under 5 times) Random remove High frequency word(over 1000 times) 9

10 Method SSN makes Style vector same as Inception-v3 feature extractor Inception v3 (Style Precision Network) Style Vector Approximate ``Leather Word 2 Vec Embedding vector Style Selector Network Style Vector 2018/12/03 UEC Yanai Lab. Tokyo.

11 Train Dataset Stands by Yahoo100M wild images data Random select 1000 images for 500 categories Adverbs tag annotation Contains 500,000 image-word pairs 11

12 Experiments - Detail Optimize with Adam Style transfer network and style prediction network are pre-trained Train Only 3 FC layers Training takes 10 min 12

13 Experiments L2 minimalize instead of adversarial loss Adversarial Loss input new urban indoor L2 Loss new urban indoor 13

14 Experiments There are some mismatches between content images and style words input Mismatch old new input match old new 14

15 Experiments - problem Word2Vec cannot distinct summer leaves and fall leaves from only leaves One word is not enough explain visual feature leaves 15

16 Experiments Leather images dataset Crawled by Google image search Search 84 keywords About 500 images each keywords Leather advanced bag Leather Advanced Ancient Recent elderly etc. * * Bags Shoes wallet Leather ancient bag 2018/12/3 UEC Yanai Lab. Tokyo. 16

17 Experiments Trained SSN only leather images model INPUT new current advanced elderly fossil old 17

18 Results and Discussions Adversarial loss generate superior feature map than L2 loss L2 loss model generates mean feature of trained images No confidence to match the word meaning and visual feature leather experiments solved several problems Cannot preserve background and object domain Style transfer architectures are not match perfect. 18

19 Conclusion and Future Work Conclusion Image Stylization with words without conditional approach There are question the transformation is it right for our feelings. Future Work LSTM units for use sentences not only words Other domain transfer techniques for image synthesis network 19

20 2018/12/1 UEC Yanai Lab. Tokyo. 20

21 Typical Bad results Input red --> image styled darker without color change Input black --> sometime image styled blight I think dataset not only that words area leather red wallet image --> black wallet with red emblem Attention model can solve this problem Segment red area of images 2018/12/1 UEC Yanai Lab. Tokyo. 21

22 Experiments input old new country urban 2018/11/27 UEC Yanai Lab. Tokyo. 22

23 2018/12/1 UEC Yanai Lab. Tokyo. 23

24 2018/12/1 UEC Yanai Lab. Tokyo. 24