Dialectical GAN for SAR Image Translation: From Sentinel-1 to TerraSAR-X

Transcription

1 Artice Diaectica GAN for SAR Image Transation: From Sentine-1 to TerraSAR-X Dongyang Ao 1,2, Corneiu Octavian Dumitru 1,Gottfried Schwarz 1 and Mihai Datcu 1, * 1 German Aerospace Center (DLR), Münchener Str. 20, Wessing, Germany; corneiu.dumitru@dr.de gottfried.schwarz@dr.de 2 Schoo of Information and Eectronics, Beijing Institute of Technoogy, Beijing , China; aodongyang@foxmai.com (D.A.); * Correspondence: mihai.datcu@dr.de; Te.: Received: date; Accepted: date; Pubished: date Abstract: Contrary to optica images, Synthetic Aperture Radar (SAR) images are in different eectromagnetic spectrum where the human visua system is not accustomed to. Thus, with more and more SAR appications, the demand for enhanced high-quaity SAR images has increased consideraby. However, high-quaity SAR images entai high costs due to the imitations of current SAR devices and their image processing resources. To improve the quaity of SAR images and to reduce the costs of their generation, we propose a Diaectica Generative Adversaria Network (Diaectica GAN) to generate high-quaity SAR images. This method is based on the anaysis of hierarchica SAR information and the diaectica structure of GAN frameworks. As a demonstration, a typica exampe wi be shown where a ow-resoution SAR image (e.g., a Sentine-1 image) with arge ground coverage is transated into a high-resoution SAR image (e.g., a TerraSAR-X image). Three traditiona agorithms are compared, and a new agorithm is proposed based on a network framework by combining conditiona WGAN-GP (Wasserstein Generative Adversaria Network - Gradient Penaty) oss functions and Spatia Gram matrices under the rue of diaectics. Experimenta resuts show that the SAR image transation works very we when we compare the resuts of our proposed method with the seected traditiona methods. Keywords: diaectica generative adversaria network; image transation; Sentine-1; TerraSAR-X. 1. Introduction In remote sensing, SAR images are we-known for their a-time and a-weather capabiities. In the 1950s, the first SAR system was invented [1]. However, the design and impementation of a SAR system is a compex system engineering and costs many resources, both in money and inteectua effort. Therefore, most SAR instruments on sateites are supported by government organizations. For exampe, the German Aerospace Center (DLR) and EADS Astrium had jointy aunched TerraSAR-X in 2007 [2] and TanDEM-X in 2010 [3]. The Canadian Space Agency (CSA) had aunched in 1995 the RADARSAT-1 and in 2007 the RADARSAT-2 sateites [4], whie the Itaian Ministry of Research and the Ministry of Defence together with the Itaian Space Agency (ASI) had aunched the COSMO-SkyMed -1, 2, 3, and 4 sateites in 2007, 2008 and 2010 [5]. The European Space Agency (ESA) had aunched the Sentine-1 SAR sateite in 2014 [6]. In addition, there are many governments and institutions having aunched their own SAR sateites [7], [8]. Nowadays, SAR has become one of the most vauabe toos for remote sensing of the Earth and its environment. In the era of big data, deep earning can accommodate arge amount of data and generate promising new appications. With the recent deveopment of deep earning, image transation is an easy way to obtain high-quaity SAR images. Transation is a word borrowed from the inguistic fied which denotes the change from one anguage to another one. This transation is often appied

2 2 of 22 when one anguage is hard to understand whie another one is more famiiar to us. Though the two anguages have different vocabuaries and grammars, the transation is premised on the identity of the contents. In genera for image transation there are two sides of the transation, namey the two images coming from different sensors. In this paper, we demonstrate a typica exampe where a ow-resoution SAR image (e.g., a Sentine-1 image) with arge ground coverage is transated using deep earning into a high-resoution SAR image (e.g., a TerraSAR-X image). To some extent, this kind of transation is reated to super-resoution and neura stye transfer. From 2013, deep earning has becomes a popuar too for many appications, such as image recognition, cassification, semantic segmentation, target detection, etc. The first miestone in deep earning based image transation is Gatys et a. s paper [9]. They introduced the Visua Geometry Group (VGG) networks, a pre-trained neura network used for ImageNet in order to define the content and stye information of images, which provides a framework for image transation under the background of deep earning. Within a neura network based framework, many researchers have proposed their own methods for their specific purposes [10], [11]. The second miestone is the invention of Generative Adversaria Networks (GANs) that was made by Goodfeow et a. [12]. As a generative neura network, it seems that a GAN is we-suited for image transation. According to the conception presented in [13], the image transation can be regarded as the pix2pix task, and the authors of [13] have unitized a conditiona GAN to carry out image transations. Inspired by this paper, we think that we can appy these agorithms to do SAR image transation. In SAR image processing, there are many papers about how to use deep earning for cassification, segmentation, etc. [14], [15]. However, itte attention has been paid to the transation between different SAR instruments using deep earning. Transation of Sentine-1 data to high-resoution images ike TerraSAR-X has attracted great interest within the remote sensing community. First, the high resoution of TerraSAR-X generates SAR images rich in information that aow innovative appications. Second, the wide area coverage of Sentine-1 images reduces the need for mutipe acquisitions and decreases the demand for high-cost data. Third, it is much easier for researchers to access Sentine-1 images than TerraSAR-X images because the Sentine-1 images are freey avaiabe, whie the TerraSAR-X images are usuay commercia. To meet these requirements for high-quaity data, we propose a Diaectica GAN method based on the anaysis of the hierarchica SAR information and the diaectica structure of GAN frameworks. The data used for vaidation is covering urban areas, so we can appy a spatia matrix to extract geometrica arrangement information. By using a GAN, we were abe to achieve good resuts with fine visua effects and our indicators show that our proposed method is better than the existing traditiona methods discussed in this paper. This paper is organized as foows. Section 2 presents the data set and the characteristics of both sateites (Sentine-1 and TerraSAR-X). In Sections 3 and 4, we deepy expain the deep earning methods for SAR image transation, incuding the deveopment of traditiona methods and the creation of the proposed method. Section 5 describes the experiments based on an urban area using the traditiona and proposed methods, whie Section 6 discusses the advantages of the proposed method compared with the traditiona methods. Finay, Section 7 concudes this paper and gives future research perspectives. 2. Data set In the fied of radar remote sensing, they are many sateites for different appications [16]. In this paper, we chose two typica sateite systems, Sentine-1 and TerraSAR-X, which serve the same purpose but with different characteristics. Sentine-1 is a C-band SAR sateite system aunched by ESA, whose missions incude sea and and monitoring, emergency response after environmenta disasters, and commercia appications [17]. In contrast, TerraSAR-X is an X-band Earth observation SAR sateite being operated under a pubic-private-partnership between the German Aerospace Center (DLR) and EADS Astrium (now Airbus), whose main features are its high resoution with exceent geometrica accuracy [18]. In our opinion, Sentine-1 is a good option to generate arge-scae SAR images, whie TerraSAR-X is an

3 3 of 22 adept soution for high resoution. To avoid being infuenced by radar configurations, we try to keep the radar system parameters of two products as consistent as possibe. A comparison of the radar parameters of two image products we used in this paper is shown in Tabe 1. Tabe 1. Seected data set parameters SAR instrument TerraSAR-X Sentine-1A Carrier frequency band X-band C-band Product eve Leve 1b Leve 1 Instrument mode High Resoution Spotight Interferometric Wide Swath Poarization VV VV Orbit branch Descending Ascending Incidence ange Product type Enhanced Eipsoid Corrected (EEC) (ampitude data) Ground Range Detected High Resoution (GRDH) (ampitude data) Enhancement Radiometricay enhanced Muti-ooked Ground range resoution 2.9 m 20 m Pixe spacing 1.25 m 10 m Equivaent number of ooks (range azimuth) = = 5 Map projection WGS-84 WGS-84 Acquisition date Origina fu image size (cos rows) ,255 18,893 Used image sizes (cos rows) Image quantization The ampitude of SAR image products is usuay not in the range of [0, 255] which is the dynamic range where optica image products stay. The ampitude of SAR images reates with the radar cross section (RCS) and has a arge dynamic range. There are many methods for SAR image quantization [19]. Because we need to use pre-trained neura networks designed for optica images, the SAR data shoud be scaed to the brightness range of optica pixes. In order to generate the SAR images with good visua effects, an 8-bit uniform quantization is appied in different brightness range. For Sentine-1 images, the range is [0, 800] whie for TerraSAR-X images it is [0, 570]. These parameters were defined by the brightness eves of our test data which contain 98% of the pixes in the pixe brightness histograms.

4 4 of Image co-registration The image transation between two different products shoud be done with co-registered image pairs. Fortunatey, remote sensing products can be projected the same coordinates by using geo-coding. Geo-coding is a technique that yieds every pixe its ongitude and atitude on Earth. Thus, for each pixe, once its ocation is determined, the pixe information from both Sentine-1 and TerraSAR-X images can be retrieved. In order that the two images have the same content and the same pixe size, the pixe spacing for both images is set to the same vaue, where the scae is 1:10. Finay, the interpoation and the co-registration are competed automaticay in the QGIS software, which is an open source too. In this software, the interpoation is based on IDW (Inverse Distance Weighted) method [20], and the co-registration reies on the annotation data of the image product resuting the accuracy of a few meters. 2.3 Training data and test data The seection of a training data set and a test data set for quaity contro is a primary task in deep earning. There are severa hyper-parameters to be determined and they can finay impact the capabiities of the trained networks. The seected patch size is one of the hyper-parameters that can affect both the fina resuts and the amount of the training data. When the patch size is too arge, the number of the training data becomes sma, even the data augmentation can be appied. Based on the discoveries in [21], which yieded a best patch size for SAR image cassification, we chose for our studies a patch size of pixes [21]. Using an overap of 50% between the tied patches, we obtained 1860 patches for training and 224 patches for testing. 3. Reated work Deep earning has been widey used in the ast years in computer vision, bioogy, medica imaging, and remote sensing. Athough the theory of deep earning is not yet mature, its capabiities shown in numerous appications have attracted the attention of many researchers. Let us simpy review the deveopment of image transation with deep earning. In 2016, Gatys et a. demonstrated the power of Convoutiona Neura Networks (CNNs) in creating fantastic artistic imagery. With a good understanding of the pre-trained VGG networks, they have achieved the stye transfer and demonstrated that semantic exchange coud be made by using neura networks. Since then, Neura Stye Transfer has become a trending topic both in academic iterature and industria appications [22]. To acceerate the speed of Neura Stye Transfer, a ot of foow-up studies were conducted. A typica one is Texture networks. With the appearance of GANs, severa researchers turned to GANs to find more genera methods without defining the texture. In this paper, we examine three typica methods, the method of Gatys et a. [9], Texture Networks [10] and Conditiona GANs [13]. By anayzing their advantages and disadvantages in SAR image transations, we propose a new GAN-based framework which is the combination of the manifestations of SAR images in the VGG-19 network, the definition of texture content, and the WGAN method VGG-19 network VGG-19 is a key too to conduct stye transfers. It is a pre-trained CNN mode for arge-scae visua recognition deveoped by Visua Geometry Group at the University of Oxford, which has achieved exceent performances in the ImageNet chaenge. Gatys et a. [9] firsty introduced this CNN in their work. Then, the next studies were focused on the utiization of the outcomes of VGG-19. However, VGG-19 has been trained on the ImageNet dataset which is the coection of optica images. In order to find the capabiities of VGG-19 for SAR images, we first visuaize the content of each ayer in VGG-19 when the input is a SAR image and then anayze the meaning of each ayer. The input SAR images are in the 8-bit dynamic range without histogram changes for fitting the optica type. There are 19 ayers in the VGG-19 network, but the most commony used

5 5 of 22 ayers are the ayers after down-samping, which are caed ReLU1_1, ReLU2_1, ReLU3_1, ReLU4_1, and ReLU5_1. A visuaization of SAR images via the VGG-19 ayers is shown in Figure 1. Origina ReLU1_1 ReLU2_1 ReLU3_1 ReLU4_1 ReLU5_1 Sentine-1 TerraSAR-X Figure 1. Visuaization of Sentine-1 and TerraSAR-X SAR images in the VGG-19 ayers As can be seen from Figure 1, the images in ReLU 1_1, ReLU 2_1, and ReLU 3_1 ayers are quite different, whie the images in ReLU 4_1 and ReLU5_1 of both two sensors are simiar. According to the conception of deep earning, the higher ayers contain higher semantic information [9], which supports the resuts in Figure 1. Therefore, Gatys et a. used the shaow (i.e., ower) ayers as the components of texture and took the deep ayers as the content information. However, we find that the ReLU5_1 images in both Sentine-1 and TerraSAR-X are amost featureess. In another paper [23], the authors found that ReLU5_1 has rea content for optica images. This may be because this training of VGG-19 is based on optica images. Whatever, we decide to ignore the ReLU5_1 ayer in our agorithm in order to acceerate the computation. It wi be discussed in the experiment part Texture definion-gram matrix The success of Gatys paper is to some extent achieved by the introduction of a Gram matrix. If we regard the pixes of the feature map in each ayer as a set of random variabes, the Gram matrix is a kind of second-order moment. The Gram matrix in that paper is computed on the seected ayers as described in Section 3.1. Assuming L ayers are seected and their corresponding number of feature maps is N, the Gram matrix of the th ayer is G = 1 M [ F T 1: F T 2: [F 1: T ] F N : F 2: F N :], (1) where F i: is the coumn vector generated from the i th feature map of ayer, and M is the size number of each feature map in this ayer. An eement of the N N Gram matrix is G ij M = 1 F M ik F jk k=1 = 1 F M i:, F j:, (2) where denotes the inner product. When we get the Gram matrices {G } Lseected, where L seected is the set of the seected ayers to define the texture information. Having the Gram matrices, the definition of the stye difference between two images is L stye = w G G F 2 L seected, (3) where w is a kind of hyper-parameter define the weight of the stye in the th ayer, G is the Gram matrix of the being generated image in the th ayer, G is the corresponding term for the reference image, and F is the Frobenius norm of the matrices. In our case, the stye image is no

6 6 of 22 onger an artistic painting of art, and the Gram matrices did not perform we. Figure 2 shows the mismatch of utiizing these Gram matrices to transate between SAR images. (a) (b) (c) Figure 2. Experiment using the Gatys et a. method (a) content image (Sentine-1) (b) transferred image (Gram matrix) (c) stye image (TerraSAR-X) Figure 2(b) contains many fake targets. For exampe, there is nothing at the ower right part of both Figure 2(a) and Figure 2(c), but some bright ines, usuay from buidings, appear at that part of Figure 2(b). Besides, contrary to Figure 2(c), the ayout of buidings in Figure 2(b) is hard to understand. In our experiment, the SAR data are depicting an urban area, where most targets are buidings. The city structure is quite different from the design of artistic works, which means the stye definition shoud vary for different appications. Refecting upon the Gram matrices, their format shoud be changed. The vectorization of the feature maps makes the Gram matrices fuy bind to the arrangement information inside the maps [24]. To maintain the arrangement information, which is usefu for urban area, we shoud discuss this arrangement information and how to make it suitabe for our appications. The arrangement most often indicates the pacing of items according to a pan, but without necessariy modifying the items themseves. Thus, an image with arrangement information shoud contain simiar items and the simiar items are paced in different ocations. When we tie the images into sma pieces (caed patches) according to the scheme they beong to, the sma pieces shoud be simiar. Their simiarity can be determined by the Gram matrix, whie the way to tie the image is the part of our approach. The manifestation of most objects of urban areas in remote sensing images is usuay rectanguar. Thus, the main outine of urban SAR images shoud be straight ines. The Spatia Gram method is a good way to represent this arrangement texture, which defined by the sef-simiarity matrices themseves and by appying spatia transformations when generating these matrices. A Gram matrix is a measurement of the reationship of two matrices, and the spatia transformation determines which two. G. Berger et a. have proposed a series of CNN-based Spatia Gram matrices to define the texture information. Based on their ideas in [24], we appy a spatia transform tiing the feature map horizontay and verticay in different eves to represent the straight texture information. As we have severa options to tie an image, how to compute their Gram matrices to define the texture is sti a question, either to add them or to regard them as parae structures. When the Spatia Gram computation just has one eement, it degenerates into the traditiona Gram matrix ike the one used by Gatys et a. But when it has too many eements, the utimate configuration is that a the pixes are in the Gram matrix individuay and it wi ose its capabiity to generate diverse textures. A ine, which is the basic unit of our images, can be determined by two parameters. Thus, we use the two orthogona dimensions (row and co), as two rows of the Spatia Gram matrix, and the spatia transform types as the coumns. Thus, the Spatia Gram matrix we appied in this paper is S spatia = ( G row,2 G co,2 G row,4 G row,2 7 G co,4 G co,2 7 ), (4) where the type of transformation is reated to the size of the feature maps in this ayer. Δ = {2,, 2 7 } where the 7 is determined by the input size of patches ( ), and L seeted = {1, 2, 3}. G row,δ and G co,δ are two kinds of spatia transformation which is reated to the dimensions row

7 7 of 22 and co, and the shifted amount δ. Assuming the feature map is F, and its transformations are T(F ) where T denotes the function of spatia transformation. For exampe, the spatia transformations of feature maps in the row dimension are defined as T row,δ (F ) = F (δ: M, 1: N), T row, δ (F ) = F (1: M δ, 1: N), (5) where M, N are the height and width of the feature map F. T row,δ (F ) is the transformation on the row dimension. The vectorization of T row,δ (F ) is written as T row,δ (F ) : which is the coumn vector Having these definitions, G row,δ can be written as G row,δ = 1 M T row,δ (F 1 ) : T T row,δ (F T 2 ) : [T row, δ(f 1 ) : T row, δ (F 2 ) : T row, δ (F N ) : ], (6) T [ T row,δ (F N ) : ] where G row,δ can be written in the same way but the spatia transformation takes paces in the row direction. Thus, the spatia stye oss function is L stye = w S spatia 2 S spatia. F L seected (7) where the S spatia if the spatia matrices of the target images and S spatia is for the generated image. The stye oss function L stye is ony dominated by the Spatia Gram matrices, it is not necessary to add the traditiona Gram matrices because when δ is sma, it is amost the same as the traditiona one. Figure 3 shows the resuts appying the new Spatia Gram matrix. (a) (b) (c) (d) Figure 3. Experiment using Spatia Gram matrices (a) content image (Sentine-1) (b) transferred image (Spatia Gram matrix) (c) transferred image (Gatys et a. s Gram matrix) (d) stye image (TerraSAR-X) 3.3 Conditiona generative adversaria networks The introduction of GANs is a miestone in deep earning, and it becomes popuar where hundreds of papers were pubished under the name of GAN [25]. A conditiona GAN makes a genera GAN more usefu because the inputs are no onger the noise but the things we can contro. In our case, the conditiona inputs are Sentine-1 images. The conditiona GANs have achieved impressive resuts on many image processing tasks, such as stye transfer [26], supper-resoution [27], or other tasks [28], [29]. Isoa et a. [13] summarized the tasks of image transation as pix2pix transations and demonstrated the capabiities of conditiona GANs in their paper. Inspired by their works, we modified the pix2pix framework by adding new discoveries about GANs and specific features of the SAR images transations. When we used the pix2pix framework in our appication this faied. Figure 4 shows the overfitting of the pix2pix conditiona GAN because the training set has good performances whie the test set has bad resuts. Without any modification, we coud not reach our goas. In the next section, we propose a new method to reaize Sentine-1 to TerraSAR-X image transations.

8 8 of 22 Input Output Target Training set (a) (b) (c) Test set (d) (e) (f) Figure 4. SAR image transation using the pix2pix framework in both training and test set (a) input image in the training set (b) GAN output of image (a) (c) target of image (a) (d) input image in the test set (e) GAN output of image (d) (f) target of image (d) 4. Method Athough the conditiona GAN is overfitting in our case, it is sti a good strategy to compete our task, which is to have a mapping function from Sentine-1 to TerraSAR-X. In mathematica notation, it is G: x y, (8) where G is the mapping function, x is a Sentine-1 image, and y is a TerraSAR-X image. Actuay, this task can be achieved by designing a neura network and by presetting a oss function ike traditiona machine earning. Indeed, this idea has aready been accompished in [10] and [11]. However, the preset oss function is not genera for a cases. A GAN provided an idea that the oss function is not preset, and it can be trained through a network which is caed Discriminator. The mapping function G is reaized through a Generator neura network. In this paper, we use the concept of diaectics to unify the GANs and traditiona neura networks. There is a triad in the system of diaectics, thesis, antithesis and synthesis, and they are regarded as a formua for the expanation of change. The formua is summarized as (1) a beginning proposition caed a thesis, (2) a negation of that thesis caed the antithesis, and (3) a synthesis whereby the two conficting ideas are reconcied to form a new proposition [30]. We appy this formua to describe the change of image transation. The Generator network is regarded as thesis and it can be inherit the parameters from the previous thesis. In our case, the Generator inherits from the texture network. The Discriminator network acts as a negation of the Generator. The synthesis is based on the aw of the Negation of the Negation. Thus, we can generate a new Generator through the diaectica method. When the new data comes, it wi enter the next state of changing and deveopment. The goba fowchart of our method is shown in Figure 5. There are two phrases, training phrase and operationa phrase. The training phrase is the processing to generate a fina generator, and the operationa phrase appies the fina generator to conduct the image transation task. In the foowing, we discuss the Generator network, the Discriminator network and the detais to train them.

9 9 of 22 Visuaization resuts for inspection... Test data Inputs Training data Inputs Texture network Generator Thesis Generator 1 Diaectica generative adversaria networks Generative adversaria networks... Thesis Thesis Thesis Fina thesis Generator 2 Generator n Fina Generator Training data Targets Antithesis Spatia Gram Matrices Synthesis Antithesis Synthesis Antithesis Synthesis Antithesis Synthesis Discriminator 1 Discriminator 2... Discriminator n Outputs Training phase Operationa phase Figure 5 Goba fowchart of Diaectica GAN 4.1 Genertor network thesis The purpose of the generator is to generate an image G(x) has the content of image x and the stye of image y. Thus, the oss function has two parts, content oss and texture oss, which is defined as L Generator = L content + λl stye = F (G(x)) F 2 2 (x) F + λ w S spatia (G(x)) S spatia (y), (9) F L content L stye where λ is a reguarization parameter, F ( ) are the feature maps of the th ayer of an image, S spatia ( ) are the Spatia Gram matrices that were defined in Section 3.2. According to the discussion in Section 3.1, there is no information in ReLU5_1. Therefore, we chose ReLU4_1 as the content ayer, and ReLU1_1, ReLU2_1 and ReLU3_1 as the stye ayers. Consequenty, L content = {4}, and L stye = {1, 2, 3}. G can be any kind of functions, it can be as simpe as a inear function or as compex as a mutipe composition of non-inear functions. As a powerfu too to approximate functions [31] [32], deep neura networks are used as our notation of G in this paper. The input and the target images, x and y, are from different SAR sensors, but they are observing the same test site. The properties of SAR systems resut in their own characteristics of image representation, such as fina resoution, poarization response, and the dynamic ranges. But the same observed area makes them share identica compounds. Regardess of the changes in time, x and y are generated from identica objects. For the anaysis of our input and target images, there are penty of network structures that sove this probem. Previous reated works [28] [33] have used an encoder-decoder network [34] where the input image is compressed in down-samped ayers and then be expanded again in up-samped ayers where the process is reversed. The main probem of this structure is whether the information is preserved in the down-samped ayers. Based on the discussion in [13], we chose the U-Net network to execute our tasks. The U-Net is very we known for its skip connections which are a way to protect the information from oss during transport in neura networks. According to the behavior of our SAR images in the VGG-19 network, we set the U-Net to 6 ayers. The structure of the network we used is shown in Figure 5.

10 10 of Generator network Figure 6. Architecture of the U-Net Generator network Athough the network in Figure 5 has too many eements and is hard to be trained, we think it is necessary to use a deep network because the architecture of a network can affect its expressiveness of compex functions. Maybe there wi be more efficient methods to approximate the mapping function, but this is not the topic of this paper. Our goa is a powerfu too to describe the mapping from Sentine-1 to TerraSAR-X where the soution is a deep neura network. 4.2 Discriminator network antithesis A deep neura network is a suitabe soution, but on the other hand, it can aso easiy generate non-target resuts. Based on the concept of diaectics, when the appearance is not fit for the conception, it is needed to deny the existence of this thing. In this case, it is the negation of the generated images. In other words, we need a oss function yieding a sma vaue when the output equas the target whie yieding a high vaue when the two are different. Usuay, the oss function is predefined. For exampe, the most common oss function, Mean Squared Error (MSE), is a preinstaed function which is defined as MSE = 1 N N (Yi Y i ) 2 i=1, (10) where Y is the generated vector of Y whose eements arey i. When computing the MSE function, it outputs a scaar vaue to describe the simiarity of the input and the target. But it is predefined, and the ony freedom are the input data. How it reates to the negation of the generated images is sti a question. There are three steps to sove the probem. First, the oss function shoud criticize the existence of Y, so it has a term Y. Second, it shoud approve the subsistence of Y, the target; thus, the term Y sha appear. Third, the square operator makes sure the function is a kind of distance. Through these three steps, the MSE has accompished the negation of the generated vectors or images. When the generated image differs from the target image, the distance is arge. When the generated image is the target image itsef, their distance sha be zero. In contrast, a arge distance sha be generated when the input is markedy different from the target to ead to better negation. It is reasonabe to expect that the oss function is a kind of distance function because the distance space is a weak assumption for the space of generated images. For instance, the oss function in (9) is another kind of distance compared with the MSE that directy computes pixe vaues. However, it is hard to find a unique common distance because our tasks differ whie the distance remains invariant. Using a neura network scheme to train a distance is a good choice. Fortunatey, the appearance of GANs has supported us soutions to find the proper distances. In GAN systems, the negation of generated images is processed in the oss function of the Discriminator. The discriminator is a mapping function, or a neura network to describe the

11 11 of 22 existence of the input image. However, the properties of the discriminator have been itte discussed. In this paper, we try to use the theory of metric spaces to discuss this question. Assuming that the distance in the image domain M 1 is d 1 ( ) and the distance in the discriminator domain M 2 is d 2 ( ), the discriminator is the map D: M 1 M 1 [35]. The distance of the conditiona case, which is aso the contradiction between two images, can be defined as L contradiction = d 2 (D(y x), D(G(x) x)), (11) where D( x) is the discriminator of an image under the condition that the input is x. If D( ) is a map to map the image to itsef, and d 2 ( ) is the Frobenius norm, the contradiction becomes L contradiction = y G(x) F, (12) which is the L 1 norm that usuay acts as a oss function in machine earning. This is one case of a determined map. As for a training map function, the most important thing is to design its format. If we sti set d 2 ( ) as the Frobenius norm, the distance of the discriminator becomes L contradiction = D(y x) D(G(x) x) F, (13) when the discriminator is a predefined network such as the Spatia Gram matrix, we concude that the oss function in (9) can be regarded as a specific case of (13). If the range of d 2 ( ) is [0,1], it is considered that the output is the possibiity of being rea. There are many concepts to re-unite the formats of different oss functions. In f-gan [36], the oss functions are regarded as f-divergences, which are the measurements for the simiarity of two distributions. However, the drawback of divergences is that they don t satisfy the triange inequaity and the symmetry which are requirements of distance functions [37]. In LSGAN [38], the east squares method is used to measure the output of the discriminator. In this method, the generated images are in an inner product space which is aso a metric space. Therefore, we infer that the contradiction of the rea image and the generated image shoud be contained in a function that can define the distance of some metric space, and the map D shoud be constrained. One constraint of D is that the range of D shoud be bounded because we compute it in a computer. Or it wi become an infinite number. Second, D shoud be continuous, even uniformy continuous, because the gradient descent agorithms may fai when the oss function is not continuous. In WGAN, the Wasserstein distance is used, where the Lipschitz-continuous map ensures the property of uniformy continuous. In this paper, we focus on the WGAN framework. When d 2 ( ) is the Wasserstein distance [39], the oss function of the discriminator becomes L discriminator = W(D(y x), D(G(x) x)), (14) where W( ) is the Wasserstein distance function which behaves better than the f divergence being used in traditiona GANs. The reaization of the Wasserstein distance enforces a Lipschitz constraint on the Discriminator. In the WGAN-GP framework [40], the Lipschitz constraint is reaized by enforcing a soft version of the constraint with a penaty on the gradient norm for random sampes x ~ P x, where x = εy + (1 ε)g(x). Based on the concusions in WGAN [40], the maximum of the Wasserstein distance between P r,y x and P g,x becomes D = max D (L discriminator ) = min D ( E G(x)~P g,x,x~p r,x [D(G(x) x)] E y~pr,y,x~p r,x, [D(y x)] +λ gp E x ~ Px [( x D(x x) 2 1) 2 ), (15) ] where D is the best discriminator, P r,y x is the distribution of given rea images, P g,x is the distribution of generated images and x D(x x) is the gradient of the discriminator D( ). When adding the penaty of the distance between the norma of x D(x x) and 1 in the oss function, the Discriminator is forced to become a 1 Lipschitz function. λ gp is usuay set to 10 according to the experiments conducted in [40]. Intuitivey, the remova of the absoute operator ensures the continuity of the derivation of the oss function at the origin. The 1 Lipschitz constraint imits the norma of the derivation from growing too arge, which is a way to increase the distance but not in the way we want.

12 12 of 22 Once the oss function is determined, the next step is to design the architecture of D( x) that can be easiy trained for computers. Considering the ready-made function aready discussed in the previous section, the oss function of stye defined by Gram matrices is a good choice because it can be regarded as processing on a Markov random fied [13] [26]. The pix2pix summarized it as the PatchGAN whose input is the combination of x and y. The architecture of the discriminator is shown in Figure 7. Discriminator network Output Figure 7. Architecture of PatchGAN Discriminator network 4.3 Diaectica Generative adversaria network synthesis According to the diaectic, the third step is the negation of the negation. The negation of the generated image is described by the oss function of the discriminator. Thus, the negation of negation shoud be the negation of the oss function of the discriminator. The negation is trying to make the distance defined by the discriminator to become arger, whie the negation of negation shoud make the distance smaer. In our WGAN framework, the negation is defined by equation (15). Thus, the negation of negation can be reaized by maximizing it. Therefore, the maximization of the oss function in (15) is the negation of negation. At the ast step of the diaectic, the negation of negation shoud be combined with the thesis to form a synthesis. The thesis can be regarded as a synthesis from the former diaectics. For exampe, the pix2pix used the L 1 norm as their thesis, and the SRGAN used the Gram matrices on ayer 5 of the VGG-19 network as their thesis. These initia oss functions are distance functions and contain the negation of the generated images. In this paper, we start from the thesis defined by a Spatia Gram matrix. In other words, we set the initia oss function as defined in (9). The negation of negation is the maximization of (15). Therefore, the synthesis of our Diaectica GAN is the combination of (9) and (15). Reducing the terms in (15) that independent of Generator networks, the oss function of the Diaectica GAN becomes GAN L Generator = L Generator λ GAN L critica = L content + λl stye λ GAN E G(x)~P g,x,x~p r,x [D(G(x) x)] (16) To optimize this new oss function, we need four steps: set up the generator, update the discriminator, update the generator and iterate. Step 1, having a Generator G( ) and an input image x, use the to generate G(x), and then run the Discriminator D( ). Step 2, use gradient descent methods to update D( ) foowing (16) Step 3, use gradient descent methods to update G( ) foowing (15). Step 4, repeat Step 1 and Step 3 unti the stopping condition is met. Then the training of the Diaectica GAN is competed. Every oop can be considered as a reaization of the diaectics. The basic framework is based on the WGAN-GP. As for the mathematica anaysis of the GANs and deep earnings, pease refer to [41], [42], [43]. Athough the

13 13 of 22 Deep Learning sti ooks ike a back box, we tried to provide a ogica anaysis of it and attempted to achieve rea artificia inteigence with the capabiities of diaectics. 5. Experiments The data used for demonstration has aready been described in Section 2. Based on the method proposed in Section 4, the GAN network used in this paper has two neura networks, Generator and Discriminator. The Generator is a U-Net with 6 ayers, and the Discriminator is a PatchGAN convoutiona neura network with 4 ayers. In tota, we had 1860 image pair-patches in the training data set and 224 image pair-patches in the test data set. With these data sets, the training took two days on a aptop with Inte Xeon CPU E3, an NVidia Q2000M GPU and 64 GB of memory. We conducted three experiments with respect to the foowing networks further presented beow SAR images in VGG-19 networks VGG-19 has an essentia roe in this paper because its ayers are the components of the texture information determined by a Gram matrix. Besides, the seection of the content ayer is a new probem for SAR images. First, we compared the differences between Sentine-1 and TerraSAR-X images in each ayer. Two image patch-pairs are the inputs in the VGG-19 networks and their intermediate resuts are shown in Figure 8. Origina ReLU1_1 ReLU2_1 ReLU3_1 ReLU4_1 ReLU5_1 Image pair 1 Image pair 2 Figure 8. Two image patch-pairs input to in the VGG-19 networks and their intermediate resuts Visuay, the images of the ReLU4_1 ayer have common parts. But this is not sufficient, and we decided to introduce the MSE and the Structura Simiarity Index (SSIM) [44] in order to compare the image in different ayers. The MSE is defined as: N 1 M 1 M 1 MSE 1 = (M ) 2 N [x k (i, j) y k (i, j)] 2, (17) k=0 i=0 where M is the size of the feature maps in th ayer, N is the number of the feature maps in th ayer, x k (i, j) is the pixe vaue of (i, j) in the k th feature map of the th ayer of a Sentine-1 image, and y k (i, j) is the counterpart of a TerraSAR-X image. In order to overcome the drawbacks of the MSE, we appied the SSIM whose definition is j=0

14 14 of 22 SSIM(x, y) = (2μ xμ y + c 1 )(2σ xy + c 2 ) (μ x 2 + μ y 2 + c 1 )(σ x 2 +σ y 2 + c 2 ), (18) where μ x and σ x are the mean vaue and the standard deviation of image x; the same to appies to y. c 1 and c 2 are two constants reated with the dynamic range of the pixe vaues. For more detais, we refer the reader to [44]. The SSIM vaues range between -1 and 1, where 1 indicates perfect simiarity. The evauation resuts with the two indicators are shown in Tabe 2. Tabe 2. Evauation resuts with MSE and SSIM Layers MSE SSIM ReLU1_ ReLU2_ ReLU3_ ReLU4_ ReLU5_ Athough ReLU5_1 has the best performance with two indicators, we sti ignore this ayer due to the poor diversity in this ayer. Excuding ReLU5_1, the ReLU4_1 ayer gives us the best resut. Therefore, the ReLU4_1 is chosen as the content ayer, and the first three ayers are used to define texture information Gram martrices vs. Sptatia Gram martrices A Spatia Gram matrix is an extension of a Gram matrix, which is used to describe the texture information and is good at representing arrangement information. In Section 3.2, we have shown the visua difference between two stye definitions. In this experiment, we used the quantity indicators to evauate the two methods. Two image patch-pairs were chosen to conduct the comparison, whose resuts are shown in Figure 9. In order to evauate the image quaity of the SAR images, we introduce the equivaent numbers of ooks (ENL), which act as a contrast factor to represent the image resoutions approximatey. A higher ENL vaue indicates that the image is smooth whie a ower vaue means that the image is in high resoution [45]. For our case, we need high-resoution images and as a resut, the ower their ENL vaue, the better. The definition of ENL is ENL = μ2 σ2, (19) where μ is the mean vaue of the image patch, and σ is its standard deviation. Content image (Sentine-1) Transferred image (Spatia Gram matrices) Transferred image (Gatys et a. Gram matrices) Stye image (TerraSAR-X) Image pair 1 Image pair 2 Figure 9. Comparison between a Spatia Gram matrix and a Gatys et a. Gram matrix in two patch-pairs

15 15 of 22 Tabe 3. Evauation of two methods for both image pairs 1 and 2 Image pairs Methods MSE SSIM ENL Gatys et a. Gram Spatia Gram Gatys et a. Gram Spatia Gram As can be seen from Figure 9 and Tabe 3, the Spatia Gram method performs better than Gatys et a. s method, both visuay and according to evauation indicators. However, the ENL of image pair 1 indicates that Gatys et a. s method is better. To sove this probem, we need more experiments. Because the traditiona generative mode regards every pixe as a random pixe and ignores the reationships among neighboring pixes, its computing efficiency is imited. Nevertheess, a Spatia Gram matrix is a good too to determine the image stye for our cases. In the next subsection, we abandon the Gatys et a. s method and repaced it with a U-net network to generate the enhanced images. This method is caed Texture network Spatia Gram matrices vs. traditiona GANs The texture network moves the computationa burden to a earning stage and no onger needs the stye images as an aide to produce an image because the stye information is aready mapped in the network through the earning steps. Athough the feed-forward network supersedes the soution of random matrices, the oss function is sti the same. According to the above experiments, the Spatia Gram matrix is the winner of the determinate oss function. In contrast to the determinate one, other researchers found that the oss function can aso be earned, though the Spatia Gram matrix is aso earned from the VGG-19 network. Nonetheess, the earning of the oss function enabes the definition of image stye to become more optiona. We use the WGAN-GP framework to represent this kind of idea, which is the most stabe one among the GAN famiy. The resuts of the texture network and the WGAN-GP are compared in Figure 10 and the evauation resuts are isted in Tabe 4. The test set components in Tabe 4 are the average performances of images in whoe test set. The texture network and the WGAN-GP are fast ways to conduct stye transfer. According to the vaues in Tabe 4, we concude that the WGAN-GP has a better performance than the texture network method with the given indicators. However, the WGAN-GP is not abe to preserve the content information of Sentine-1 and its output images are mudded without obvious structures ike the texture network. Athough texture network has no good performance in the evauation system, it has preferabe visua effect in contrast to the WGAN-GP. How to baance the indicator vaues and the visua performance is a crucia probem. The texture information is defined by the VGG-19 network which has been trained by optica images. Thus, we have grounds to beieve that there is texture information that cannot be described by Spatia Gram matrices. In a foowing experiment, we wi compare the texture network with the proposed Diaectica GAN.

16 16 of 22 Input image (Sentine-1) Texture network WGAN-GP Target image (TerraSAR-X) Image pair 1 Image pair 2 Figure 10. Comparison between Texture network and WGAN-GP for two patch-pairs Tabe 4. Evauation of Texture network and WGAN-GP in both image pair 1 and 2 Image pairs Methods MSE SSIM ENL Texture network WGAN-GP Test set Texture network WGAN-GP Texture network WGAN-GP Diaectica GAN vs. Spatia Gram matrices The texture network defined the texture information in a determinate way whie the WGAN-GP uses a fexibe method to describe the difference between generative images and target images. In this paper, we proposed a new method that combines a determinate way and a fexibe way to enhance the generative images, and we caed it Diaectica-GAN because the idea is enightened by the diaectica ogic. The Diaectica-AN initiaizes its oss function with the Spatia Gram matrix that was found a good way to describe the texture information of urban area and the content oss defined by the ReLU4_1 ayer of the VGG-19 network. Through the training of the Diaectica GAN, new texture information can be earned and represented in the Discriminator network. The comparison between a Diaectica-GAN and the texture network with a Spatia Gram oss function are shown in Figure 11 and Tabe 5. Both visua performance (Figure 11) and the indicator anaysis (Tabe 5) proved that our method is better than the texture network. However, these experiments a remained imited to the patch eve, and the figures of a whoe scene have not yet been considered. Therefore, we show the entire image composited with every path to check the overa performance and to estimate the reationship between neighboring patches.

17 17 of 22 Input image (Sentine-1) Our method Texture network Target image (TerraSAR-X) Image pair 1 Image pair 2 Figure 11. Comparison between Diaectica GAN and Texture network for two image patch-pairs Tabe 5. Evauation of Texture network and Diaectica GAN for both image pairs 1 and 2 Image pairs Methods MSE SSIM ENL Texture network Diaectica GAN Test set Texture network Diaectica GAN Texture network Diaectica GAN Overa visua perfomance One of the most important merits of remote sensing images are their arge-scae observations. In this section, we are discussing how a remote sensing image ooks when its patches are processed by the seected neura networks. A fu image is generated by concatenating the sma processed patches to produce a fina image. In this paper, we focus on three networks, the texture network with a Spatia Gram matrix, the WGAN-GP method, and our Diaectica GAN method. They are shown in Figure 12, Figure 13, and Figure 14, respectivey. As for the overa visua performance, we consider that the Diaectica GAN has the best subjective visua performances. Figure 12. The overa resuts of a Diaectica-GAN

18 18 of 22 Figure 13. The overa resuts of a texture network Figure 14. The overa resuts of a WGAN-GP (L1 +WGAN-GP) The SAR image transation resuts compared with inputs and outputs image are shown in Figure 15. First, we can see the entire effect of the image transation.in the Munich urban area. To dispay detai resuts, we have three bounding box with different coors (Red, Green and Yeow) to extract the patches from the fu image. They are in Figure 15(d).

19 1 19 of 22 (a)sentine-1 image of Munich, Germany (b)terrasar-x image of Munich, Germany (c) Diaectica GAN image of Munich, Germany (d) Zoom in resuts Sentine-1 Diaectica GAN TerraSAR-X Figure 15 Overa visua performance of Diaectica GAN compared with Sentine-1 and TerraSAR-X images (a) Sentine-1 image (b) TerraSAR-X image (c) Diaectica GAN image (d) Zoom in resuts 6. Discussion Compared with traditiona image enhancement methods, deep earning is an end-to-end method that is quite easy to be impemented. Deep earning has exceent performances and is standing out among the machine earning agorithms, especiay in the case of big data. Soutions for remote sensing appications were discovered by the advent of deep earning. More importanty, deep earning is now paying a crucia part in transferring the stye of images. Concerning SAR image transation, itte attention has been focused on it and the performances of deep earning on this topic are sti unknown. The task that this paper addresses is reated with super-resoution tasks, but our image pairs are not of the same appearances due to the differences in incidence anges, radar poarization, and acquisition times. From this aspect, our task beongs to stye transfer to some extent, ike generating a piece of artistic painting without the constraint that two images shoud be focused on same objects. Therefore, the SAR image transation is a mix of super-resoution and stye transfer and has never been focused in the conception of deep earning. From Gatys et a. s method to GAN frameworks, we have tested the capabiities of deep earning in transating Sentine-1 images to TerraSAR-X images. The resuting images of Gatys et a. s method are of high quaity but they don t preserve we the structure information, which is an essentia characteristic of remote sensing SAR images, especiay for urban areas. The improvement can be accompished by introducing Spatia Gram matrices instead of the traditiona ones in the oss function. A Spatia Gram matrix is a compromise between the arrangement structure and the freedom of stye. In this paper, we compose Gram matrices computed in spatia shifting mode as a new matrix-vector for each ayer. The spatia matrix is a good indicator to describe arrangement structures such as buidings and roads. However, our oss function modifications can ony sove the stye presentation probem, but the high computation effort sti imits the appications of image transation for remote sensing. Fortunatey, deep neura networks are a powerfu too for fitting compicated functions that provides soutions to speed up the image transation. Instead of taking every pixe as a random variabe, a deep neura network regards an image as an input of the system, and the ony thing the deep earning can do is to approximate the mapping function. That is to say,