arxiv: v5 [cs.cv] 29 Sep 2018

Transcription

1 Bi-Real Net: Enhancing the Pefomance of 1-it CNNs With Impoved Repesentational Capaility and Advanced Taining Algoithm axiv: v5 [cs.cv] 29 Sep 2018 Zechun Liu 1, Baoyuan Wu 2, Wenhan Luo 2, Xin Yang 3, Wei Liu 2, and Kwang-Ting Cheng 1 1 Hong Kong Univesity of Science and Technology 2 Tencent AI la 3 Huazhong Univesity of Science and Technology zliuq@connect.ust.hk, {wuaoyuan1987, whluo.china}@gmail.com, xinyang2014@hust.edu.cn, wliu@ee.columia.edu, timcheng@ust.hk Astact. In this wok, we study the 1-it convolutional neual netwoks (CNNs), of which oth the weights and activations ae inay. While eing efficient, the classification accuacy of the cuent 1-it CNNs is much wose compaed to thei countepat eal-valued CNN models on the lage-scale dataset, like ImageNet. To minimize the pefomance gap etween the 1-it and eal-valued CNN models, we popose a novel model, dued Bi-Real net, which connects the eal activations (afte the 1-it convolution and/o BatchNom laye, efoe the sign function) to activations of the consecutive lock, though an identity shotcut. Consequently, compaed to the standad 1-it CNN, the epesentational capaility of the Bi-Real net is significantly enhanced and the additional cost on computation is negligile. Moeove, we develop a specific taining algoithm including thee technical novelties fo 1- it CNNs. Fistly, we deive a tight appoximation to the deivative of the non-diffeentiale sign function with espect to activation. Secondly, we popose a magnitude-awae gadient with espect to the weight fo updating the weight paametes. Thidly, we pe-tain the eal-valued CNN model with a clip function, athe than the ReLU function, to ette initialize the Bi-Real net. Expeiments on ImageNet show that the Bi-Real net with the poposed taining algoithm achieves 56.4% and 62.2% top-1 accuacy with 18 layes and 34 layes, espectively. Compaed to the state-of-the-ats (e.g., XNOR Net), Bi-Real net achieves up to 10% highe top-1 accuacy with moe memoy saving and lowe computational cost. 4 1 Intoduction Deep Convolutional Neual Netwoks (CNNs) have achieved sustantial advances in a wide ange of vision tasks, such as oject detection and ecognition Coesponding autho. 4 Code is availale on:

2 2 Zechun Liu et al. [12,23,25,5,3,20], depth peception [2,16], visual elation detection [29,30], face tacking and alignment [24,32,34,28,27], oject tacking [17], etc. Howeve, the supeio pefomance of CNNs usually equies poweful hadwae with aundant computing and memoy esouces. Fo example, high-end Gaphics Pocessing Units (GPUs). Meanwhile, thee ae gowing demands to un vision tasks, such as augmented eality and intelligent navigation, on moile hand-held devices and small dones. Most moile devices ae not equipped with a poweful GPU neithe an adequate amount of memoy to un and stoe the expensive CNN model. Consequently, the high demand fo computation and memoy ecomes the ottleneck of deploying the poweful CNNs on most moile devices. In geneal, thee ae thee majo appoaches to alleviate this limitation. The fist is to educe the nume of weights, such as Spase CNN [15]. The second is to quantize the weights (e.g., QNN [8] and DoReFa Net [33]). The thid is to quantize oth weights and activations, with the exteme case of oth weights and activations eing inay. In this wok, we study the exteme case of the thid appoach, i.e., the inay CNNs. It is also called 1-it CNNs, as each weight paamete and activation can e epesented y 1-it. As demonstated in [19], up to 32 memoy saving and 58 speedup on CPUs have een achieved fo a 1-it convolution laye, in which the computationally heavy matix multiplication opeations ecome light-weighted itwise XNOR opeations and it-count opeations. The cuent inaization method achieves compaale accuacy to eal-valued netwoks on small datasets (e.g., CIFAR-10 and MNIST). Howeve on the lage-scale datasets (e.g., ImageNet), the inaization method ased on AlexNet in [7] encountes sevee accuacy degadation, i.e., fom 56.6% to 27.9% [19]. It eveals that the capaility of conventional 1-it CNNs is not sufficient to cove geat divesity in lage-scale datasets like ImageNet. Anothe inay netwok called XNOR-Net [19] was poposed to enhance the pefomance of 1-it CNNs, y utilizing the asolute mean of weights and activations. The ojective of this study is to futhe impove 1-it CNNs, as we elieve its potential has not een fully exploed. One impotant osevation is that duing the infeence pocess, 1-it convolution laye geneates intege outputs, due to the it-count opeations. The intege outputs will ecome eal values if thee is a BatchNom [10] laye. But these eal-valued activations ae then inaized to 1 o +1 though the consecutive sign function, as shown in Fig. 1(a). Oviously, compaed to inay activations, these integes o eal activations contain moe infomation, which is lost in the conventional 1-it CNNs [7]. Inspied y this osevation, we popose to keep these eal activations via adding a simple yet effective shotcut, dued Bi-Real net. As shown in Fig. 1(), the shotcut connects the eal activations to an addition opeato with the eal-valued activations of the next lock. By doing so, the epesentational capaility of the poposed model is much highe than that of the oiginal 1-it CNNs, with only a negligile computational cost incued y the exta element-wise addition and without any additional memoy cost.

3 Bi-Real Net: Enhancing the Pefomance of 1-it CNNs 3 Fig. 1. Netwok with intemediate featue visualization, yellow lines denote value popagated inside the path eing eal while lue lines denote inay values. (a) 1-it CNN without shotcut () poposed Bi-Real net with shotcut popagating the eal-valued featues. Moeove, we futhe popose a novel taining algoithm fo 1-it CNNs including thee technical novelties: Appoximation to the deivative of the sign function with espect to activations. As the sign function inaizing the activation is non-diffeentiale, we popose to appoximate its deivative y a piecewise linea function in the ackwad pass, deived fom the piecewise polynomial function that is a second-ode appoximation of the sign function. In contast, the appoximated deivative using a step function (i.e., 1 x <1 ) poposed in [7] is deived fom the clip function (i.e., clip(-1,x,1)), which is also an appoximation to the sign function. We show that the piecewise polynomial function is a close appoximation to the sign function than the clip function. Hence, its deivative is moe effective than the deivative of the clip function. Magnitude-awae gadient with espect to weights. As the gadient of loss with espect to the inay weight is not lage enough to change the sign of the inay weight, the inay weight cannot e diectly updated using the standad gadient descent algoithm. In BinayNet [7], the eal-valued weight is fist updated using gadient descent, and the new inay weight is then otained though taking the sign of the updated eal weight. Howeve, we find that the gadient with espect to the eal weight is only elated to the sign of the cuent eal weight, while independent of its magnitude. To deive a moe effective gadient, we popose to use a magnitude-awae sign function duing taining, then the gadient with espect to the eal weight depends on oth the sign and the magnitude of the cuent eal weight. Afte convegence, the inay weight (i.e., -1 o +1) is otained though the sign function of the final eal weight fo infeence. Initialization. As a highly non-convex optimization polem, the taining of 1-it CNNs is likely to e sensitive to initialization. In [17], the 1-it CNN model is initialized using the eal-valued CNN model with the ReLU function pe-tained on ImageNet. We popose to eplace ReLU y the clip

4 4 Zechun Liu et al. function in pe-taining, as the activation of the clip function is close to the inay activation than that of ReLU. Expeiments on ImageNet show that the aove thee ideas ae useful to tain 1-it CNNs, including oth Bi-Real net and othe netwok stuctues. Specifically, thei espective contiutions to the impovements of top-1 accuacy ae up to 12%, 23% and 13% fo a 18-laye Bi-Real net. With the dedicatedlydesigned shotcut and the poposed optimization techniques, ou Bi-Real net, with only inay weights and activations inside each 1-it convolution laye, achieves 56.4% and 62.2% top-1 accuacy with 18-laye and 34-laye stuctues, espectively, with up to 16.0 memoy saving and 19.0 computational cost eduction compaed to the full-pecision CNN. Compaing to the state-of-theat model (e.g., XNOR-Net), Bi-Real net achieves 10% highe top-1 accuacy on the 18-laye netwok. 2 Related Wok Reducing the nume of paametes. Seveal methods have een poposed to compess neual netwoks y educing the nume of paametes and neual connections. Fo instance, He et al. [5] poposed a ottleneck stuctue which consists of thee convolution layes of filte size 1 1, 3 3 and 1 1 with a shotcut connection as a peliminay uilding lock to educe the nume of paametes and to speed up taining. In SqueezeNet [9], some 3 3 convolutions ae eplaced with 1 1 convolutions, esulting in a 50 eduction in the nume of paametes. FitNets [21] imitates the soft output of a lage teache netwok using a thin and deep student netwok, and in tun yields 10.4 fewe paametes and simila accuacy to a lage teache netwok on the CIFAR-10 dataset. In Spase CNN [15], a spase matix multiplication opeation is employed to zeo out moe than 90% of paametes to acceleate the leaning pocess. Motivated y the Spase CNN, Han et al. poposed Deep Compession [4] which employs connection puning, quantization with etaining and Huffman coding to educe the nume of neual connections, thus, in tun, educes the memoy usage. Paamete quantization. The pevious study [13] demonstated that ealvalued deep neual netwoks such as AlexNet [12], GoogLeNet [25] and VGG-16 [23] only encounte maginal accuacy degadation when quantizing 32-it paametes to 8-it. In Incemental Netwok Quantization, Zhou et al. [31] quantize the paamete incementally and show that it is even possile to futhe educe the weight pecision to 2-5 its with slightly highe accuacy than a fullpecision netwok on the ImageNet dataset. In BinayConnect [1], Couaiaux et al. employ 1-it pecision weights (1 and -1) while maintaining sufficiently high accuacy on the MNIST, CIFAR10 and SVHN datasets. Quantizing weights popely can achieve consideale memoy savings with little accuacy degadation. Howeve, acceleation via weight quantization is limited due to the eal-valued activations (i.e., the input to convolution layes). Seveal ecent studies have een conducted to exploe new netwok stuctues and/o taining techniques fo quantizing oth weights and activations

5 Bi-Real Net: Enhancing the Pefomance of 1-it CNNs 5 Fig. 2. The mechanism of xno opeation and it-counting inside the 1-it CNNs pesented in [19]. while minimizing accuacy degadation. Successful attempts include DoReFa- Net [33] and QNN [8], which exploe neual netwoks tained with 1-it weights and 2-it activations, and the accuacy dops y 6.1% and 4.9% espectively on the ImageNet dataset compaed to the eal-valued AlexNet. Additionally, BinayNet [7] uses only 1-it weights and 1-it activations in a neual netwok and achieves compaale accuacy as full-pecision neual netwoks on the MNIST and CIFAR-10 datasets. In XNOR-Net [19], Rastegai et al. futhe impove BinayNet y multiplying the asolute mean of the weight filte and activation with the 1-it weight and activation to impove the accuacy. ABC-Net [14] poposes to enhance the accuacy y using moe weight ases and activation ases. The esults of these studies ae encouaging, ut admittedly, due to the loss of pecision in weights and activations, the nume of filtes in the netwok (thus the algoithm complexity) gows in ode to maintain high accuacy, which offsets the memoy saving and speedup of inaizing the netwok. In this study, we aim to design 1-it CNNs aided with a eal-valued shotcut to compensate fo the accuacy loss of inaization. Optimization stategies fo ovecoming the gadient dismatch polem and discete optimization difficulties in 1-it CNNs, along with a customized initialization method, ae poposed to fully exploe the potential of 1-it CNNs with its limited esolution. 3 Methodology 3.1 Standad 1-it CNNs and Its Repesentational Capaility 1-it convolutional neual netwoks (CNNs) efe to the CNN models with inay weight paametes and inay activations in intemediate convolution layes. Specifically, the inay activation and weight ae otained though a sign function, { { 1 if a < 0 1 if a = Sign(a ) = +1 othewise, w w < 0 = Sign(w ) = +1 othewise, (1) whee a and w indicate the eal activation and the eal weight, espectively. a exists in oth taining and infeence pocess of the 1-it CNN, due to the

6 6 Zechun Liu et al. Fig. 3. The epesentational capaility (R) of each laye in (a) 1-it CNNs without shotcut () 1-it CNNs with shotcut. A l indicates the output of the Sign function; A l m denotes the output of the 1-it convolution laye; A l+1 epesents the output of the BatchNom laye; The supescipt l indicates the lock index. convolution and atch nomalization (if used). As shown in Fig. 2, given a inay activation map and a inay 3 3 weight kenel, the output activation could e any odd intege fom 9 to 9. If a atch nomalization is followed, as shown in Fig. 3, then the intege activation will e tansfomed into eal values. The eal weight will e used to update the inay weights in the taining pocess, which will e intoduced late. Compaed to the eal-valued CNN model with the 32-it weight paamete, the 1-it CNNs otains up to 32 memoy saving. Moeove, as the activation is also inay, then the convolution opeation could e implemented y the itwise XNOR opeation and a it-count opeation[19]. One simple example of the itwise opeation is shown in Fig. 2. In contast, the convolution opeation in eal-valued CNNs is implemented y the expensive eal value multiplication. Consequently, the 1-it CNNs could otain up to 64 computation saving. Howeve, it has een demonstated in [7] that the classification pefomance of the 1-it CNNs is much wose than that of the eal-valued CNN models on lage-scale datasets like ImageNet. We elieve that the poo pefomance of 1- it CNNs is caused y its low epesentational capacity. We denote R(x) as the epesentational capaility of x, i.e., the nume of all possile configuations of x, whee x could e a scala, vecto, matix o tenso. Fo example, the epesentational capaility of 32 channels of a inay featue map A is R(A) = = Given a inay weight kenel W, each enty of A W (i.e., the itwise convolution output) can choose the even values fom (-288 to 288), as shown in Fig 3. Thus, R(A W) = Note that since the BatchNom laye is a unique mapping, it will not incease the nume of diffeent choices ut scale the (-288,288) to a paticula value. If adding the sign function ehind the output, each enty in the featue map is inaized, and the epesentational capaility shinks to again. 3.2 Bi-Real Net Model and Its Repesentational Capaility We popose to peseve the eal activations efoe the sign function to incease the epesentational capaility of the 1-it CNN, though a simple shotcut.

7 Bi-Real Net: Enhancing the Pefomance of 1-it CNNs 7 Fig. 4. A gaphical illustation of the taining pocess of the 1-it CNNs, with A eing the activation, W eing the weight, and the supescipt l denoting the l th lock consisting with Sign, 1-it Convolution, and BatchNom. The suscipt denotes eal value, denotes inay value, and m denotes the intemediate output efoe the BatchNom laye. Specifically, as shown in Fig. 3(), one lock indicates the stuctue that Sign 1-it convolution atch nomalization addition opeato. The shotcut connects the input activations to the sign function in the cuent lock to the output activations afte the atch nomalization in the same lock, and these two activations ae added though an addition opeato, and then the comined activations ae inputted to the sign function in the next lock. The epesentational capaility of each enty in the added activations is Consequently, the epesentational capaility of each lock in the 1-it CNN with the aove shotcut ecomes (289 2 ) As oth eal and inay activations ae kept, we call the poposed model as Bi-Real net. The epesentational capaility of each lock in the 1-it CNN is significantly enhanced due to the simple identity shotcut. The only additional cost of computation is the addition opeation of two eal activations, as these eal activations aleady exist in the standad 1-it CNN (i.e., without shotcuts). Moeove, as the activations ae computed on the fly, no additional memoy is needed. 3.3 Taining Bi-Real Net As oth activations and weight paametes ae inay, the continuous optimization method, i.e., the stochastic gadient descent(sgd), cannot e diectly adopted to tain the 1-it CNN. Thee ae two majo challenges. One is how to compute the gadient of the sign function on activations, which is nondiffeentiale. The othe is that the gadient of the loss with espect to the inay weight is too small to change the weight s sign. The authos of [7] poposed to adjust the standad SGD algoithm to appoximately tain the 1-it CNN. Specifically, the gadient of the sign function on activations is appoximated y the gadient of the piecewise linea function, as shown in Fig. 5(). To tackle the second challenge, the method poposed in [7] updates the eal-valued weights y the gadient computed with egad to the inay weight and otains the inay weight y taking the sign of the eal weights. As the identity shotcut will not add difficulty fo taining, the taining algoithm poposed in [7] can also

8 8 Zechun Liu et al. Fig. 5. (a) Sign function and its deivative, () Clip function and its deivative fo appoximating the deivative of the sign function, poposed in [7], (c) Poposed diffeentiale piecewise polynomial function and its tiangle-shaped deivative fo appoximating the deivative of the sign function in gadients computation. e adopted to tain the Bi-Real net model. Howeve, we popose a novel taining algoithm to tackle the aove two majo challenges, which is moe suitale fo the Bi-Real net model as well as othe 1-it CNNs. Besides, we also popose a novel initialization method. We pesent a gaphical illustation of the taining of Bi-Real net in Fig. 4. The identity shotcut is omitted in the gaph fo claity, as it will not change the main pat of the taining algoithm. Appoximation to the deivative of the sign function with espect to activations. As shown in Fig. 5(a), the deivative of the sign function is an impulse function, which cannot e utilized in taining. L = L = L Sign(A l,t ) L F (A l,t ), (2) whee F (A l,t ) is a diffeentiale appoximation of the non-diffeentiale Sign(A l,t ). In [7], F (A l,t ) is set as the clip function, leading to the deivative as a stepfunction (see 5()). In this wok, we utilize a piecewise polynomial function (see 5(c)) as the appoximation function, as Eq. (3) left. 1 if a < 1 2a F (a ) = + a 2 if 1 a < 0 2a a 2 if 0 a < 1 1 othewise, F (a ) a = 2 + 2a if 1 a < 0 2 2a if 0 a < 1, 0 othewise (3) As shown Fig. 5, the shaded aeas with lue slashes can eflect the diffeence etween the sign function and its appoximation. The shaded aea coesponding to the clip function is 1, while that coesponding to Eq. (3) left is 2 3. We conclude that Eq. (3) left is a close appoximation to the sign function than the clip function. Consequently, the deivative of Eq. (3) left is fomulated as Eq. (3) ight, which is a piecewise linea function.

9 Bi-Real Net: Enhancing the Pefomance of 1-it CNNs 9 Magnitude-awae gadient with espect to weights. Hee we pesent how to update the inay weight paamete in the l th lock, i.e., W l { 1, +1}. Fo claity, we assume that thee is only one weight kenel, i.e., W l is a matix. The standad gadient descent algoithm cannot e diectly applied as the gadient is not lage enough to change the inay weights. To tackle this polem, the method of [7] intoduced a eal weight W l and a sign function duing taining. Hence the inay weight paamete can e seen as the output to the sign function, i.e., W l = Sign(Wl ), as shown in the uppe su-figue in Fig. 4. Consequently, W l is updated using gadient descent in the ackwad pass, as follows Note that Wl,t W l,t 1 if W l,t W l,t+1 = W l,t η L W l,t = W l,t η L indicates the element-wise deivative. In [7], Wl,t (i,j) (i, j) [ 1, 1], othewise 0. The deivative chain ule, as follows L = L A l+1,t A l+1,t m whee θ l,t = Al+1,t m and has a negative coelation to W l,t m = L L A l+1,t. (4) W l,t (i,j) is set to is deived fom the θ l,t A l, (5) denotes the deivative of the BatchNom laye (see Fig. 4). As W l,t { 1, +1}, the gadient L is only elated to the sign of W l,t, while is independent of its magnitude. Based on this osevation, we popose to eplace the aove sign function y a magnitude-awae function, as follows: W l,t = Wl,t 1,1 W l,t Sign(W l,t ), (6) whee W l,t denotes the nume of enties in W l,t. Consequently, the update of W l ecomes W l,t+1 = W l,t η L = W l,t Sign(Wl,t ) η L A l+1,t θ l,t A l W l,t, (7) whee Wl,t Wl,t W l,t 1,1 1 W l,t W l,t <1 and θ l,t is associated with the magnitude of W l,t L. Thus, the gadient is elated to oth the sign and W l,t magnitude of W l,t. Afte taining fo convegence, we still use Sign(W) l to otain the inay weight W l (i.e., -1 o +1), and use θl to aso Wl 1,1 W l and to associate with the magnitude of W l used fo infeence.

10 10 Zechun Liu et al. Initialization. In [14], the initial weights of the 1-it CNNs ae deived fom the coesponding eal-valued CNN model pe-tained on ImageNet. Howeve, the activation of ReLU is non-negative, while that of Sign is 1 o +1. Due to this diffeence, the eal CNNs with ReLU may not povide a suitale initial point fo taining the 1-it CNNs. Instead, we popose to eplace ReLU with clip( 1, x, 1) to pe-tain the eal-valued CNN model, as the activation of the clip function is close to the sign function than ReLU. The efficacy of this new initialization will e evaluated in expeiments. 4 Expeiments In this section, we fistly intoduce the dataset fo expeiments and implementation details in Sec 4.1. Then we conduct alation study in Sec. 4.2 to investigate the effectiveness of the poposed techniques. This pat is followed y compaing ou Bi-Real net with othe state-of-the-at inay netwoks egading accuacy in Sec 4.3. Sec. 4.4 epots memoy usage and computation cost in compaison with othe netwoks. 4.1 Dataset and Implementation Details The expeiments ae caied out on the ILSVRC12 ImageNet classification dataset [22]. ImageNet is a lage-scale dataset with 1000 classes and 1.2 million taining images and 50k validation images. Compaed to othe datasets like CIFAR-10 [11] o MNIST [18], ImageNet is moe challenging due to its lage scale and geat divesity. The study on this dataset will validate the supeioity of the poposed Bi-Real netwok stuctue and the effectiveness of thee taining methods fo 1-it CNNs. In ou compaison, we epot oth the top-1 and top-5 accuacies. Fo each image in the ImageNet dataset, the smalle dimension of the image is escaled to 256 while keeping the aspect atio intact. Fo taining, a andom cop of size is selected. Note that, in contast to XNOR-Net and the full-pecision ResNet, we do not use the opeation of andom esize, which might impove the pefomance futhe. Fo infeence, we employ the cente cop fom images. Taining: We tain two instances of the Bi-Real net, including an 18-laye Bi- Real net and a 34-laye Bi-Real net. The taining of them consists of two steps: taining the 1-it convolution laye and etaining the BatchNom. In the fist step, the weights in the 1-it convolution laye ae inaized to the sign of ealvalued weights multiplying the asolute mean of each kenel. We use the SGD solve with the momentum of 0.9 and set the weight-decay to 0, which means we no longe encouage the weights to e close to 0. Fo the 18-laye Bi-Real net, we un the taining algoithm fo 20 epochs with a atch size of 128. The leaning ate stats fom 0.01 and is decayed twice y multiplying 0.1 at the 10th and the 15th epoch. Fo the 34-laye Bi-Real net, the taining pocess includes 40 epochs and the atch size is set to The leaning ate stats fom 0.08 and is multiplied y 0.1 at the 20th and the 30th epoch, espectively. In the

11 Bi-Real Net: Enhancing the Pefomance of 1-it CNNs 11 Fig. 6. Thee diffeent netwoks diffe in the shotcut design of connecting the locks shown in (a) conjoint layes of Sign, 1-it Convolution, and the BatchNom. () Bi- Real net with shotcut ypassing evey lock (c) Res-Net with shotcut ypassing two locks, which coesponds to the ReLU-only pe-activation poposed in [6] and (d) Plain-Net without the shotcut. These thee stuctues shown in (), (c) and (d) have the same nume of weights. second step, we constaint the weights to -1 and 1, and set the leaning ate in all convolution layes to 0 and etain the BatchNom laye fo 1 epoch to aso the scaling facto. Infeence: we use the tained model with inay weights and inay activations in the 1-it convolution layes fo infeence. 4.2 Alation Study Thee uilding locks. The shotcut in ou Bi-Real net tansfes eal-valued epesentation without additional memoy cost, which plays an impotant ole in impoving its capaility. To veify its impotance, we implemented a Plain-Net stuctue without shotcut as shown in Fig. 6 (d) fo compaison. At the same time, as ou netwok stuctue employs the same nume of weight filtes and layes as the standad ResNet, we also make a compaison with the standad ResNet shown in Fig. 6 (c). Fo a fai compaison, we adopt the ReLU-only pe-activation ResNet stuctue in [6], which diffes fom Bi-Real net only in the stuctue of two layes pe lock instead of one laye pe lock. The laye ode and shotcut design in Fig. 6 (c) ae also applicale fo 1-it CNN. The compaison can justify the enefit of implementing ou Bi-Real net y specifically eplacing the 2-conv-laye-pe-lock Res-Net stuctue with two 1-conv-layepe-lock Bi-Real stuctue. As discussed in Sec. 3, we poposed to ovecome the optimization challenges induced y discete weights and activations y 1) appoximation to the deivative of the sign function with espect to activations, 2) magnitude-awae gadient with espect to weights and 3) clip initialization. To study how these poposals enefit the 1-it CNNs individually and collectively, we tain the 18-laye stuctue and the 34-laye stuctue with a comination of these techniques on the ImageNet dataset. Thus we deive = 48 pais of values of top-1 and top-5 accuacy, which ae pesented in Tale 1. Based on Tale 1, we can evaluate each technique s individual contiution and collective contiution of each unique comination of these techniques towads the final accuacy.

12 12 Zechun Liu et al. Tale 1. Top-1 and top-5 accuacies (in pecentage) of diffeent cominations of the thee poposed techniques on thee diffeent netwok stuctues, Bi-Real net, ResNet and Plain Net, shown in Fig.6. Initiali- Weight Activation Bi-Real-18 Res-18 Plain-18 Bi-Real-34 Res-34 Plain-34 zation update ackwad top-1 top-5 top-1 top-5 top-1 top-5 top-1 top-5 top-1 top-5 top-1 top-5 ReLU Clip Oiginal Oiginal Poposed Oiginal Poposed Poposed Real-valued Net Oiginal Oiginal Poposed Oiginal Poposed Poposed Real-valued Net Full-pecision oiginal ResNet[5] ) Compaing the 4 th 7 th columns with the 8 th 9 th columns, oth the poposed Bi-Real net and the inaized standad ResNet outpefom thei plain countepats with a significant magin, which validates the effectiveness of shotcut and the disadvantage of diectly concatenating the 1-it convolution layes. As Plain-18 has a thin and deep stuctue, which has the same weight filtes ut no shotcut, inaizing it esults in vey limited netwok epesentational capacity in the last convolution laye, and thus can hadly achieve good accuacy. 2) Compaing the 4 th 5 th and 6 th 7 th columns, the 18-laye Bi-Real net stuctue impoves the accuacy of the inaized standad ResNet-18 y aout 18%. This validates the conjectue that the Bi-Real net stuctue with moe shotcuts futhe enhances the netwok capacity compaed to the standad ResNet stuctue. Replacing the 2-conv-laye-pe-lock stuctue employed in Res-Net with two 1-conv-laye-pe-lock stuctue, adopted y Bi-Real net, could even enefit a eal-valued netwok. 3) All poposed techniques fo initialization, weight update and activation ackwad impove the accuacy at vaious degees. Fo the 18-laye Bi-Real net stuctue, the impovement fom the weight (aout 23%, y compaing the 2 nd and 4 th ows) is geate than the impovement fom the activation (aout 12%, y compaing the 2 nd and 3 d ows) and the impovement fom eplacing ReLU with Clip fo initialization (aout 13%, y compaing the 2 nd and 7 th ows). These thee poposed taining mechanisms ae independent and can function collaoatively towads enhancing the final accuacy. 4) The poposed taining methods can impove the final accuacy fo all thee netwoks in compaison with the oiginal taining method, which implies these poposed thee taining methods ae univesally suitale fo vaious netwoks. 5) The two implemented Bi-Real nets (i.e. the 18-laye and 34-laye stuctues) togethe with the poposed taining methods, achieve appoximately 83%

13 Bi-Real Net: Enhancing the Pefomance of 1-it CNNs 13 Tale 2. This tale compaes oth the top-1 and top-5 accuacies of ou Bi-eal net with othe state-of-the-at inaization methods: BinayNet [7], XNOR-Net [19], ABC-Net [14] on oth the Res-18 and Res-34 [5]. The Bi-Real net outpefoms othe methods y a consideale magin. Bi-Real net BinayNet ABC-Net XNOR-Net Full-pecision Top % 42.2% 42.7% 51.2% 69.3% 18-laye Top % 67.1% 67.6% 73.2% 89.2% Top % 52.4% 73.3% 34-laye Top % 76.5% 91.3% and 89% of the accuacy level of thei coesponding full-pecision netwoks, ut with a huge amount of speedup and computation cost saving. In shot, the shotcut enhances the netwok epesentational capaility, and the poposed taining methods help the netwok to appoach the accuacy uppe ound. 4.3 Accuacy Compaison With State-of-The-At While the alation study demonstates the effectiveness of ou 1-laye-pe-lock stuctue and the poposed techniques fo optimal taining, it is also necessay to compae with othe state-of-the-at methods to evaluate Bi-Real net s oveall pefomance. To this end, we cay out a compaative study with thee methods: BinayNet [7], XNOR-Net [19] and ABC-Net [14]. These thee netwoks ae epesentative methods of inaizing oth weights and activations fo CNNs and achieve the state-of-the-at esults. Note that, fo a fai compaison, ou Bi-Real net contains the same amount of weight filtes as the coesponding ResNet that these methods attempt to inaize, diffeing only in the shotcut design. Tale 2 shows the esults. The esults of the thee netwoks ae quoted diectly fom the coesponding efeences, except that the esult of BinayNet is quoted fom ABC-Net [14]. The compaison clealy indicates that the poposed Bi-Real net outpefoms the thee netwoks y a consideale magin in tems of oth the top-1 and top-5 accuacies. Specifically, the 18-laye Bi-Real net outpefoms its 18-laye countepats BinayNet and ABC-Net with elative 33% advantage, and achieves a oughly 10% elative impovement ove the XNOR- Net. Simila impovements can e oseved fo 34-laye Bi-Real net. In shot, ou Bi-Real net is moe competitive than the state-of-the-at inay netwoks. 4.4 Efficiency and Memoy Usage Analysis In this section, we analyze the saving of memoy usage and speedup in computation of Bi-Real net y compaing with the XNOR-Net [19] and the full-pecision netwok individually. The memoy usage is computed as the summation of 32 it times the nume of eal-valued paametes and 1 it times the nume of inay paametes

14 14 Zechun Liu et al. 18-laye 34-laye Tale 3. Memoy usage and FLOPs calculation in Bi-Real net. Memoy usage Memoy saving FLOPs Speedup Bi-Real net 33.6 Mit XNOR-Net 33.7 Mit Full-pecision Res-Net Mit Bi-Real net 43.7 Mit XNOR-Net 43.9 Mit Full-pecision Res-Net Mit in the netwok. Fo efficiency compaison, we use FLOPs to measue the total eal-valued multiplication computation in the Bi-Real net, following the calculation method in [5]. As the itwise XNOR opeation and it-counting can e pefomed in a paallel of 64 y the cuent geneation of CPUs, the FLOPs is calculated as the amount of eal-valued floating point multiplication plus 1/64 of the amount of 1-it multiplication. We follow the suggestion in XNOR-Net [19], to keep the weights and activations in the fist convolution and the last fully-connected layes to e eal-valued. We also adopt the same eal-valued 1x1 convolution in Type B shot-cut [5] as implemented in XNOR-Net. Note that this 1x1 convolution is fo the tansition etween two stages of ResNet and thus all infomation should e peseved. As the nume of weights in those thee kinds of layes accounts fo only a vey small popotion of the total nume of weights, the limited memoy saving fo inaizing them does not justify the pefomance degadation caused y the infomation loss. Fo oth the 18-laye and the 34-laye netwoks, the poposed Bi-Real net educes the memoy usage y 11.1 times and 16.0 times individually, and achieves computation eduction of aout 11.1 times and 19.0 times, in compaison with the full-pecision netwok. Without using eal-valued weights and activations fo scaling inay ones duing infeence time, ou Bi-Real net equies fewe FLOPs and uses less memoy than XNOR-Net and is also much easie to implement. 5 Conclusion In this wok, we have poposed a novel 1-it CNN model, dued Bi-Real net. Compaed with the standad 1-it CNNs, Bi-Real net utilizes a simple shot-cut to significantly enhance the epesentational capaility. Futhe, an advanced taining algoithm is specifically designed fo taining 1-it CNNs (including Bi-Real net), including a tighte appoximation of the deivative of the sign function with espect the activation, the magnitude-awae gadient with espect to the weight, as well as a novel initialization. Extensive expeimental esults demonstate that the poposed Bi-Real net and the novel taining algoithm show supeioity ove the state-of-the-at methods. In futue, we will exploe othe advanced intege pogamming algoithms (e.g., Lp-Box ADMM [26]) to tain Bi-Real Net.

15 Bi-Real Net: Enhancing the Pefomance of 1-it CNNs 15 Refeences 1. Couaiaux, M., Bengio, Y., David, J.P.: Binayconnect: Taining deep neual netwoks with inay weights duing popagations. In: Advances in neual infomation pocessing systems. pp (2015) 2. Gag, R., BG, V.K., Caneio, G., Reid, I.: Unsupevised cnn fo single view depth estimation: Geomety to the escue. In: Euopean Confeence on Compute Vision. pp Spinge (2016) 3. Gishick, R., Donahue, J., Daell, T., Malik, J.: Rich featue hieachies fo accuate oject detection and semantic segmentation. In: Poceedings of the IEEE confeence on compute vision and patten ecognition. pp (2014) 4. Han, S., Mao, H., Dally, W.J.: Deep compession: Compessing deep neual netwoks with puning, tained quantization and huffman coding. axiv pepint axiv: (2015) 5. He, K., Zhang, X., Ren, S., Sun, J.: Deep esidual leaning fo image ecognition. In: Poceedings of the IEEE confeence on compute vision and patten ecognition. pp (2016) 6. He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep esidual netwoks. In: Euopean confeence on compute vision. pp Spinge (2016) 7. Huaa, I., Couaiaux, M., Soudy, D., El-Yaniv, R., Bengio, Y.: Binaized neual netwoks. In: Lee, D.D., Sugiyama, M., Luxug, U.V., Guyon, I., Ganett, R. (eds.) Advances in Neual Infomation Pocessing Systems 29, pp Cuan Associates, Inc. (2016), inaized-neual-netwoks.pdf 8. Huaa, I., Couaiaux, M., Soudy, D., El-Yaniv, R., Bengio, Y.: Quantized neual netwoks: Taining neual netwoks with low pecision weights and activations (2016) 9. Iandola, F.N., Han, S., Moskewicz, M.W., Ashaf, K., Dally, W.J., Keutze, K.: Squeezenet: Alexnet-level accuacy with 50x fewe paametes and 0.5 m model size. axiv pepint axiv: (2016) 10. Ioffe, S., Szegedy, C.: Batch nomalization: Acceleating deep netwok taining y educing intenal covaiate shift. axiv pepint axiv: (2015) 11. Kizhevsky, A., Hinton, G.: Leaning multiple layes of featues fom tiny images. Tech. ep., Citesee (2009) 12. Kizhevsky, A., Sutskeve, I., Hinton, G.E.: Imagenet classification with deep convolutional neual netwoks. In: Advances in neual infomation pocessing systems. pp (2012) 13. Lai, L., Suda, N., Chanda, V.: Deep convolutional neual netwok infeence with floating-point weights and fixed-point activations. axiv pepint axiv: (2017) 14. Lin, X., Zhao, C., Pan, W.: Towads accuate inay convolutional neual netwok. In: Advances in Neual Infomation Pocessing Systems. pp (2017) 15. Liu, B., Wang, M., Fooosh, H., Tappen, M., Pensky, M.: Spase convolutional neual netwoks. In: Poceedings of the IEEE Confeence on Compute Vision and Patten Recognition. pp (2015) 16. Liu, F., Shen, C., Lin, G., Reid, I.D.: Leaning depth fom single monocula images using deep convolutional neual fields. IEEE Tans. Patten Anal. Mach. Intell. 38(10), (2016) 17. Luo, W., Sun, P., Zhong, F., Liu, W., Zhang, T., Wang, Y.: End-to-end active oject tacking via einfocement leaning. ICML (2018)

16 16 Zechun Liu et al. 18. Netze, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natual images with unsupevised featue leaning. In: NIPS wokshop on deep leaning and unsupevised featue leaning. vol. 2011, p. 5 (2011) 19. Rastegai, M., Odonez, V., Redmon, J., Fahadi, A.: Xno-net: Imagenet classification using inay convolutional neual netwoks. In: Euopean Confeence on Compute Vision. pp Spinge (2016) 20. Ren, S., He, K., Gishick, R., Sun, J.: Faste -cnn: Towads eal-time oject detection with egion poposal netwoks. In: Advances in neual infomation pocessing systems. pp (2015) 21. Romeo, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: Hints fo thin deep nets. axiv pepint axiv: (2014) 22. Russakovsky, O., Deng, J., Su, H., Kause, J., Satheesh, S., Ma, S., Huang, Z., Kapathy, A., Khosla, A., Benstein, M., et al.: Imagenet lage scale visual ecognition challenge. Intenational Jounal of Compute Vision 115(3), (2015) 23. Simonyan, K., Zisseman, A.: Vey deep convolutional netwoks fo lage-scale image ecognition. axiv pepint axiv: (2014) 24. Sun, Y., Wang, X., Tang, X.: Deep convolutional netwok cascade fo facial point detection. In: Poceedings of the IEEE confeence on compute vision and patten ecognition. pp (2013) 25. Szegedy, C., Liu, W., Jia, Y., Semanet, P., Reed, S., Anguelov, D., Ehan, D., Vanhoucke, V., Rainovich, A.: Going deepe with convolutions. In: Poceedings of the IEEE confeence on compute vision and patten ecognition. pp. 1 9 (2015) 26. Wu, B., Ghanem, B.: lp-ox admm: A vesatile famewok fo intege pogamming. IEEE Tansactions on Patten Analysis and Machine Intelligence (2018) 27. Wu, B., Hu, B.G., Ji, Q.: A coupled hidden makov andom field model fo simultaneous face clusteing and tacking in videos. Patten Recognition 64, (2017) 28. Wu, B., Lyu, S., Hu, B.G., Ji, Q.: Simultaneous clusteing and tacklet linking fo multi-face tacking in videos. In: Poceedings of the IEEE Intenational Confeence on Compute Vision. pp (2013) 29. Zhang, H., Kyaw, Z., Chang, S.F., Chua, T.S.: Visual tanslation emedding netwok fo visual elation detection. In: CVPR. vol. 1, p. 5 (2017) 30. Zhang, H., Kyaw, Z., Yu, J., Chang, S.F.: Pp-fcn: Weakly supevised visual elation detection via paallel paiwise -fcn. axiv pepint axiv: (2017) 31. Zhou, A., Yao, A., Guo, Y., Xu, L., Chen, Y.: Incemental netwok quantization: Towads lossless cnns with low-pecision weights. axiv pepint axiv: (2017) 32. Zhou, E., Fan, H., Cao, Z., Jiang, Y., Yin, Q.: Extensive facial landmak localization with coase-to-fine convolutional netwok cascade. In: Poceedings of the IEEE Intenational Confeence on Compute Vision Wokshops. pp (2013) 33. Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., Zou, Y.: Doefa-net: Taining low itwidth convolutional neual netwoks with low itwidth gadients. axiv pepint axiv: (2016) 34. Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment acoss lage poses: A 3d solution. In: Poceedings of the IEEE confeence on compute vision and patten ecognition. pp (2016)