Neural Networks A Model of Boolea Fuctios Berd Steibach, Roma Kohut Freiberg Uiversity of Miig ad Techology Istitute of Computer Sciece D-09596 Freiberg, Germay e-mails: steib@iformatik.tu-freiberg.de kohut@math.tu-freiberg.de Abstract This paper deals with the represetatio of Boolea fuctios usig artificial eural etworks ad poits out three importat results. First, usig a polyomial as trasfer fuctio, a sigle euro is able to represet a o-mootoous Boolea fuctio. Secod, the umber of iputs i the eural etwork ca be decreased if the biary values of the Boolea variables are ecoded. This approach simplifies sigificatly the ecessary umber of euros i the artificial eural etwork. Fially, a algorithm to compute the miimal umber of euros was developed. The lower boud, calculated by this algorithm, correspods to a suggested structure of artificial eural etworks. A eample shows, how such a simple artificial eural etwork may represet a Boolea fuctio. Itroductio The rapid developmet of computer systems is oe of the reasos why theoretical ad practical researches of the priciples of the ature were forced. May ew scietific ad techological bruches, which have coectio both with biological laws of the ature ad applied techical methods, appeared i the last fifty years, e.g. bioelectroics, artificial ielegace. Especially methods of the artificial itelligece are widely used i moder techologies. New waves of developmet such methods like data miig [8], fuzzy logic, cluster aalysis, epert systems, geetic algorithms, ad visual data recogitio were caused by the computer sciece. Some of these methods base o eural etworks. The coectio betwee eural etworks ad Boolea fuctio is ot ew. Oe of the pessimistic results i the history of eural etworks was the coclusio of Misky ad Papert about impossibility of represetatio of all fuctioal depedece. They proved this property o the Boolea fuctio eclusive OR (XOR) [4, 8, 0]. This statemet is correct oly for simplest artificial eural etwork, amed Perceptro of Roseblatt [6]. The result of Misky ad Papert has bee later by Mkrttschja disproved [5]. I geeral, all types of fuctioal depedece icludig Boolea fuctios ca be represeted usig eural etworks. This paper gives a itroductio, how artificial eural etworks ca model Boolea fuctios. The rest of this paper is orgaized as follows. Sectio itroduces both Boolea fuctios ad euros. Sectio 3 shows, how o-mootoous Boolea fuctio may represet by sigle euros. Sectio 4 describes, how the ecodig of the Boolea values ca be used to miimize the size of a artificial eural etwork. A algorithm to calculate a strog lower boud of euros is preseted i sectio 5. Fially, sectio 6 cocludes the paper. Prelimiaries Let,, 3,,, be Boolea variables, i {0, } = B. A Boolea fuctio f is defied by f B : B, where B = {(0...00),(0...0),(0...0),...,(...)} [, 7]. The umber of vectors i the Boolea space B is N =. The simplest represetatio of a Boolea fuctio is a table that defies the fuctio value for each of the iput vectors, see eample i figure a.
There are may other represetatios of Boolea fuctios. Oe of them is the visualizatio of the Boolea fuctio as - dimesioal hypercube [, 7]. Each iput vector of the Boolea fuctio correspods to eactly oe verte i - dimesioal space. The fuctio values are labeled by if f = or о if f = 0. The visualizatio of the fuctio f = ( + 3 ) shows figure b. 3 f 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a) b) Figure. The Boolea fuctio f = ( + 3 ), a) Fuctio Table, b) Visualizatio Net, the artificial eural etwork is iputs itroduced. These etworks models are weights of syaptic coectios simple way biological eural etworks. w addig elemet A artificial eural etwork is a w coectio of simple processig w 3 ao output elemets, called euros, which operates 3 Y S i parallel [, 3, 4, 6, 0]. I this paper w artificial eural etworks are restricted to feed forward etworks, which ca be = S w i i Y=F(S) syapses represeted by directed acyclic graphs (DAG), see figures 4, 6. The structure of Figure. Geeral structure of euro a euro is show i figure. The mathematical descriptio of euro is: Y = F i w i () where Y - output sigal of euro, F - activatio fuctio, i - sigal from the iput і to the syapse i, w i - weight coefficiet of the iput і, - umber of iputs. I geeral, the coditio to activate a euro is (), where θ is threshold of trasfer fuctio. w i i θ 0 () The uequatio () icludes the equatio (3) of a -dimesioal hyper-plae that divides the vertices of the hypercube ito two part sets. w i i θ = 0 (3) The output of a euro may be coected to several syapses of other euros. The activatio fuctio is sometimes called a trasfer fuctio. I the case of a bouded rages the activatio fuctios are called squashig fuctios, such as the commoly used tah (hyperbolic taget) ad the logistic fuctio (+ e ) [3, 0], see figure 3. Y Y Y Y 0 T 0 - a) b) c) 0 d) Figure 3. Activatio fuctios: a) jump, b) threshold, c) hyperbolic taget, d) logistic fuctio
3 Neural Networks of No-mootoous Boolea Fuctios 3. The Problem We cosider elemetary Boolea fuctios NOT, OR, AND, XOR. Table shows these fuctios. Table. Elemetary Boolea fuctios + 0 0 0 0 0 0 0 0 0 0 0 0 The basic questio is, whether oe euro is eough to represet each of these Boolea fuctios. The eural etworks to represet the first three fuctios are show i figure 4. w y w y a) w b) Figure 4. Neural etwork s structures: a) for NOT fuctio; b) for OR-, or AND - fuctios The fuctios NOT, OR, or AND ca be represeted usig oe euro. A simple eplaatio for that is show i figure 5. The activatio fuctios from figure 3 distiguish the areas havig the output zero or oe. Based o the sum of the weighted iputs oly mootoous Boolea fuctios may calculate directly by such simple activatio fuctios. If the Boolea fuctio is depeded o oe variable there eist a cut poit, see figure 5 a. I case of two variables i the Boolea fuctio, a lie separates all vertees labeled by 0 from the all other vertees labeled by, see figures 5 b ad c. Geerally, if there are more the two Boolea variables, a plae or hyper-plae separates both areas. For the XOR-fuctio such a lie or hyper-plae does ot eist, see figure 5 d. Eactly o this eample Misky ad Papert proved impossibility of represetatio some fuctioal depedeces usig oe euro [4]. I the case of eural elemet with two iputs ad threshold θ, we have from (3) the equatio of the lie i the plae - : w + w = θ (4) 0 0 0 0 NOT 00 OR 0 00 AND 0 00 XOR 0 Figure 5. Visualizatio the separator of eural etworks i the Boolea space a) NOT-fuctio; b) OR-fuctio; c) AND-fuctio; d) XOR-fuctio 3. Kow Approaches for No-mootoous Boolea Fuctios The simplest approach is to epress the o-mootoous Boolea fuctio as disjuctive ormal form, which icludes oly mootoous operatios. Each of these mootoous operatios is mapped to
Y oe euro. This simple method is useable for each Boolea fuctio, but already the simplest o-mootoous Boolea fuctio (5) eeds more the oe euro. The figure 6 shows the eural etwork, associated to the XOR-fuctio (5). Figure 6. Neural etwork of the XOR-fuctio usig mootoous euros Y = XOR( = = + (5), ) The secod kow method uses oe additioal iput ito oe sigle euro i order to realize the XOR-fuctio. This method eeds a additioal OR-gate ad was proposed by Mkrttschja [5]. The mai idea is the trasformatio of the two valued Boolea fuctio ito the three dimesioal Boolea space, which chages (4) ito the equatio (6), where w =, w =, w 3 =, θ = 0,5 ad a =, a =, a 3 = OR i order to model the XOR-fuctio (5). w a + w a + w 3 a 3 = θ (6) The equatio (6) substitutes the lie (4) ito a plae; see figure 7 a. Correspodigly, The Boolea fuctio f ( ) = + is trasformed ito f ( α ) = αα α 3 + αα α 3 + αα α3. a) b) Figure 7. The represetatio of the XOR-fuctio, a) plae defied by (6), b) projectio of the plae i the two-dimesioal space The chose parameter of (6) map the four vertices of B ito the vertices labeled by a dotted circle i figure 7 a. Thus, the ais ca be modeled i the plae a a 3 ad the ais i the plae a a 3. The retur trasformatio of the separatio pla, defied by (6) ad visualized i figure 7 a, leads to the broke lie, visualized i figure 7 b. This broke lie separates the vertices labeled by 0 form that vertices labeled. Note, usig this approach, the XOR-fuctio eeds two euros i the etwork, the first two syapses euro calculates the OR-fuctio ad the secod three syapses euro computes the fial XOR-fuctio. 3.3 Polyomial Trasfer Fuctios The two methods, discussed i sectio 3., eed more tha oe euro to realize a o-mootoous Boolea fuctio. These methods icrease umber of sigals, modeled i a lager dimesioal space. Alteratively, a sigle euro ca realize the o-mootoous Boolea fuctio, if this euro takes advatages of a more comple activatio fuctio, like a polyomial (7) or trigoometrical (8) fuctio.
0 00 0 Figure 8. Represetatio of the XOR-fuctio usig sigle euros that evaluates a polyomial trasfer fuctio d 0 0 w + w i i θ (7) tf 0 0 w + w i i θ (8) The advatage of this method is a oe to oe mappig of the Boolea fuctio to the eural etwork. I case of the XOR-fuctio we chose d = w 0 = -, w = w =, k =, ad θ = -0.5, ad get the formula (9), which is true oly if, the XOR-fuctio is equal to oe. + + 0.5 0 i (9) Figure 8 illustrates the valid area of the uequatio (9). k 4 Ecodig the Iputs of Boolea Fuctios Now the case of more comple Boolea fuctios depedig o a large umber of variables is cosidered. I geeral there are three methods to create the euroal etwork that describes a comple Boolea fuctio. First, the Boolea fuctio is mapped to a special euroal etwork usig such euros that realizes the NOT-, AND-, ad OR-fuctio. I a secod step, mergig of selected euros ca miimize the umber of euros of this euroal etwork. This is a time cosumig process. Secod, a geeral artificial euroal etwork is used. This euroal etwork has oe iput for each Boolea variable ad a sufficiet umber of layers. The advatage of such simple structure of the euroal etwork is coected with the disadvatage of a etremely time cosumig traiig process. For eample, a Boolea fuctio with 30 argumets has 30 = 073 74 84 differet biary vectors. Each of them must apply to trai the artificial euroal etwork. Usig the traiig method Back Propagatio this task may eed approimately oe year. A quick algorithm, like Fuctioal o the sets of tabled fuctios [8, 9] to trai the eural etwork, speed up this procedure 50 times, but this algorithms required 3 54 70 byte = 3 Gigabyte radom access memory (RAM). The both methods, described above, are restricted i the time, eeded to specify all details of the artificial euroal etwork. Alteratively, we suppose the ecodig of a certai umber of Boolea variables ito real umber. A euro is able to process real umbers up to a give precisio. A possible ecodig for two or Boolea variable ito the iterval [0, ] is show i table 3. The umber of iputs ad the umber of euros as well is decreased, usig this ecodig. Table 3. Ecodig iput biary vectors for eural etwork = 0 0 0 0 0 0 =/3 0 For geeral case 0 0 3 = /3 0 4 = For istace, a 8-bits umber represetatio of a sigal allow to distiguish betwee 8 values. Uder this assumptio, a Boolea fuctio of 6 argumets ca be realized by a eural etwork with two iputs ( 8 8 = 6 ). This decreases the umber of iputs by oe magitude (from 6 to ), ad cosequetly the umber of layers ad the umber of ecessary euros is decreases as well. The smaller umber of euros speeds-up of traiig process sigificatly. This approach allows to epress very comple Boolea fuctios i much simpler structures of eural etworks. The restrictio of this method is sesitivity of euros.
5 Lower Boud of the Number Neuros i the Neural Network I this sectio we propose a algorithm, which calculates the lower boud of euros o each layer of eural etwork. Thus, this algorithm fids the structure of the smallest possible eural etworks of Boolea fuctios i terms of the umber of layers ad the umber of euros i each layer as well. The algorithm uses the followig assumptio. The iterval of the sigals level is [0, ]. The sesitivity of euros is labeled by, which meas that the euro ca distiguish betwee N s = - differet values of a sigal. A Boolea fuctio of m argumets eeds a sesitivity, defied by (0). (0) m Algorithm (Number of Neuros ad Layers). Assume, the Boolea fuctio may be represeted by oe hidde layer, r =.. Compute the miimal umber of euros K r o the layer r usig the formula (). The special brackets deote the smallest iteger value that is lager the the eclosed real value. if >, the log K r = else K r = 0 () log Kr 3. If NS, the the assumptio of step is true ad miimal umber of euros to represetatio Boolea fuctio ca be calculated by (). ma M = + () z K r r = r Otherwise, the assumptio of step is false, the umber of hidde layers r is icremeted by oe, the ecessary sesitivity for et layer is elarged by (3). : = K r (3) Eample ad the steps ad 3 must be repeated for the et hidde layer. What is the miimal euroal etwork of the o-mootoous Boolea fuctio (4)? = 8 f We assume, that the sesitivity of the available euros is = 4, so that the euro ca distiguish betwee N s = - = 4 values of a sigal. Because the Boolea fuctio has = 8 argumets, the ecessary sesitivity is = 8. A sigle hidde layer is assumed ad the umber of euros i this hidde layer K is calculated (5). i (4) log 8 log 8 K = = = = 4 log 4. (5) log K Sice 4 N = 6, the algorithm fids i step 3 the miimal umber of euros (6). = S M z = + K = + = 3. (6) The simplest feed forward eural etwork to represet the Boolea fuctio (4) eeds oe iput layer, oly oe hidig layer, ad a oe-euro output layer, see figure 9. Three euros havig a sesitivity = -4 determie the Boolea fuctio which depeds o 8 variables.
f 8 = i = i Figure 9. Simple architecture of feed forward eural etwork to represet the Boolea fuctio (4) 6 Coclusio ad Future Work Each Boolea fuctio ca represet by several artificial eural etworks. Usig Boolea sigals o the syapses ad oe of the traditioal activatio fuctios, a sigle euro is oly able to realize a mootoous Boolea fuctio. There eist a oe-to-oe mappig of NOT-, AND- or OR-gates to such simple euros. Artificial eural etworks created by this simple oe-to-oe mappig describe a upper boud of euroal etworks ad have o beefit compared to correspodig gate circuit. The reaso for that is the more comple structure of the euros i compariso to the logic gates. Two approaches to simplify such euroal etworks were suggested. First, usig polyomial or trigoometrical activatio fuctios, a sigle euro ca represet o-mootoous Boolea fuctios, which covers a certai set of logic gates. This approach reduces the umber of euros sigificatly ad a real beefit of such artificial eural etworks i terms of space ad time is possible. The secod way to take advatages of the capability of euros cosists i the ecodig of a lager umber of Boolea variables ito each iput sigal of the euro. This approach is oly limited by the sesitivity of the used euros ad reduces the umber of euros ad layers of the artificial eural etwork drastically. Thus, both the memory space to store eural etwork ad the time of the learig ad usig phase are decreased. I the future we will eted the theory artificial eural etworks i order fid a optimal represetatio of Boolea fuctios usig such etworks. Based o these results we will desig ad implemet a package, that uses artificial eural etworks for compact represetatio ad fast computatios ad evaluatios of Boolea fuctios. 7 Refereces [] Bochma, D.: Steibach, B.: Logiketwurf mit XBOOLE. Verlag Techik, Berli, 99. [] Gschwedter A. B. DARPA Neural Network Study. AFCEA Iteratioal Press, p. 60, 988. [3] Kröse, B.; v. d. Smagt P.: A itroductio to Neural Networks. Uiversity of Amsterdam, 996. [4] Misky M. ad Papert S. Perceptros: A Itroductio to Computatioal Geometry. MIT Press, Cambridge, MA, 969. [5] Mkrttschja S.O. Neuros ad eural etworks - Itroductio i the theory of formal euros. (I Russia), Eergy, Moscow, 97. [6] Roseblatt F. Priciples of Neurodyamics. Sparta, New York, 96. [7] Steibach, B.: XBOOLE - A Toolbo for Modelig, Simulatio, ad Aalysis of Large Digital Systems. System Aalysis ad Modelig Simulatio, Gordo & Breach Sciece Publishers, 9(99), Number 4, pp. 97-3, 99. [8] Tkacheko, R.: Kohut, R.: Feed forward eural etworks: the problems of sythesis ad usig. (I Ukraiia), Bulleti of Lviv Polytechic Natioal Uiversity: Computer Egieerig ad Iformatio Techologies, 433, Lviv, pp. 66-7, 00. [9] Tkacheko, R.: Kohut, R.: Fuctioal etesio of iputs i feed forward eural etwork with o-iteratio learig. (I Ukraiia), Techical ews (), (3), Lviv, pp. 9-94, 00. [0] Wasserma P.D.: Neural Computig Theory ad Practice. Va Nostrad Reihold, New York, 989.