A survey of the SpiNNaker Project : A massively parallel spiking Neural Network Architecture

Transcription

1 A survey of the SpiNNaker Project : A massively parallel spiking Neural Network Architecture Per Lenander Mälardalen University Robotics program Västerås, Sweden plr07001@student.mdh.se ABSTRACT Scientists have through the decades tried many different approaches to creating true Artificial Intelligence (AI). There are two main goals to this endeavour; First, to create more complex and intelligent machines and software, and second, to further the understanding of the human brain. One line that some has followed is to model the biological structure of the brain hard AI instead of modelling the problem solving process soft AI. This paper is a survey of a hard AI endeavour: the SpiNNaker project, a massively parallel spiking neural network architecture. The SpiNNaker is designed as many smaller specialized chips connected together in a network architecture. The focus is on high message throughput and clever routing algorithms, rather than on the processing power of each individual node. Another big focus is how to manage power consumption in a system consisting of a million processor cores. This paper briefly explains the purpose and the inner workings of the SpiNNaker, and provides a glimpse of what is to come in the Artificial Intelligence field in the near future. Keywords Neural Network, Parallel computing, SpiNNaker. 1. INTRODUCTION The concept of the neural network is by no means a new idea. It has it roots in recreating how the human brain works by modelling neurons and the connections between these. This field of research is usually referred to as hard Artificial Intelligence. A key feature of this model is learning; the network is exposed to a large number of training examples, and modified through one of several learning algorithms to better solve the given problem. Another method is to have the network Anton Fosselius Mälardalen University Robotics program Västerås, Sweden afs07002@student.mdh.se learn while the system is on-line, by providing positive and negative feedback on resulting output. This model can be constructed either in hardware or in software (an Artificial Neural Network). Neural networks have many applications, from classification of objects[10], pattern recognition[16][1] and similar problems, to modelling and optimizing complex mathematical functions[12]. From the simplest model of a neural network the feedforward structure using perceptrons[14][9] to more complex structures such as bi-directional recurrent networks, there are many different approaches to constructing neural networks. A comprehensive summary of the field can be found in Simon Haykins book[3]. We will further examine one such model called the spiking neural network. The basic idea is to build a network of interconnected neurons, where each neuron has an electrical potential. Once this potential reaches a certain level, the neuron sends out signals to all other neurons connected to it. These connections are called synapses, and can be either exciting or inhibiting. This model is much closer to the human brain than the simplistic feed-forward network, but also much more computationally expensive. To facilitate learning in the network, synaptic weights can be adjusted through synaptic plasticity algorithms. This will be explained further in section 4. A general problem with artificial neural networks is how computationally expensive it is to simulate and train the network. This problem, combined with knowledge of how neurons work in a human brain through asynchronous communication makes neural networks perfect for a parallel architecture. An example of such an architecture is the GPU-SNN[11], which models a large number of spiking neurons on a General Purpose Graphical Processing Unit 1. Another architecture with a similar vision the SpiNNaker project is the focus of this paper. The project was started at the University of Manchester in 2006 by Stephen Furber 2, ICL Professor of Computer Engineering, and is scheduled to finish Or, to put it in simpler terms, an ordinary desktop graphics card. 2 Prof. Furbers earlier work includes designing the 32-bit ARM architecture. 1

2 at the average rate of 10 Hz. Figure 1: Biological model of a neuron. Image from Phenomenon of Science by Valentin Turchin, The purpose of this project is to create a neural network with approximately 1% the number of neurons in the human brain. To do this, the project team designed a specialized hardware architecture focused on massive parallelism. The project is funded by the Engineering and Physical Sciences Research Council in the period 01 October 2006 to 31 Mars 2010, with to University of Southampton and to The University of Manchester, a grand total of In addition to this, ARM Limited and Silistix Limited are sponsoring the project with hardware and technology. The paper is divided into sections describing the different aspects of the SpiNNaker project. Section 2 begins by introducing the concept of a spiking neural network. Section 3 discusses the hardware architecture and communication protocols of the system, while section 4 describes the software implemented in each processor core. Section 5 and 6 discusses future work and conclusions about the project. 2. SPIKING NEURAL NETWORK 2.1 Biological inspiration The idea of neural networks stem from observations of nature, specifically from the human brain. The working of the human brain is significantly different from an ordinary sequential computer. While the computers of today are significantly faster and has much higher computational power than single neurons, the parallel nature and complexity of the brain allows for a high level of pattern recognition, deduction and adaptation. The brain consists of a vast network of neurons, a form of cells able to transmit electrical signals (see figure 1). The neuron has a set of dendrites (inputs) and an axon (output) that is connected to the dendrites of other neurons through synapses. When the neuron reaches a certain electrical potential, it will send out an electrical signal through the axon, which will propagate through the synapses to all connected neurons. Each synapse may modify the signal to either excite of inhibit the potential of the receiving neuron. A human brain consists of roughly one hundred billion neurons, where each neuron is connected to several tens of thousands other neurons. Each neuron transmits a signal a spike 2.2 Artificial Model This biological construction can be modelled, either in software or in hardware. In its simplest form, the spiking neuron model has a membrane potential. Over time, the neuron will receive short energy bursts spikes from other neurons. These spikes will slightly change the membrane potential in a linear fashion, but after some time the neuron will degrade to a resting potential. Once a threshold potential is reached, the neuron will send out a spike of its own to nearby connected neurons and return to the resting potential. If a neuron has many incoming exciting spikes, it will send out spikes more often. Computationally, the spiking neurons essentially are independent of each other, so there is a great potential for taking advantage of parallelism and asynchronous hardware and software features. Each neuron can be designed as an object that can send and receive messages (spikes), much like a computer network. 2.3 Application of theory The SpiNNaker is developed with all this in mind. Instead of focusing on high speed computer chips, the SpiNNaker is a network of low power nodes, connected through a set of high speed package routers for communication. This design adds a great redundancy to the SpiNNaker when compared to conventional computers. There is no central point of failure, and the network can still function even if packets are lost, or entire nodes drop of the grid. This is also similar to how the brain works. Though a general model of the low level building block of the brain the neuron is known, and some of the high level brain activity can be measured, scientists do not currently know how the network of neurons form the complex output that they do. The project team hopes to further research within this area by replicating the low level structure of the brain, so that neuroscientists have a simulation environment to experiment on. 3. HARDWARE 3.1 Chip design The core of the project is a hardware architecture designed with massive parallelism in mind. The system consists of a series of specially designed chips connected together in a network with a two-dimensional toroidal triangular mesh structure (see figure 2). Each node in the network is relatively low in computational power, but when put together forms a very powerful and robust system. Every node consists of twenty ARM9 3 cores, each clocked at 200 MHz. The chip also has a router for communication between different cores, both locally on the chip and globally in the network. Each core can simulate about 1000 neurons in biological real-time, however only nineteen of the cores 3 ARM is a 32-bit Reduced Instruction Set Computer (RISC) processor architecture. The design focuses on low power consumption. 2

3 Figure 3: Overview of the network architecture of SpiNNaker. Each green node is a SpiNNaker chip with twenty ARM9 processor cores. Image from SpiNNaker website. Figure 2: Model of the two-dimensional toroidal triangular mesh structure. Image from A GALS Infrastructure for a Massively Parallel Multiprocessor, by Luis A. Plana[13]. are used for simulation. The spare core is used to perform management tasks and is assigned by being the first core on the chip that finishes its self-test on power-up. This means that one SpiNNaker chip can simulate about neurons. To achieve the goal of simulating 1% of the human brain, the SpiNNaker network will have to be able to simulate one billion neurons. If one chip can simulate about neurons then SpiNNaker chips is needed to achieve this goal. To build a neural network of this size is a demanding task, where the constrains are power consumption, communication and flexibility. When the initial design was being considered there were a comparison between Complex Instruction Set Computers (CISC) and Reduced Instruction Set Computer (RISC) processors. These are two different archetypes in processor design, one focusing on many complex and powerful instructions while the other uses a small number of highly optimized instructions. CISC is the CPU architecture that is most common to use for heavy calculations today because its superior processor power. However those CPU:s are expensive and needs a lot of power. RISC CPUs are a lot cheaper and demands less power to do the same amount of calculations as the CISC CPUs. The solution was to use a lot of cheap low power RISC CPUs (ARM9) and running them in parallel. 3.2 Communication In this section we explain the different techniques used to makes a network of this size possible. Ordinary network protocols such as Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) would create a huge overhead and take a lot of time to set up, and the communication speed would be too slow. Because of this, a huge part of the SpiNNaker project is about communication, both on the SpiNNaker chip and between different nodes. The SpiNNaker chip does not have a standard data bus (as a motherboard in an ordinary PC would have). The reason for this is that the system is designed to be globally asynchronous, which means that every processor runs on its own clock. To realize an ordinary bus solution with this design in mind, the bus would have to have twenty-one bus masters. Every ARM9 core, as well as the router needs to be masters and connected to the same bus. This would lead to a really complex and poorly optimized communication architecture. Instead the SpiNNaker chip has two Network on Chips (NoC), used as follows: The System NoC is SpiNNaker s way to give the ARM9 cores access to shared system resources. One of those system resources is the off-chip 1 Gbit Synchronous Dynamic Random Access Memory (SDRAM) that is shared between the twenty ARM cores. The System NoC has a bandwidth of 8 Gbps to give the cores fast access to the system resources like the SDRAM. Another example of the system resources that can be accessed through the system NoC is the Ethernet interface. The Ethernet interface allows an external computer to access the SpiNNaker network (see figure 3), this is used to map and initiate the routing tables on the network but can also be used to monitor and configure the neural network. The other Network on Chip, the Communications NoC allows each core to connect to other ARM9 cores on- or offchip. The chip is designed to be Globally Asynchronous, Locally Synchronous (GALS) and use Delay Insensitive communication (DI), two concepts that will shortly be explained GALS As Furber states in [2]: One of the three key ideas behind the SpiNNaker architecture is that time models itself : time is free running and there is no global synchronization. This is sometimes referred to as bounded asynchrony since interactions are asynchronous but threads progress at very similar rates. 3

4 topology] is that physical and logical connectivity are decoupled. Figure 4: 1 of 4 Delay Insensitive (DI) code. Image from A GALS Infrastructure for a Massively Parallel Multiprocessor, by Luis A. Plana[13]. The SpiNNaker uses Global Asynchronous, Locally Synchronous communication (GALS)[13], where each ARM9 core uses its own clock (locally synchronous), and all communication to and from the core is asynchronous. This is the core design of the entire SpiNNaker project, everything will run asynchronous and in parallel with locally synchronous elements. A synchronous design of a system of this size would demand a vast amount of resources just to keep the system synced. With a asynchronous system design, you free a lot of system resources that can be put to better use. However, we will have to use asynchronous communication, which brings its own problems. How this asynchronous communication works is explained in the next sections Delay Insensitive communication Delay Insensitive communication (DI)[13] is a type of failsafe asynchronous communication that is used for all internal communication in the SpiNNaker network. DI expects that all delays are positive and finite, and is used to make communication between nodes very robust. Protocols such this are more robust than simple parallel communication, but faster than simple serial communication (since one symbol can be sent each communication cycle, rather than just one bit). DI uses 1-of-N code, where N are the number of wires that are used and the 1 is how many of the wires that can be active at the same time. Figure 4 shows how 1-of-4 DI represents binary values. Between each value transmitted, a null value is sent. This is due to the globally asynchronous structure, to make sure that the correct value is received. This is called Return-To-Zero (RTZ) protocol. When data is received, an acknowledgement (ACK) is sent to confirm that the data was received. The null token that RTZ always alternate with limits the bandwidth of the communication. However, there is an alternative called Non Return to Zero (NRZ)[13] that can be used to achieve a higher bandwidth. In the NRZ protocol there is no alternation with the null token, which gives it twice the bandwidth but the same energy consumption compared to the RTZ protocol, at the expense of accuracy. NRZ is used in the SpiNNaker project, as it can communicate data faster than the alternatives. The risk for errors in communication are compensated for at the architectural level SpiNNaker is built to be fault tolerant Routing and Network communication In this section we will explain how a signal travels from one neuron to another and go through all the steps on the way. Furber states in [2]: The second key idea behind SpiNNaker [Virtual What this entails is that each neuron in the SpiNNaker system is independent of its actual physical location in the system. Due to the fast routing protocols used, the same neural network can be mapped to the SpiNNaker architecture in any way, with no need for connected neurons to be mapped to physically closely connected nodes. The SpiNNaker network is built up by a large number of circuit boards, and on each of those circuit boards there are several SpiNNaker chips nodes in the network. On every SpiNNaker chip there are twenty ARM9 cores where each core can simulate a number of neurons. Each neuron is connected to a large number of other neurons through one way communication (either on the same SpiNNaker node or somewhere else in the network). Those connections are stored as pre and post-synaptic connections for every neuron. The pre-synaptic connections are the neurons that the current neuron can receive packages from, while the post-synaptic connections are the neurons that the current neuron can send to. The packages emitted through the network when a neuron spikes are sent without concern if the packages is received or not. If a package is sent and not received it is said to be lost, and nothing will be done about it. The Communications NoC is what makes it possible for two cores to be able to communicate with each other independent of their location in the SpiNNaker network (and thus, for two neurons to communicate). To be able to route the packages that are sent through the network there is a router embedded on the SpiNNaker chip. The router takes about 10% of the chip size and is connected to the Communications NoC, where it allows communication between ARM cores on different chips with a maximum speed of 1 Gbps. The router connects each SpiNNaker chip to six others in an network formed as a two-dimensional toroidal triangular mesh structure. Rather than writing all the spike destinations in each package, or sending one package for each destination neuron, there is no destination within the package, only its source is stored. This works since the router keeps a table on what neurons are connected to each source neuron. The router then sends the package to all nodes that contain one of these destination neurons. The routing information is stored on 1024 word routing tables that are distributed throughout the system. When a package is received, the router checks its routing table and decides where to send the package. One way to speed up the routing is through default routing. When a package is routed through default routing the package will be sent through the opposite port from the port that the package was received from. To make the routing more robust emergency routing can be used. Emergency routing tries to send a package on an alternative route if it fails to send a package on its assigned port. If the router is still not able to send the package then the package is dropped to prevent deadlock. Every package has an age, and when a package gets 4

5 Figure 5: The figure of the SpiNNaker chip design as seen above, shows how the ARM9 cores is connected to the System and Communications NoC. Image from A GALS Infrastructure for a Massively Parallel Multiprocessor, by Luis A. Plana[13]. too old it is dropped to prevent livelock (a lock where the system continues to run as usual, but the current state is not improved). An overview of how the communication to and from the ARM9 cores works is provided by figure 5 (on chip) and figure 2 (off chip). 3.3 Energy consumption In a system of this size the energy consumption is of huge interest. Furber states in[2]: The third key idea behind SpiNNaker [energy frugality] is that processors are free; the real cost of computing is energy. We have seen a continuing trend with cheaper and cheaper CPUs, along with higher and higher energy costs. If this trend continues in the same direction, this will eventually lead us to a situation where CPU cores will be almost free, while there will be a huge cost for running them. With this in mind we want to consider building a computer architecture that demands less power to do the same amount of work as today s high power CPUs. In the SpiNNaker project this is solved by using a lot of power effective ARM9 CPUs to do the same work as a high power Intel or AMD CPU. Another advantage with the ARM9 CPUs is that they can be put in sleep mode when they are not processing an incoming spike, and then be activated when they receive a package. This makes the ARM9 CPU even more power efficient. Since the SpiNNaker network is built on such a large scale with the final design goal containing over nodes even small changes towards energy efficiency will make a huge difference when you look at the whole network. Due to this, every aspect of the system is designed with energy Figure 6: Neuron potential over time. A spike is emitted from the neuron when the potential rises over the threshold potential. efficiency in mind, from low level communication protocols to efficient processors and electrical components. 4. SOFTWARE 4.1 Neuron model We have previously discussed different models for neurons and for overall structure of the neural network. As mentioned, the SpiNNaker system s main application will be a spiking neural network. The project team plans to use the Izhikevich model of a spiking neuron[6]. Izhikevich models the neuron from an electrical perspective, and the algorithm contains variables such as neuron membrane potential, membrane recovery (after a spike), and their respective derivatives. The resulting output from the neuron will be periodical signals, as the neuron will send spikes trough the network at a rate depending on the input it receives. Figure 6 shows how the neuron membrane potential changes over time in this neuron model. When a spike arrives, the potential rise slightly, but decays towards a resting potential over time. Once the potential exceeds a threshold value the neuron will emit a spike. For a short while after a spike has been emitted, it becomes harder for incoming spikes to affect the membrane potential. For a more in-depth look at the model, study Izhikevich s original paper[4]. 4.2 Spike-timing-dependent plasticity In the brain, neurons communicate with each other through synapses. A synapse is the connection through which signals or electrical charges are transferred between one neuron and another. From the synapses perspective the neuron that have sent the signal is said to be pre-synaptic, and the receiving neuron is called post-synaptic. When a neuron has reached its threshold value, it sends a signal through its synapses to the connected neurons. This signal is called a spike or electrical charge. The synapse can modify the potential of the 5

6 receiving neuron according to weights associated with the synapse. These synaptic weights can either be increased or decreased depending on the timing of the spikes, and they can be changed both by the pre-synaptic and post-synaptic neuron. A common algorithm for this is called Spike-timingdependent plasticity[5]. It is this algorithm that allow the behaviour of the neural network to change over time based on the most active neurons in short, it is how the network can learn. In the SpiNNaker system, the STDP learning algorithm only triggers on pre-synaptic spikes. The formula to calculate the change in the synaptic weight, F, dependent on the timing t between a pre-synaptic and post-synaptic spike is described below: A +e τ t + t < 0, F ( t) = A t e τ t 0. A pre-synaptic spike is defined as a spike arriving to a neuron, while a post-synaptic spike is defined as that neuron sending a spike. The factors A + and A are roughly equivalent to the learning rate in a standard neural network. The higher these values are, the more the synaptic weight will change each time STDP is triggered. τ + and τ are the sizes of the time windows in which learning can occur. In the theoretical case, this will decrease the impact of spikes with a long delay between them. In practise, the STDP algorithm will only consider spikes that are within the time windows when learning. However, since we only trigger STDP on pre-synaptic events (in practice, this means that the algorithm will have to be able to predict future spikes of the receiving neuron), we have to modify the STDP. Without the modifications proposed by Jin[8], performance of the algorithm will be reduced due to how synaptic weights are stored in the shared memory. Because of how the data is addressed, the algorithm is optimized either for pre-synaptic triggering or postsynaptic triggering, but not both at the same time. The problem is solved by using a deferred event-driven model of a pre-sensitive scheme. For further information on the event-driven model of a pre-sensitive scheme please read Jins article[8]. 4.3 General applications It is interesting to note that the SpiNNaker is merely a specialized hardware architecture, and not a neural network in itself. Any and all actual implementations of a network will model its neurons completely in software. This means that even though the main application of the SpiNNaker network will be a spiking neural network, it is far from impossible to implement other network structures on it. One such example is an algorithm described by Jin[7] a multi-layer back-propagation algorithm using perceptrons. (1) The project team has also found out that many other problems can be modelled using the SpiNNaker architecture, as long as they can be mapped to the system. Also, the principles of the SpiNNaker[2] bounded asynchrony, virtual topology and energy frugality can be applied to other problems, thus making them appropriate for the SpiNNaker system. 5. FUTURE WORK At the current stage in the SpiNNaker project, a prototype chip with two ARM9 cores has been developed and is in the testing phase. In addition to this, the full chip has been simulated using Field Programmable Gate Array (FPGA) technology. The project team plans to realize a 50 node network by years end, and after this create bigger and bigger networks, with 500, 5000 and finally nodes. The main focus of the project so far has been developing the hardware architecture of the SpiNNaker. This has brought forth innovations in many different areas, from routing to chip design. However, the team has yet to focus on further developing the software of the system. In an interview with the magazine New Electronics[15], professor Steven Furber the leader of the project commented: I don t think we have contributed anything to neural networks at this stage. However, I ll be disappointed if, in a year from now, that is still the case. 6. CONCLUSIONS The completion of the SpiNNaker project could well be the first step to implementing hard AI, and create a platform on which to further studies both in AI and in how the human brain works. Once this architecture has been tested, over time it could reasonably be scaled up even further, finally modelling 100 billion neurons. The SpiNNaker approach to large scale neural networks has many benefits when compared to other methods. Dedicated neural chips may be optimized for a specific scalable neural network, but once the design is fixed, it is hard to change the neuron model or the network architecture. FPGA and General Purpose Graphic Processing Units (GPGPU) solutions are flexible since the model can easily be changed in software. However, they are both limited in scale unless a network similar to the SpiNNaker is constructed. Several applications for the SpiNNaker are already planned, among them a continuation of a project at Manchester University. Psychologists have developed a neural network and taught it to read, with the intention of damaging parts of the network selectively and compare symptoms with brain damaged patients. Their current model is limited by its scale to contain only a small vocabulary, which could be extended by mapping the problem to the completed SpiNNaker system. Another interesting field of application for the SpiNNaker is the field of robotics. If an autonomous robot had its sensors and actuators connected to the SpiNNaker, many interesting tests could be performed. For example, how the system would adapt to a changing environment, and how the performance changes over time as the network learns new things. 6

7 Note that due to the physical size of the SpiNNaker network, in this case the robot could be a host connected to the SpiNNaker, for example through wireless communication. It should be noted that this report deals with a current research project in the middle of its life cycle, and that final results are yet to be presented. 7. REFERENCES [1] O. Boumbarov, S. Sokolov, and G. Gluhchev. Combined face recognition using wavelet packets and radial basis function neural network. In CompSysTech 07: Proceedings of the 2007 international conference on Computer systems and technologies, pages 1 7, New York, NY, USA, ACM. [2] S. Furber and A. Brown. Biologically-inspired massively-parallel architectures - computing beyond a million processors. In Proc. 9th International Conference on the Application of Concurrency to System Design, pages ACSD 09, [3] S. Haykin. Neural Networks: A Comprehensive Foundation. Prentice Hall PTR, Upper Saddle River, NJ, USA, [4] E. M. Izhikevich. Simple model of spiking neurons. IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 14, NO. 6, pages , [5] E. M. Izhikevich. Polychronization: Computation with spikes. In Neural Computation, 18, pages , [6] X. Jin, M. Luján, L. A. Plana, S. Davies, S. Temple, and S. B. Furber. Modeling spiking neural networks on spinnaker. IEEE Computing in Science and Engineering, September/October 2010 (vol. 12 no. 5), pages 91 97, [7] X. Jin, M. Luján, L. A. Plana, A. D. Rast, S. R. Welbourne, and S. B. Furber. Efficient parallel implementation of multilayer backpropagation networks on spinnaker. In CF 10: Proceedings of the 7th ACM international conference on Computing frontiers, pages 89 90, New York, NY, USA, ACM. [8] X. Jin, A. Rast, F. Galluppi, S. Davies, and S. Furber. Implementing spike-timing-dependent plasticity on spinnaker neuromorphic hardware. WCCI 2010 IEEE World Congress on Computational Intelligence, pages , July [9] M. M. L. and P. S. A. Perceptrons. MIT Press., Cambridge, [10] D. F. Leite, P. Costa, and F. Gomide. Evolving granular classification neural networks. In IJCNN 09: Proceedings of the 2009 international joint conference on Neural Networks, pages , Piscataway, NJ, USA, IEEE Press. [11] J. M. Nageswaran, N. Dutt, J. L. Krichmar, A. Nicolau, and A. Veidenbaum. Efficient simulation of large-scale spiking neural networks using cuda graphics processors. In IJCNN 09: Proceedings of the 2009 international joint conference on Neural Networks, pages , Piscataway, NJ, USA, IEEE Press. [12] C. D. Paternina-Arboleda, J. R. Montoya-Torres, and A. Fábregas-Ariza. Simulation-optimization using a reinforcement learning approach. In WSC 08: Proceedings of the 40th Conference on Winter Simulation, pages Winter Simulation Conference, [13] L. Plana, S. Furber, S. Temple, M. Khan, Y. Shi, J. Wu, and S. Yang. A gals infrastructure for a massively parallel multiprocessor. IEEE Design and Test of Computers, 24(5): , [14] F. Rosenblatt. The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Cornell Aeronautical Laboratory, [15] R. Rubenstein. Linking arms to make a brain. New Electronics, July 2010, pages 16 18, [16] H. Zhang, J. Guan, and G. C. Sun. Artificial neural network-based image pattern recognition. In ACM-SE 30: Proceedings of the 30th annual Southeast regional conference, pages , New York, NY, USA, ACM. 7