Artificial Intelligence Introduction Handwriting Recognition Kadir Eren Unal ( ), Jakob Heyder ( )

Transcription

1 Structure: 1. Introduction 2. Problem 3. Neural network approach a. Architecture b. Phases of CNN c. Results 4. HTM approach a. Architecture b. Setup c. Results 5. Conclusion 1.) Introduction Artificial Intelligence Introduction Handwriting Recognition Kadir Eren Unal ( ), Jakob Heyder ( ) In the field of Artificial Intelligence, scientists have made many enhancements that helped for development of smart computers or devices. Image processing is one of them. One of the biggest challenges in it to identify documents in hand-written format. In this project uses Neural Network Modeling and HTM for identification of handwriting from optical images.inputs are two datasets collected by U.S. National Institute of Standards and technology. It contains labeled images of handwritten digits. Expected output is that the program reads the handwritten numbers and translates them ASCII-encoded numbers. The intention of this document is to give an introduction to two significantly different approaches for the problem of handwritten digit recognition: The traditional neural network approach and a biologically inspired hierarchical temporal memory approach. 2.) Problem Description In the given problem we try to interpret handwritten digits. The dataset is published freely and a common test for handwriting recognition with artificial neural networks. It was created by american census bureau employees and contains labeled images inclusive test images.our scope is only arabic digits 0-9 and challenged is maximum accuracy in this project. 3.) Neural network approach One of the type of artificial neuron called a perceptron were developed 50 s they takes several binary inputs and produces a single output. After that they use the complementary logic for human decision making illustration. Sigmoid neurons are develop version of perceptrons which is looking To see how learning might work, suppose we make a small

2 change in some weight (or bias) in the network. And this type of neurons obviously help us with handwriting recognition. a)architecture of neural networks Figure from google to layers of architecture of CNN In the architecture of network leftmost layer is Input layer and neurons are Input neurons.basicly rightmost is Output layers which contains output neurons. In our project we have one output. Middle of these layers network has hidden layer since it is not a mysterious part of network they are basically neither inputs nor outputs.the network has just a single hidden layer in above. b) Algorithm developed from Python Tensorflow library and use Deep neural networks and Convolutional Neural Networks contains code samples from book "Neural Networks and Deep Learning". For this part of theory detailed information can be found here: Mainly we use the components explained in the following:

3 MNIST Dataset a subset of a larger set NIST. that database has digits which is handwritten and divided into training examples. MNIST array is 28x28 values. In MNIST data available both Training and Testing images labels have first 2 column consists number of items in the file. CNN: Convolutional Neural Networks is type of feed forward Artificial neural network between neurons like animal visual cortex inspiration. Data input in CNN arrange like width and height and depth and that 3 element create a deep learning. CNN has 5 layers which are Input, convolutional layer, rectified linear unit, pooling layer and fully connected layer. Picture for showing CNN qualities b) Phases of CNN CNN works with 3 phase. In first phase is basicly input phase. Input MNIST data is take like array of 748-d of pixels and convert it matrixes of that pixels. After that Phase 2 starts and build network architecture like as I mentioned before. In that creation have 3 part such as Convolution Layer, ReLu function and pooling Layer. First Convolution Layer take 20 filters to go slide window 5 times 5 into 28x28 matrices and try to get pixels. After that ReLU function activate Back Propagation and convolutional layer reduces vanish gradient and avoid sparisty. Lastly Pooling Layer get ReLU function and activate #D tensors. Pool the all of the previous pixels a new matrix of smaller sizes. Lastly Phase 3 is Fully connected layer and that phase connect previous layers to nexts.

4 Schematic is basicly shows convolutional phases. Output is confusion matrix for model. We can add more number of layers but adding more might be affect accuracy. Because of the Multiple layers we called Deep learning system. c) Results and Accuracy of Deep-Neural Networks Four Layer Deep Neural Network using Tensorflow: 96.60% : Epoch 0 completed out of 10 loss: Epoch 1 completed out of 10 loss: Epoch 2 completed out of 10 loss: Epoch 3 completed out of 10 loss: Epoch 4 completed out of 10 loss: Epoch 5 completed out of 10 loss: Epoch 6 completed out of 10 loss: Epoch 7 completed out of 10 loss: Epoch 8 completed out of 10 loss: Epoch 9 completed out of 10 loss: Accuracy: We run CNN 3 times and we saw the when we train CNN more it learns more and accuracy is increasing. First run :Two Layer Convolutional Neural Network using Tensorflow: 97.10% : Epoch 0 completed out of 10 loss: Epoch 1 completed out of 10 loss: Epoch 2 completed out of 10 loss: Epoch 3 completed out of 10 loss: Epoch 4 completed out of 10 loss: Epoch 5 completed out of 10 loss: Epoch 6 completed out of 10 loss: Epoch 7 completed out of 10 loss: Epoch 8 completed out of 10 loss: Epoch 9 completed out of 10 loss: Accuracy: Second run: Two Layer Convolutional Neural Network using Tensorflow: 97.24% :

5 Epoch 0 completed out of 10 loss: Epoch 1 completed out of 10 loss: Epoch 2 completed out of 10 loss: Epoch 3 completed out of 10 loss: Epoch 4 completed out of 10 loss: Epoch 5 completed out of 10 loss: Epoch 6 completed out of 10 loss: Epoch 7 completed out of 10 loss: Epoch 8 completed out of 10 loss: Epoch 9 completed out of 10 loss: Accuracy: Third run:two Layer Convolutional Neural Network using Tensorflow: 97.29% : Epoch 0 completed out of 10 loss: Epoch 1 completed out of 10 loss: Epoch 2 completed out of 10 loss: Epoch 3 completed out of 10 loss: Epoch 4 completed out of 10 loss: Epoch 5 completed out of 10 loss: Epoch 6 completed out of 10 loss: Epoch 7 completed out of 10 loss: Epoch 8 completed out of 10 loss: Epoch 9 completed out of 10 loss: Accuracy: ) HTM-Approach a) The hierarchical temporal memory(htm) algorithm is a theory developed from Numenta ( ) to build biologically inspired intelligent systems. It is out of scope to discuss the whole theory here but we will try to cover parts that are relevant for the experiment. Nupic v1.3 is their most recent python implementation of the theories. It is open sourced and can be found here: Mainly we use the components explained in the following: SDRs : Sparse distributed representations are essentially bit arrays which try to cover the essential features of an input representation. An important property is that they are sparse, meaning only about 2% of the bits are active at the same time. This gives very good properties to e.g. detect similarities

6 using overlap scores even there is a lot of noise. Further information can be found: R.pdf where mathimatical properties are discussed in detail. Encoders : This component exists for different data types and tries to convert general data formats e.g. timestamp, scalar values, strings etc. into their SDR representation. There are certain properties that need to be preserved when converting to an SDR e.g. that similar semantic meanings have a large overlap (on bits are the same or close to each other) and that the sparsity is about the same percentage for different features of the data. It does not need to be sparse, but the percentage will represent the importance of a feature. They should also be deterministic meaning the same input always produces the same output. Spatial pooler : The spatial pooler is an array of mini-columns. More specifically for our experiments we usually use an array of size about 2048 mini-columns, each consisting of a bit-array of cells on its own. The features-vocabulary of the spatial pooler: Initially the mini-columns of the spatial pooler will have potential connections to a randomly assigned pool of bits of the input array. [Initial potential connections e.g. 85% can be adjusted] The bits marked yellow are potential connections of the selected column in the spatial pooler. (85% of input space) Each of this potential connections has a permanence value between 0-1 assigned. This are distributed around the connection threshold.

7 Heatmap of the permanence values of potential connections. White color marks input bits that are no potential connections. Red are potential connections which permanence values does not exceed the permanence threshold. Blue dots indicate actual connections. A connection threshold is set, each permanence value exceeding it will turn a potential connection to an actual connection. Inhibition-phase : Permanence values get incremented (reinforced) when the input cell of the connection is active and decremented if a connection is not active. This happens only to the permanence values of active mini-columns. Inactive columns will not be changed. This visualizes the learning process of the system. [Increment and Decrement values for synaptic permanence can be adjusted]

8 The connection history of a column. By comparing to the connections in the previous timestamp we can see permanence values changed which led to new potential connections crossing the threshold (newly connected synapses) and old ones disconnecting. Resulting the overlap score of a column changes and its rank compared to others to be selected as winning column. Overlap score of a mini-column shows how many connections are active for the specific input representation. Overlap threshold sets the # of mini-columns active. After ranking them based on their overlap score the defined number will be active. This columns are also called Winning columns

9 Winning columns are marked green. The overlap threshold is set to 30 columns. They are ranked on their overlap score to the input (formed connections). Boost-factor : A multiplier is applied to the overlap score to slightly boost the weaker scores and possibly shrinks higher ones. This leads to a more distributed (higher granularity) representation of the columns to represent all different features more accurately. This is needed because only active (winner) columns will learn and thus the other columns are inhibited and often do not have a chance to represent their insights of the input. b) In the experiment we will use the MNIST-dataset of handwritten digits. It can be found here: It is already preprocessed as they are all the same size and nicely centered. We will let the HTM system learn to recognize the digits and then use the other provided images of digits to test its accuracy of predictions. We will only use one randomized spatial pooler for the whole image as receptive field, this cuts of a lot of features of the HTM theory such as the temporal pooling, sequence memory, location-paired-information or a local receptive field. Due to this the archived results might be

10 in no sense comparable and nothing more than a beginning how to use the current Nupic 1.3 system for such a task. The main challenges were to setup a network with the right parameters and correctly encode the data into an SDR format. Something similar was done with previous Nupic versions, but all outdated and not compatible anymore. We setup a network with the following parameters: CPP SpatialPooler Parameters Number of Inputs = 1024 (32x32 images) Number of Columns = 4096 Number Active Columns = 240 (~17%) Potential Pooler Connections = And the rest left to mostly default values: globalinhibition = 1 localareadensity = -1 stimulusthreshold = 0 synpermactiveinc = 0 synperminactivedec = 0 synpermconnected = 0.2 minpctoverlapdutycycles = minpctactivedutycycles = dutycycleperiod = 1000 boost-strength = 1 wraparound = 1 CPP SP seed = This could be changed and we could do swarming to find better parameters ( ) but this would require time and computational power. Also the used parameters were already tested from earlier experiments from the Numenta team with other Nupic versions. The code is essentially doing the following steps: Setting up a new network with the given parameters. Adding the 3 needed regions: Sensor input (Image Data) Spatial pooler, connecting to input data region Classifier, to interpret the results (e.g. overlap scores/predictions/anomalies) Train the input - spatial pooler(sp) connections of the network Train the SP-classifier and input-classifier results of the network After this steps the network is ready to be tested on test-data and predictions compared to actual labels. This will be covered in the next part. c) Results:

11 After training the network with images we test it on a different set of hand written digit images. The resulting accuracy is 95.35%. 5.) Conclusion Both approaches reached a high accuracy and could be further improved significantly with fine-tuning. The current state of the art algorithms have a 99.9% accuracy and perform better than humans in recognizing digits. However it gets really interesting if we try to interpret whole alphabets or 2D structures (drawings etc.) and to recognize objects etc. This needs often to be done in context of the data, in the same way humans can infer information of a word often by having it in context of the sentence structure. ( Study ) It is therefore essential to learn in sequences and temporal, to interpret the data in a meaningful context. However we can see that traditional neural networks reach state of the art performances on specific tasks. Further experiments have to be conducted to have meaningful comparisons in this field. The intention of this document was to give an introduction into both approaches.