CSCI 252: Neural Networks and Graphical Models. Fall Term 2016 Prof. Levy. Architecture #6b: Vector Symbolic Architectures (Gayler 2003)

Size: px

Start display at page:

Download "CSCI 252: Neural Networks and Graphical Models. Fall Term 2016 Prof. Levy. Architecture #6b: Vector Symbolic Architectures (Gayler 2003)"

Jocelin Floyd
5 years ago
Views:

1 CSCI 252: Neural Networks and Graphical Models Fall Term 2016 Prof. Levy Architecture #6b: Vector Symbolic Architectures (Gayler 2003)

2 Tensor Binding: Inputs of size N give 2 us an output of size N

3 Solution #1: Circular Convolution

4 Holographic Reduced Representations (T. Plate 1995) HRR uses circular convolution to reduce the tensor product back to a vector of size N. Uses a similar principle to holography: recording a 3D image in a 2D medium

5 HRR: Notes conv_circ(conv_circ(a, b), c) Convolution is A B C computationally intense, but supports embedding / hierarchy straightforwardly A B C conv_circ(a, conv_circ(b, c))

6 Solution #2: Keep only the diagonal

7 Vector Symbolic Architectures (R. Gayler 2003) Gayler uses the term VSA to refer to tensorproduct architectures that reduce the tensor product (matrix) back to a vector. Hence HRR is a kind of VSA. Gayler favors the keep only the diagonal approach because of its computational simplicity. He calls this approach MultiplyAdd-Permute (MAP)

8 MAP Architecture: Multiply Binding implemented as simple Hadamard product (element-by-element multiply) : Self-inverse property: if we use -1,+1 as vector values, then every vector is its own inverse: A*(A*B) = (A*A)*B = B Contrast this with Tensor Products and HRR, where we have to use vector division (or multiply by computed vector inverse) to decode.

9 MAP Architecture: Add Recall Tensor Product Bundling operation: simple element-by-element addition. MAP uses the same trick: <Red>*<Square> + <Blue>*<Triangle> is a little database saying that we have a red square and a blue triangle, where <X> means the vector representing concept X. Combined with self-inverse, these two operations (Multiply, Add) provide interesting behaviors...

10 MAP Architecture: Queries Recall the Problem of Two: Big star, little star To make it even more interesting, let s add in a red planet So we have <S>tar, ig, <L>ittle, lanet, <R>ed: <S>* + <S>*<L> + *<R> Note that this is a single vector, of the same size as each of the vectors we put into it. Let s ask the question What is big?. The answer should of course be star

11 MAP Architecture: Queries To ask What is big?, we simply multiply our knowledge-base vector by the vector . The ordinary mathematical properties (commutative, associative, distributive) apply: * [<S>* + <S>*<L> + *<R>] *<S>* + *<S>*<L> + **<R> **<S> + *<S>*<L> + **<R> <S> + *<S>*<L> + **<R> <S> + noise So now we have a result vector that is the sum of <S> plus a lot of other junk, or noise. How do we deal with this?

12 MAP Architecture: Dealing with Noise <S> + *<S>*<L> + **<R> Despite the noise, the result vector will still have a much higher similarity (dot product) with <S> than with any other vector. So the likeliest answer to what is big? is star.

13 MAP Architecture: Cleanup Memory If we want a perfect restoration of our noisy result, we can use a Hopfield Net as an auto-associative cleanup memory (as we did in Assignment #3). Other, more biologically realistic, cleanup memories have also been implemented:

14 MAP: What s it Goodfer? MAP provides the ability to process analogies in a novel way. Kanerva (2010) asks: What is the dollar of Mexico?

15 Note that this process is holistic: at no point do we decompose the vectors to figure out what goes with what (contrast with discrete-math version) This process also shows how MAP solves the Variable Binding challenge, without the need for explicit variables.

16 MAP Architecture: Permute We ve seen the M(ultiply) and A(dd), but what about the P(ermute)? Permute is needed because of the self-inverse property. Consider the problem of encoding a sequence; ABCBAABCA the essence of behavior! Simple binding (A*B*C*B*A*A*B*C*A) will just give us a big mess when we try to query (unbind) Special positioning vectors are a possible solution: P *A + P *B + P *C + P *B + P *A + P *A + P *B Pretty bad solution, because need a new vector for each position (potentially infinite)

17 MAP Architecture: Permute Instead, we introduce a permute operator P(): a simple random shuffling of the elements of the vector. Then the same permute operator can be used to encode all elements of a sequence recursively: S = P(A + P(B + P(C + P(B + P(A + P(A + P(B + P(C +P(A))))))))) To recover the sequence, apply the inverse permutation P-1() successively: -1 P (S) = A + noise P-1(P-1(S)) = B + noise P (P (P (S))) = C + noise etc.

18 MAP Architecture: Permute Looks complicated, but corresponds to a straightforward recurrent network circuit: item sequence Encoder: P Decoder: cleanup -1 + P item sequence -

19 Embedding / Hierarchy via Permute P(P(A+B) + C) A B C P(A + P(B+C)) A B C

20 VSA: Additional Goodfers Levy, S.D., S. Bajracharya, and R.W. Gayler (2013) Learning Behavior Hierarchies via High-Dimensional Sensor Projection. In Learning Rich Representations from Low-Level Sensors: Papers from the 2013 AAAI Workshop.

21 VSA: Additional Goodfers

22 VSA: Additional Goodfers

CSCI 252: Neural Networks and Graphical Models. Fall Term 2016 Prof. Levy. Architecture #3: Sparse Distributed Memory (Kanerva 1988)

CSCI 252: Neural Networks and Graphical Models Fall Term 2016 Prof. Levy Architecture #3: Sparse Distributed Memory (Kanerva 1988) What we ve seen so far Neural nets use vector representations of data