NumPy. Daniël de Kok. May 4, 2017

Transcription

1 NumPy Daniël de Kok May 4, 2017

2 Introduction

3 Today Today s lecture is about the NumPy linear algebra library for Python. Today you will learn: How to create NumPy arrays, which store vectors, matrices, and higher order tensors.

4 Today Today s lecture is about the NumPy linear algebra library for Python. Today you will learn: How to create NumPy arrays, which store vectors, matrices, and higher order tensors. How to index into and modify NumPy arrays.

5 Today Today s lecture is about the NumPy linear algebra library for Python. Today you will learn: How to create NumPy arrays, which store vectors, matrices, and higher order tensors. How to index into and modify NumPy arrays. How to perform common linear algebra operations on NumPy arrays.

6 Today Today s lecture is about the NumPy linear algebra library for Python. Today you will learn: How to create NumPy arrays, which store vectors, matrices, and higher order tensors. How to index into and modify NumPy arrays. How to perform common linear algebra operations on NumPy arrays. How to reshape and transpose tensors.

7 Today Today s lecture is about the NumPy linear algebra library for Python. Today you will learn: How to create NumPy arrays, which store vectors, matrices, and higher order tensors. How to index into and modify NumPy arrays. How to perform common linear algebra operations on NumPy arrays. How to reshape and transpose tensors. How arrays are broadcast in NumPy.

8 What is NumPy? NumPy is a linear algebra library. Main NumPy data types: multidimensional arrays. Supports tensors arbitrary-rank tensors, including: rank-1 tensors (vectors) rank-2 tensors (matrices) Numpy uses C (or Fortran) arrays and BLAS operations.. Avoids the memory and computation overhead associated with Python lists/interpretation. Operator overloading for convenient notation.

9 Relation to Tensorflow Although we will use Tensorflow to construct computation graphs and for parameter estimation, mastering NumPy is important for three reasons: The input to a Tensorflow graph is typically provided as a NumPy array. The Tensorflow tensor API is strongly inspired by NumPy. Quick experimentation is usually easier with NumPy (Tensorflow constructs computation graphs that need to be executed).

10 Importing NumPy The NumPy package is called numpy. It is customary to import numpy as np: 1 import numpy as np From here onwards, these lecture notes assume that NumPy is imported in this manner.

11 Tensors So far, we have been dealing with: Vectors: R M Matrices: R M N

12 Tensors So far, we have been dealing with: Vectors: R M Matrices: R M N Vectors are 1st order tensors, matrices 2nd order tensors.

13 Tensors So far, we have been dealing with: Vectors: R M Matrices: R M N Vectors are 1st order tensors, matrices 2nd order tensors. Sometimes we need tensors of higher orders, e.g. a 3rd order tensor: R M N O

14 NumPy arrays

15 Constructing a new array A NumPy array can be constructed directly from a Python list using the array function: 1 np.array([1,2,3]) This returns a one-dimensional array (a vector), which has the following string representation: array([1, 2, 3])

16 Multi-dimensional lists The array function can also handle multi-dimensional Python lists. That is, list with embedded lists: 1 np.array([[1,2,3],[4,5,6]]) Each inner array in represents a row. NumPy uses a pretty printer to give the output as a formatted string: array([[1, 2, 3], [4, 5, 6]])

17 Array properties NumPy arrays have the following properties that we will frequently use: ndim The rank of the tensor. shape The shape of the tensor a tuple specifying the size of each tensor dimension. dtype The data type of tensor elements.

18 Array properties: example Here we see the values of these properties for an array that is constructed from a two-dimensional Python list: 1 m = np.array([[1,2,3],[4,5,6]]) 2 "ndim: %s, shape: %s, size: %s, dtype: %s" % \ 3 (m.ndim, m.shape, m.size, m.dtype) ndim: 2, shape: (2, 3), size: 6, dtype: int64

19 Data type inference NumPy bases the data type on the data type of the elements in the original array. If we used float literals instead, the array would have float64 as the data type: 1 np.array([1.0, 2.0, 3.0]).dtype dtype('float64')

20 Data types NumPy supports the typical data types one would expect from a linear algebra library: Data types int{8,16,32,64} uint{8,16,32,64} float{16,32,64} bool_ Description Signed integers of the given widths Unsigned integers of the given widths Half/single/double precision floating point Boolean values

21 Data type specification The data type for the elements can be specified using the dtype argument of array-creating functions. For example: 1 example = np.array([1,2,3], dtype=np.float32) 2 example.dtype dtype('float32')

22 Zero array It is common that you want to have an array consisting of only zeros. The np.zeros function creates such an array with the given data type: 1 np.zeros((3,2), dtype=np.float16) array([[ 0., 0.], [ 0., 0.], [ 0., 0.]], dtype=float16)

23 One array Similarly to zeros, there is a ones function: 1 np.ones((3,2), dtype=np.float16) array([[ 1., 1.], [ 1., 1.], [ 1., 1.]], dtype=float16)

24 One array Similarly to zeros, there is a ones function: 1 np.ones((3,2), dtype=np.float16) array([[ 1., 1.], [ 1., 1.], [ 1., 1.]], dtype=float16) More generally, np.full creates an array filled with the a given number: 1 np.full((3,2), 42, dtype=np.float16) array([[ 42., 42.], [ 42., 42.], [ 42., 42.]], dtype=float16)

25 Values from an interval np.arange creates a vector with values in the half-open interval [start, end): 1 np.arange(1, 4) array([1, 2, 3])

26 Values from an interval np.arange creates a vector with values in the half-open interval [start, end): 1 np.arange(1, 4) array([1, 2, 3]) Leaving the starting index of the interval, assumes 0: 1 np.arange(4) array([0, 1, 2, 3])

27 Values from an interval (2) Finally, arange also allows you to indicate a step, similarly to the regular Python range: 1 np.arange(0, 10, 2) array([0, 2, 4, 6, 8])

28 Random array Another interesting array initialization is the random array. np.random.random returns floating point random numbers from a uniform distribution [0.0, 1.0). For example: 1 np.random.random((3,2)) array([[ , ], [ , ], [ , ]])

29 Distributions The np.random module also supports drawing random numbers from other distributions, such as normal, poisson, and zipf distributions. The following code generates a matrix of random numbers from a normal distribution with mean 1.0 and standard deviation np.random.normal(loc=1.0, scale=0.1, size=(3,2)) array([[ , ], [ , ], [ , ]])

30 Indexing An array can be indexed to retrieve or assign a cell. Indexing is performed with the subscript operator ([]). In indexing, dimensions are separated by a comma. 1 a = np.array([[1,2,3],[4,5,6]]) 2 a[1,2] 6

31 Indexing (2) If the index does not specify an offset into each dimension, the tensor at the given index is returned: 1 a = np.array([[1,2,3],[4,5,6]]) 2 a[1] array([4, 5, 6])

32 Indexing (3) Consequently, we can also get the component that we are interested in by indexing on the returned tensor: 1 a = np.array([[1,2,3],[4,5,6]]) 2 a[1][2] 6

33 Slicing As with Python lists, arrays can be sliced by using the subscript operator in conjunction with colon (:): 1 a = np.arange(0,10) 2 a[2:8] array([2, 3, 4, 5, 6, 7])

34 Strides Adding an extra colon gives the possibility to specify a stride: 1 a = np.arange(0,10) 2 a[2:8:2] array([2, 4, 6])

35 Strides (2) Sometimes, it is also useful to only specify a stride: 1 a = np.arange(0,10) 2 a[::2] array([0, 2, 4, 6, 8])

36 In-class assignment: slicing Extract the following slices from np.arange(0, 20): array([10, 11, 12, 13, 14, 15]) array([1, 4, 7, 10, 13]) array([18, 14, 10]) Extract the following slices from np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]]): array([[1, 2, 3], [7, 8, 9]]) array([[1, 3], [4, 6], [7, 9], [10, 12]])

37 Array updates The indexing operator can also be used to assign a value to a cell: 1 a = np.arange(0, 6) 2 a[4] = 42 3 a array([ 0, 1, 2, 3, 42, 5])

38 Array updates (2) This also works for more complex indexing, such as indexing with strides. 1 a = np.arange(0, 6) 2 a[::2] = [42, 43, 44] 3 a array([42, 1, 43, 3, 44, 5])

39 Array updates (2) This also works for more complex indexing, such as indexing with strides. 1 a = np.arange(0, 6) 2 a[::2] = [42, 43, 44] 3 a array([42, 1, 43, 3, 44, 5]) Of course, you have to ensure that the shape of the right operand is compatible with the left operand.

40 Linear algebra on vectors

41 Warning NumPy does overloading on operators such as *, / and +. When using the standard NumPy array data type, these are elementwise. Be careful when you are familiar with Matlab, C++ linear algebra libraries, etc.

42 Dot product np.dot implements the dot product on vectors: 1 np.dot(np.array([1.,2.,3.]), np.array([4., 5., 6.])) 32.0

43 Dot product np.dot implements the dot product on vectors: 1 np.dot(np.array([1.,2.,3.]), np.array([4., 5., 6.])) 32.0 As we will see later, dot also provides matrix multiplication.

44 Erroneous dot product Erroneous dot products, such as the following: 1 np.dot(np.array([1.,2.]), np.array([4., 5., 6.])) will fail with an exception: 1 ValueError: shapes (2,) and (3,) not aligned: 2 (dim 0)!= 3 (dim 0)

45 PEP 465 PEP 465 proposes the infix for matrix multiplication. Supported by: Python 3.5 and later NumPy and later Documented as the np.matmul function.

46 PEP 465 PEP 465 proposes the infix for matrix multiplication. Supported by: Python 3.5 and later NumPy and later Documented as the np.matmul function. 1 np.arange(1, 4, \ 2 np.arange(4, 7, dtype=np.float32) 32.0

47 In-class assignment Implement a function l2norm that takes a vector as its argument and returns its l 2 norm. Implement a function euclid_dist that takes two vectors as its arguments and returns the euclidean distance between the vectors. Implement a function cosine_sim that takes two vectors as its arguments and returns the cosine similarity between the vectors (cos Θ). Note: you can use the math.sqrt or np.sqrt function for computing the square root. You can use the regular - operator for elementwise subtraction of vector components.

48 NumPy norms: l 2 np.linalg.norm function computes norms, the ord argument is used to specify which norm: 1 np.linalg.norm(np.arange(10), ord=2)

49 NumPy norms: l 1 1 np.linalg.norm(np.arange(10), ord=1) 45.0

50 Matrices

51 Reshaping A raw data array in memory can represent different matrices. For example, assume that we have the following array in row-major order: This array could represent a couple of different matrices: 1 [ [ ] ] ,, 3 4,

52 Reshaping (2) Reshaping changes the dimensions the data without changing its components.

53 Reshaping (2) Reshaping changes the dimensions the data without changing its components. For example: 1 np.arange(1,7).reshape((2,3)) array([[1, 2, 3], [4, 5, 6]])

54 Reshaping (3) A tensor can be reshaped into another tensor, if both tensors have the same number of components.

55 Reshaping (3) A tensor can be reshaped into another tensor, if both tensors have the same number of components. Two tensors have the same number of components if the multiplication of the dimension sizes are the same. For example, a vector with shape (8) can be reshaped into: (2,4) (4,2) (2,2,2) (1,8) But not into: (2,5) (2,2,2,2) (7)

56 Reshaping and copying reshape tries to return a view of the original array when possible: no copying necessary. Sometimes it is necessary to make a copy, e.g.: consider reshaping a transposed view of a tensor.

57 Inference of dimensions One of the dimensions of the shape can be set to have size -1. NumPy will then infer the dimension. 1 np.arange(1,7).reshape((2,-1)) array([[1, 2, 3], [4, 5, 6]])

58 Transposition The transpose() method of an array gives a transposed view (if possible) of a tensor. The T property of an array returns self when the number of array dimensions is less than 2 or the result of transpose otherwise. 1 np.arange(1,7).reshape((2,3)).t array([[1, 4], [2, 5], [3, 6]])

59 Tensor transposition How does transposition in a higher order tensor ndim > 2 work?

60 Tensor transposition How does transposition in a higher order tensor ndim > 2 work? The dimensions are reversed: A i,j,k becomes A k,j,i Example: 1 str(np.arange(0, 24).reshape((2, 3, 4)).T.shape) (4, 3, 2)

61 Broadcasting Consider the following elementwise sum: 1 a = np.array([[1, 1, 1], [2, 2, 2]]) 2 b = np.array([1, 2, 3]) 3 4 a + b array([[2, 3, 4], [3, 4, 5]])

62 Broadcasting Consider the following elementwise sum: 1 a = np.array([[1, 1, 1], [2, 2, 2]]) 2 b = np.array([1, 2, 3]) 3 4 a + b array([[2, 3, 4], [3, 4, 5]]) The shapes of the arrays are (2, 3) and (3). Why does this work?

63 Broadcasting Consider the following elementwise sum: 1 a = np.array([[1, 1, 1], [2, 2, 2]]) 2 b = np.array([1, 2, 3]) 3 4 a + b array([[2, 3, 4], [3, 4, 5]]) The shapes of the arrays are (2, 3) and (3). Why does this work? NumPy applies broadcasting.

64 Broadcasting (2) Broadcasting: when arrays have different sizes, the smaller array is broadcast accross the larger array so that they have equal sizes.

65 Broadcasting (2) Broadcasting: when arrays have different sizes, the smaller array is broadcast accross the larger array so that they have equal sizes. Constraints: NumPy compares the dimension sizes elementwise in reverse order. Two dimensions are compatible when: They have the same size. When one of the dimensions has size 1.

66 Broadcasting (2) Broadcasting: when arrays have different sizes, the smaller array is broadcast accross the larger array so that they have equal sizes. Constraints: NumPy compares the dimension sizes elementwise in reverse order. Two dimensions are compatible when: They have the same size. When one of the dimensions has size 1. The shape does not necessarily need the same dimensions. Consider our previous example with shapes (2, 3) and (3). (3) can be seen as (1, 3), making broadcasting possible.

67 In-class assignment: broadcasting Try to sum a tensor of shape (2, 1, 3) and a tensor of shape (2, 3, 1) elementwise. Explain the result.

68 Matrix multiplication The dot function operator that we have seen before can also be used for matrix multiplication: 1 a = np.arange(0, 6).reshape((2, 3)) 2 b = np.arange(0, 6).reshape((3, 2)) 3 4 b array([[10, 13], [28, 40]])

69 Matrix multiplication (2) If we use a regular vector, it is automatically converted to a matrix: 1 a = np.arange(0, 4).reshape((2, 2)) 2 b = np.array([1,2]) 3 4 b array([2, 8])

70 Matrix multiplication (2) If we use a regular vector, it is automatically converted to a matrix: 1 a = np.arange(0, 4).reshape((2, 2)) 2 b = np.array([1,2]) 3 4 b array([2, 8]) Beware: b!= a

71 Matrix multiplication (3) 1 a = np.arange(0, 4).reshape((2, 2)) 2 b = np.array([1,2]) 3 4 a array([4, 7])