Máté Lengyel, Peter Dayan: Hippocampal contributions to control: the third way

Transcription

1 Máté Lengyel, Peter Dayan: Hippocampal contributions to control: the third way David Nagy journal club at

2

3

4 1 markov decision processes 2 model-based & model-free control 3 a third way

5 1 markov decision processes 2 model-based & model-free control 3 a third way

6 1 markov decision processes 2 model-based & model-free control 3 a third way

7 1 markov decision processes 2 model-based & model-free control 3 a third way

8 1 markov decision processes

9 markov process

10 markov process

11 state space markov process

12 state space markov process transition matrix

13 state space markov process transition matrix

14 state space markov process transition matrix markov property

15 markov process

16 markov reward process

17 markov reward process

18 reward function markov reward process

19 (discount factor) reward function markov reward process

20 (discount factor) reward function markov reward process return:

21 (discount factor) reward function markov reward process return: (state) value function:

22 markov reward process

23 markov decision process

24 markov decision process

25 actions markov decision process

26 actions markov decision process

27 actions markov decision process

28 actions markov decision process policy:

29

30

31

32

33 }MDP

34 }MDP + policy

35 }MDP + policy

36 }MDP + policy MP}

37 MP} }MDPMRP } + policy

38 MP} }MDPMRP } + policy

39 state-value function:

40 state-value function: action-value function:

41 solving an MDP

42 solving an MDP find the optimal policy for which expected return is maximal

43 solving an MDP

44 solving an MDP

45 solving an MDP best possible performance

46 1 markov decision processes 2 model-based & model-free control 3 a third way

47 2 model-based & model-free control

48 solving an MDP model-based model-free

49 solving an MDP model-based model-free all optimal policies achieve the same Q!

50 model-based solving an MDP model-free

51 model-based solving an MDP model-free try to learn MDP parameters from experience

52 model-based solving an MDP model-free try to learn MDP parameters from experience do forward search for choosing action

53 model-based solving an MDP model-free try to learn MDP parameters from experience estimate Q* from experience do forward search for choosing action

54 model-based solving an MDP model-free try to learn MDP parameters from experience estimate Q* from experience do forward search for choosing action choose action with highest Q value

55 model-based solving an MDP model-free

56 model-based solving an MDP model-free experience

57 model-based solving an MDP model-free experience model estimate

58 model-based solving an MDP model-free experience model estimate virtual experiences

59 model-based solving an MDP model-free experience model estimate virtual experiences Q-learning ^ Q

60 model-based solving an MDP model-free experience experience model estimate virtual experiences Q-learning ^ Q

61 model-based solving an MDP model-free experience experience model estimate virtual experiences Q-learning Q-learning ^ Q

62 model-based solving an MDP model-free experience experience model estimate virtual experiences Q-learning ^ Q Q-learning ^ Q

63 model-based solving an MDP model-free experience experience model estimate virtual experiences Q-learning Q-learning ^ Q equal in the limit of infinite xp ^ Q

64 model-based solving an MDP model-free experience } model estimate virtual experiences Q-learning ^ Q computationally intensive experience Q-learning ^ Q

65 amount of computation model-free model-based rate of convergence

66 experimental setup for differentiating between control systems

67

68

69

70 goal directed (prefrontal control)

71 goal directed (prefrontal control) habituation (dorsolateral striatal control)

72 habituation (dorsolateral striatal control) goal directed (prefrontal control) proximal action remains goal directed

73 1 markov decision processes 2 model-based & model-free control 3 a third way

74 3 a third way

75 model-based model-free

76 episodic model-based model-free

77 episodic

78 episodic

79 exploration s1

80 exploration s2 s1

81 exploration s3 s2 s1

82 exploration s4 s3 s2 s1

83 exploration s4 R! s3 s2 s1

84 store s-a chain in memory s4 R! s3 s2 s1

85 store s-a chain in memory then choose episode with best outcome R! s4 s3 s2 s1

86 computational noise parallel sampler tree MDP

87 computational noise

88 computational noise parallel sampler tree MDP

89 exploration & exploitation parallel sampler

90 exploration & exploitation parallel sampler sample each non-terminal (s,a) n times

91 exploration & exploitation parallel sampler sample each non-terminal (s,a) n times random walk from (s,a)

92 exploration & exploitation parallel sampler sample each non-terminal (s,a) n times random walk from (s,a)

93 computational noise parallel sampler tree MDP

94 tree MDP

95 performance experience

96 performance caching experience

97 performance model-based caching experience

98 performance model-based caching episodic experience

99 performance model-based caching episodic experience

100 performance model-based caching episodic experience

101 performance f(complexity, non-stationarity of environment, cost of exploration,...) model-based caching episodic experience

102 performance experience

103 +B,D,eta performance experience

104 performance experience

105 quickly changing env. performance experience

106 +exploration cost performance experience

107

108 increase in branching

109 sources