Bellman Equation — Value Iteration Demo

Grid World — State Values V(s)

Iteration: 0 | Max ΔV: — | Not started

Bellman Update

V(s) ← max_a Σ P(s'|s,a) [R + γ·V(s')]

For each non-terminal state s:
  For each action a ∈ {↑,↓,←,→}:
    Q(s,a) = Σ P(s'|s,a)[R(s,a,s') + γ·V(s')]
  V(s) ← max_a Q(s,a)
  π(s) ← argmax_a Q(s,a)

γ (discount): 0.9 Goal reward: +1.0 Pit penalty: −1.0 Step cost: −0.04 Noise: 80% intended, 10% each side