Reinforcement Learning: How AI Learns Through Trial & Error

Part 2 of a series. If you missed Part 1: “Reinforcement Learning: Teaching Machines Through Experience.”

In Part 1, I introduced the fundamentals of reinforcement learning. Now I’m diving into the concepts that make it possible: Markov Decision Processes and the Bellman equation that help us find optimal policies.

Don’t worry about those technical-sounding terms — think of them as the “secret sauce” behind how AI learns to make smart decisions.

Disclaimer: This post simplifies complex mathematics to build intuition. The core principles are accurate, but real-world RL involves more advanced math. I’ve prioritized clarity over complexity.

When My Robot Vacuum Got Smarter

My robot vacuum demonstrated reinforcement learning perfectly: on day one, it bumped into everything and got stuck in corners. By week’s end, it glided efficiently around furniture, maximizing cleaning while minimizing battery use. This transformation — trying actions, receiving feedback, and improving — is reinforcement learning in action.

A Simple Game to Understand Complex AI

You control a character seeking treasure. Each move costs energy, some cells have obstacles, and you want the shortest path to the treasure.

In our game:

• Each step costs -1 (encouraging shorter paths)
• Reaching the treasure gives +100
• Hitting obstacles results in no movement (and still costs -1)
• Discount factor (γ): how much the agent values future vs. immediate rewards (between 0 and 1). We’ll use γ = 0.9.

Goal: find an optimal policy to reach the treasure.

This is what mathematicians call a Markov Decision Process — essentially the “rules of the game” describing how choices lead to outcomes and rewards.

How Does AI Learn the Best Path?

The key insight comes from the Bellman optimality equation, which calculates how “good” each position is:

Q(s,a) = R(s,a) + γ × max_a' Q*(s',a')
V(s)   = max_a [ R(s,a) + γ × V(s') ]

  V(s)   = value of the current state
  max_a  = the maximum value across all possible actions
  R(s,a) = the immediate reward for taking action a in state s
  γ      = the discount factor (0.9 in our example)
  V(s')  = the value of the next state after taking action a

In plain English:

• Value of position = best outcome from any move (up/down/left/right)
• Best outcome = immediate reward + (discounted value of the new position)

The γ is just a discount factor that makes immediate rewards more valuable than distant ones — much like how we’d rather have $100 today than $100 next year.

From Theory to Practice: A Step-by-Step Example

Let’s use a straight line with 5 positions. If we’re at S2 (the center) and already know the optimal state values (S0=40, S1=60, S3=80, S4=100), which way should we go?

• Going left: R(S2, ←) + γ × V(S1) = -1 + 0.9 × 60 = 53
• Going right: R(S2, →) + γ × V(S3) = -1 + 0.9 × 80 = 71

Since 71 > 53, it should go right — there’s more reward there. The AI learns this through repeated trials.

Important: the AI doesn’t immediately know these values. It starts with random guesses and progressively refines them through many iterations of the Bellman equation, gradually converging on the optimal values. This approach — iteratively updating state values until they converge — is called value iteration. Other methods include policy iteration and direct policy search.

The Goal: Finding the Optimal Policy

What we’re really after is the optimal policy (π*) — a strategy that tells us the best action to take in each position. In our treasure hunt, the policy is a map showing which direction to move at each step to reach the treasure by the shortest path. Once the AI learns the value of each position, extracting this policy becomes straightforward.

Looking Ahead: Real-World Challenges

Our grid game makes RL look straightforward, but real-world AI faces bigger challenges:

• Unknown environment dynamics: we rarely know exactly how actions will play out — we don’t know the rules of the game beforehand.
• Enormous state/action spaces: a lookup table works for a tiny grid but fails for millions or billions of states.
• Computational intensity: finding optimal policies becomes very resource-intensive as problems grow.

Conclusion

Reinforcement learning provides a powerful framework for decision-making through tools like MDPs and the Bellman equation, letting AI learn optimal behavior through direct experience. So far we’ve explored scenarios where we already know the “rules of the game.” In the next post: how RL agents can start completely clueless and still learn optimal strategies through pure trial and error.

---

Originally published on LinkedIn.