Can a robot with no prior knowledge learn to make optimal decisions through experience alone? This series answers that question by following AIRO, a warehouse robot, as it learns everything from scratch, starting with which charging station charges fastest, progressing to navigating complex layouts, optimising inventory, and eventually coordinating multi-robot teams. The series follows the structure of Sutton & Barto's foundational text, Reinforcement Learning: An Introduction.
What is Reinforcement Learning?
Reinforcement Learning (RL) is a framework that enables intelligent systems to learn by interacting with an environment. Unlike supervised learning, which trains on labeled examples, an RL agent discovers behaviours that yield optimal outcomes through exploration and feedback.
Think of training a dog. You don't tell it exactly what to do in every situation. Instead, you provide feedback treats for good behaviour, corrections for bad — and let it figure out the optimal strategy through experience. RL agents work the same way.
The Core Components
1. Agent
The learner and decision-maker. In this series, that's AIRO, a warehouse robot that progresses through stages: selecting optimal charging stations, mastering navigation and inventory, and ultimately coordinating with other robots.
2. Environment
Everything the agent interacts with but doesn't control. The environment provides the current state, responds to agent actions, and delivers reward signals. It can be deterministic or stochastic.
3. States
Representations of the current situation. For AIRO this includes position and orientation, sensor readings, task progress, and existing constraints such as battery level and obstacle proximity.
4. Actions
Available choices for the agent, which can be:
- Discrete: turn left, move forward, pick up an object
- Continuous: precise movements or force application
- Complex: multi-step procedures or sequences
5. Rewards
Feedback signals that guide learning:
- Positive: goal achievement, efficient behaviour
- Negative: collisions, wasted energy, task failures
- Sparse: provided only at task completion
- Dense: provided at each time step
6. Return
The cumulative measure of success. For episodic tasks, this is the sum of rewards until completion. For continuing tasks, it's a discounted infinite sum — future rewards are worth slightly less than immediate ones.
The Agent-Environment Loop
At its core, RL follows this simple cycle:
Every RL interaction follows six steps:
- Observe: Agent perceives current state s
- Choose: Agent selects action a based on policy π(a|s)
- Execute: Environment receives the action and transitions
- Next State: Environment provides new state s'
- Reward: Environment delivers reward r for the transition
- Goal: Learn the policy that maximises expected return
Meet AIRO
AIRO is a next-generation warehouse robot designed for package delivery and inventory management. What defines it is that it learns from experience rather than following rigid programming.
AIRO's capabilities span three domains:
Mobility & Navigation: charging optimisation, warehouse pathfinding, and dynamic routing that adapts to changing conditions.
Manipulation & Tasks: efficient storage and retrieval, package handling, quality inspection and sorting.
Multi-Robot Coordination: teamwork with other AIRO units, task allocation, and conflict resolution over shared resources.
Series Structure
The series follows Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto:
- RL Fundamentals
- Multi-Armed Bandits
- Finite Markov Decision Processes
- Dynamic Programming
- Monte Carlo Methods
- Temporal-Difference Learning
- Function Approximation
- Deep Reinforcement Learning
- Multi-Agent RL
What's Next
The next post looks at AIRO's first real learning challenge: selecting the optimal charging station among three options with unknown speeds. This introduces the fundamental exploration vs. exploitation dilemma the tension between trying new options and sticking with what's already proven to work.
Algorithms covered: ε-greedy, Upper Confidence Bound (UCB), Thompson Sampling.