Reinforcement Learning (in Trading)

Reinforcement learning (RL) sits in an awkward position in algorithmic trading: theoretically very appealing, practically much harder than competing approaches.

**RL framework basics:** - Agent: the model making decisions - Environment: the market simulator (in training) or real market (in deployment) - State: a representation of current market conditions - Action: a discrete or continuous decision (long, short, hold, close; or 'set position size to X') - Reward: a scalar feedback signal (typically realised P&L per timestep, or risk-adjusted return) - Policy: the function mapping state to action probabilities - Value function: expected cumulative reward from a state under a given policy

The RL agent learns by interacting with the environment over many episodes, updating its policy to maximise expected cumulative reward.

**Common RL approaches for trading:**

1. **Q-learning / Deep Q Networks (DQN):** - Learn the action-value function Q(s, a) = expected return from taking action a in state s - Action chosen as argmax_a Q(s, a) - DQN uses a neural network to approximate Q for high-dimensional state spaces - Limitation: works well with discrete action spaces; less effective for continuous position sizing

2. **Policy Gradient methods:** - Directly learn a policy π(a|s) without explicit value function - REINFORCE, PPO (Proximal Policy Optimization), A2C, A3C are common variants - Handle continuous action spaces (continuous position sizing) - More sample-efficient than Q-learning in many trading applications

3. **Actor-Critic methods:** - Combine policy learning (actor) with value function learning (critic) - Critic provides baseline for variance reduction in policy updates - SAC (Soft Actor-Critic), DDPG (Deep Deterministic Policy Gradient) are common

**Why RL is challenging for trading:**

1. **Sample inefficiency**: RL typically needs millions of training episodes. For trading, each 'episode' is a market regime; we have only a few decades of relevant history. Compared to game-playing RL (where you can simulate millions of game episodes), trading RL is data-starved.

2. **Non-stationarity**: market dynamics change. RL assumes the environment's transition dynamics are stationary; in markets, the rules change. An agent trained on 2018-2020 may face fundamentally different dynamics in 2025.

3. **Noise in reward signal**: trading rewards (P&L) are extremely noisy at short timeframes. The signal-to-noise ratio for individual trade outcomes is low; learning generalisable behaviour from noisy rewards is difficult.

4. **Sparse rewards**: in some trading formulations, meaningful rewards only come at trade exit (large reward or large penalty), with mostly-zero rewards in between. Sparse rewards slow learning substantially.

5. **Risk-aware reward shaping**: naive RL maximises expected return; trading requires risk-adjusted optimisation. Reward shaping to penalise large drawdowns or Sharpe-ratio-based rewards add complexity.

6. **Simulation-to-reality gap**: training in simulation typically uses historical data, which lacks transaction costs, slippage, market impact, and non-stationarity. Live deployment encounters all of these, often invalidating the trained policy.

**Successful RL trading applications (limited):**

- Market making: RL agents that learn to quote bid/ask prices and manage inventory. Some institutional adoption. - Execution optimisation: RL for breaking large orders into smaller pieces with minimal market impact. Active research area at large hedge funds. - Portfolio rebalancing: RL for asset allocation decisions over long horizons. Some success at longer timeframes where market frictions matter less.

**Retail trading reality:** - RL is far harder to deploy successfully than supervised learning for retail forex strategies - Most retail 'AI EAs' claiming to use RL are either misusing terminology or using RL in very narrow ways - The training-data shortage and non-stationarity issues are fundamental, not solvable by more compute or better algorithms - Supervised learning (predicting next-bar direction from current features) is typically a much more practical approach for retail algorithmic trading

For FxRobotEasy: our flagship AI EAs use supervised learning rather than RL because the supervised approach is more reliable and easier to validate. RL is an active research area but not yet a foundation for production trading systems at retail scale.