AI & Machine Learning Glossary — Pattern Recognition, Overfitting, Walk-Forward

Artificial intelligence and machine learning have transformed many fields; algorithmic trading is one of the harder applications because financial markets violate most of the assumptions ML relies on. This cluster collects the AI/ML concepts most relevant to algorithmic forex trading, with realistic framing about what ML can and cannot do in this domain.

The fundamental challenge is non-stationarity. ML works best when the underlying distribution is stable: train on past data, predict on similar future data. Financial markets are non-stationary — the rules change. A model trained on 2020 data may fail in 2026 because the regime is different. Walk-forward optimisation is the canonical defence: simulate periodic retraining to estimate live performance, not just in-sample fit.

Overfitting (or curve-fitting in trading terminology) is the defining failure mode. With enough complexity, any model can fit any historical pattern — including noise. Overfit models look brilliant on the data they were tuned on and fail on new data. Mitigation requires parameter parsimony, out-of-sample validation, and skepticism toward 'too good to be true' backtest results.

Supervised learning dominates trading ML applications because of its clear problem formulation: given current market features, predict the future outcome. Gradient-boosted trees (XGBoost, LightGBM) are the workhorse algorithms for tabular trading data. Neural networks are useful for sequence and image data but require more discipline to avoid overfitting.

Reinforcement learning is theoretically attractive (the agent learns optimal behaviour from market interaction) but practically challenging at retail scale. RL is sample-inefficient, the reward signal is noisy, and markets are non-stationary in ways that violate RL's assumptions. Limited live success vs supervised learning.

Ensemble models (combining multiple base models) reduce variance and overfitting. Random Forest, XGBoost (gradient boosting), and stacking ensembles are widely used. The diversity requirement is critical — combining many similar models gives no benefit.

Regime detection addresses non-stationarity by classifying current market state and adapting accordingly. A trend-following EA might disable itself during ranging regimes; a mean-reversion EA might do the opposite. Regime detection can be rules-based (ADX thresholds) or model-based (Hidden Markov Models, ML classifiers).

Pattern recognition spans rules-based detection (engulfing candles, order blocks) and ML-based detection (CNNs on chart images, LSTMs on price sequences). The rules-based approach is simpler and easier to validate; ML-based approaches can capture more subtle patterns but require more data and more validation rigour.

For commercial EAs marketed as 'AI-powered': read the technical documentation carefully. Many such products use AI/ML for parameter tuning while the trade-execution logic is deterministic — a sensible architecture but not what most buyers imagine when they hear 'AI'. The realistic expectation: AI/ML can improve parameter selection and regime detection by 10-30% over manual tuning; it does not provide oracular foresight.