Reinforcement Learning for Multi-Echelon Supply Chain Optimization

The challenge

Classical inventory theory gives optimal policies under strict assumptions, but real supply chains have stochastic lead-times, capacity constraints, and non-stationary demand. The team needed to know whether modern RL could match theory where theory applies and beat it where it does not, without turning the research codebase into an unreproducible experiment graveyard.

Approach

• Implemented DQN, A2C and PPO agents in a unified multi-echelon simulation environment with shared state, reward and observation interfaces so every algorithm faced identical demand and lead-time distributions.
• Reproduced the Clark-Scarf fixed lead-time optimal policy and a Generalized-EBs heuristic as deterministic baselines, then measured optimality gap, service level and inventory cost under the same stochastic rollouts.
• Extended the environment to stochastic lead-times and a lead-time quote setting where the agent chooses whether to order before or after observing the quote, closing the gap between academic benchmarks and procurement reality.
• Automated experiment orchestration with AWS Batch for sweep jobs, Jenkins for CI tests on every commit, and Optuna + Ray Tune for distributed hyperparameter search across dozens of actor-critic configurations.
• Tracked runs, metrics and model checkpoints in a central registry so every reported result is linked to the exact code version, seed and compute environment.

Outcome

On fixed lead-times the best RL agent reached a 0.02% optimality gap to Clark-Scarf. On varying lead-times it beat the alternative approaches by 4.1%, and in the lead-time quote setting it outperformed Clark-Scarf by 8% and Generalized-EBs by 5.5%. The automation layer cut experiment turnaround from days to hours and became the template for later supply-chain RL work.