Reinforcement Learning in Finance: QTRAN for Portfolio Optimization

Abstract

This study introduces a QTRAN-based portfolio optimization algorithm to advance the use of reinforcement learning in financial investment. Traditional methods, such as the Mean-Variance Model and classical reinforcement learning algorithms (DQN, DDPG, PPO), face challenges in capturing complex asset interactions, balancing risk and return, and managing transaction costs. QTRAN, a value decomposition-based multi-agent reinforcement learning approach, addresses these limitations by effectively modeling nonlinear asset relationships and optimizing long-term returns. Experimental results demonstrate that QTRAN surpasses existing methods in key performance metrics, including the annualized return, Sharpe ratio, and maximum drawdown, while exhibiting strong adaptability across diverse asset classes and market conditions. Further analysis of transaction cost sensitivity and portfolio diversification highlights its robustness. This study confirms the potential of QTRAN for intelligent investment decision-making and suggests future research directions, such as its application in high-frequency trading and nonlinear risk management, to further expand its relevance in financial markets.