TD3 Tracking Performance Issue in a Nonlinear Control System

[sammanramzan2001-boop]

Hello,

I am working on a research project involving TD3-based trajectory tracking of a nonlinear Twin Rotor MIMO System (TRMS) in MATLAB.

I found the “Machine Learning for Motor Control” project and its suggested TD3-based PMSM control workflow. Although my system is different (TRMS rather than PMSM), I am using TD3 for continuous control and I am facing a tracking-performance issue.

For a step reference of 0.8 rad (pitch) and 0.9 rad (yaw), my TD3 agent eventually learns partial tracking, but the performance is inconsistent.

My latest results are:

Ep 400 | Reward = -5852.5 | StepErr = 0.7652
Ep 450 | Reward = -1785.0 | StepErr = 0.0472
Ep 500 | Reward = -1743.4 | StepErr = 0.1594
Ep 550 | Reward = -3019.5 | StepErr = 0.1932
Ep 600 | Reward = -1553.3 | StepErr = 0.0674

Final evaluation:

• Step RMSE: Pitch = 0.1200 rad, Yaw = 0.0784 rad
• Low-frequency sine RMSE: Pitch = 0.1502 rad, Yaw = 0.1045 rad
• High-frequency sine RMSE: Pitch = 0.1494 rad, Yaw = 0.1170 rad

Current TD3 settings:

• Gamma = 0.99
• Actor learning rate = 1e-3
• Critic learning rate = 1e-3
• Batch size = 256
• Tau = 0.005
• Policy noise = 0.2
• Noise clip = 0.5
• Action limit = ±2.5 V
• Observation dimension = 15

My reward function is:

R = -10(e_pitch² + e_yaw²) - 0.001(u1² + u2²)

The main question is: what would you recommend checking first in a TD3 implementation when the agent can sometimes achieve good tracking during training but does not maintain it consistently?

In particular, I would appreciate guidance on whether I should investigate the reward function, learning rates, exploration noise, observation design, or the TD3 implementation itself.

I can provide the relevant MATLAB code and additional results if needed.

Thank you.