Reacher Control: Feedback Linearization vs Soft Actor-Critic 🎯

This project compares a model-based Feedback Linearization Controller (FLC) with a model-free Soft Actor-Critic (SAC) reinforcement learning agent for the Gymnasium Reacher-v5 environment.

The objective is to move the endpoint of a two-joint robotic manipulator to a randomly generated target, stop the motion and maintain a stable position.

Requirements

This project was developed and tested with Python 3.14.3.

All required Python packages are listed in requirements.txt.

Usage

1. Install dependencies

pip install -r requirements.txt

2. Select a controller

Open main.py and uncomment the controller that should be used.

For example:

# controller = RandomController(...)
# controller = PDController(...)
controller = FeedbackLinearizationController(...)
# controller = RLController(...)

Only one controller should be active at a time.

3. Run the simulation

python main.py

Extra: Train SAC model

A trained SAC model is already included in the repository and can be used directly by selecting the RL controller in main.py.

To train a new SAC model from scratch, run:

python train_rl.py 

Success criteria

An episode is considered successful when both conditions are satisfied:

$$ d < 0.002\ \text{m}, $$

$$ |\dot q| < 0.02\ \text{rad/s}, $$

for 20 consecutive simulation steps.

Where:

• $d$ is the distance between the fingertip and the target,
• $\dot{q}$ is the vector of joint angular velocities.

The manipulator must therefore:

1. reach the target,
2. reduce its joint velocity,
3. maintain the stable position for 20 steps.

Briefly passing through the target is not considered a success.

Dynamic model and Feedback Linearization Control

The two-joint manipulator is described by:

$$ M(q)\ddot q + C(q,\dot q)\dot q = \tau, $$

where:

• $q$ is the joint position vector,
• $\dot{q}$ is the joint velocity vector,
• $M(q)$ is the inertia matrix,
• $C(q,\dot{q})$ represents Coriolis and centrifugal effects,
• $\tau$ is the control torque vector.

The Cartesian target is converted into a desired joint configuration using analytical inverse kinematics.

A third-order polynomial trajectory generates the reference position, velocity and acceleration: $q_r(t)$, $\dot{q}_r(t)$ and $\ddot{q}_r(t)$.
The virtual acceleration is calculated as: $v=\ddot{q}_r+K_d(\dot{q}_r-\dot{q})+K_p(q_r-q)$.
The Feedback Linearization control law is: $\tau=M(q)v+C(q,\dot{q})\dot{q}$.
After ideal compensation of the nonlinear dynamics: $\ddot{q}=v$.
The resulting tracking-error dynamics are: $\ddot{e}+K_d\dot{e}+K_pe=0$.

The final controller parameters were:

kp = 225.0
kd = 30.0
trajectory_time = 0.25

The physical torque is converted into the MuJoCo actuator control input using the actuator gear ratio:

$$ a=\frac{\tau}{gear}. $$

Reinforcement learning method

The selected reinforcement learning algorithm is Soft Actor-Critic.

The SAC training configuration can be found in train_rl.py.

Reward function

The custom reward function is:

$$ r_t = -0.1 -5d_t +2e^{-100d_t^2} +5e^{-20000d_t^2} -0.5|\dot q_t| +10I_{\text{hold}} +2000I_{\text{success}}. $$

The reward contains:

Component	Purpose
$-0.1$	Encourages faster episode completion
$-5d_t$	Penalizes distance from the target
$2e^{-100d_t^2}$	Broad bonus for approaching the target
$5e^{-20000d_t^2}$	Precise positioning bonus
$-0.5|\dot{q}_t|$	Encourages the manipulator to slow down
$10I_{\text{hold}}$	Rewards maintaining a stable position
$2000I_{\text{success}}$	Final reward after satisfying the success condition

The standard Reacher-v5 reward is completely replaced by this custom reward.

Results

The final evaluation was performed over 1000 episodes.

Method	Success rate	Average number of steps
Feedback Linearization Control	99.5%	49.209
Soft Actor-Critic	97.2%	48.341

FLC achieved a success rate higher by 2.3 percentage points, while SAC completed successful episodes approximately 0.868 simulation steps faster on average.

Although FLC performed better in this experiment, SAC may be a more suitable choice for systems that are difficult to model accurately, contain complex nonlinearities or include unknown disturbances and constraints. In such cases, a model-free controller can learn the required behaviour directly from interaction with the environment without relying on an analytical dynamic model.

Feedback Linearization Control	Soft Actor-Critic

📜 License

This project is licensed under the MIT License. See the LICENSE file for details.