A reinforcement learning project for competitive dynamic pricing under stochastic, partially observable market dynamics. A Soft Actor-Critic (SAC) agent learns a continuous pricing policy while facing a reactive competitor, noisy demand, long-term market growth, and seasonality.
The project includes a custom Gymnasium environment, a modular SAC implementation in PyTorch, reproducible training and evaluation pipelines, checkpointing, experiment configuration through YAML, automated plots, tests, and an analysis notebook.
This project was jointly developed by:
Both authors collaboratively contributed to the design, implementation, and development of this project.
School of Electrical and Computer Engineering, University of Tehran — 2026
At each time step, the agent chooses a price for a single product. A competitor then sets its own price using an autoregressive reaction model, demand is realized, and the agent receives profit minus a penalty for changing its price too aggressively.
The environment is partially observable by default: the latent market baseline contains trend and seasonality, but explicit time features are disabled in the default configuration. Instead, the policy receives a rolling history of agent prices, competitor prices, and realized demand and must infer the underlying market dynamics from that history.
The learning problem combines:
- continuous control over price,
- a stochastic and reactive competitor,
- noisy price-sensitive demand,
- non-stationary market size with growth and seasonality,
- a price-adjustment cost that encourages smooth policies,
- partial observability through hidden temporal market factors.
The action is a single continuous price
Under the default configuration,
The policy observes a rolling history of length
With the default k = 10 and include_time_features: false, the observation dimension is 30.
If include_time_features is enabled, sine and cosine features for the seasonal phase are appended, increasing the observation dimension to 32 for the default history length.
The competitor follows a noisy autoregressive reaction model:
with the resulting price clipped to the configured competitor price range.
The default experiment uses m = 1, so the competitor reacts to the most recent agent and competitor prices.
The latent baseline market size combines slow logarithmic growth with periodic seasonality:
By default, this quantity is not directly observed by the agent.
Demand depends on the latent market size, the agent's price, the competitor's price, and Gaussian noise:
This captures both ordinary price sensitivity and a substitution effect: demand improves when the competitor becomes relatively more expensive.
The raw reward is profit minus an absolute price-adjustment cost:
A reward wrapper then scales this value before it is stored in replay and used for learning:
The default reward scale is 0.01.
The agent uses Soft Actor-Critic (SAC), an off-policy maximum-entropy actor-critic algorithm for continuous control.
The implementation includes:
- a squashed Gaussian actor,
- twin Q-critics,
- target critics with Polyak averaging,
- experience replay,
- automatic entropy-temperature tuning,
- reparameterized policy sampling,
- deterministic actions for evaluation.
The actor first samples an action in normalized [-1, 1] coordinates and then maps it to the environment's price range. Entropy is defined in these normalized action coordinates, so the standard target-entropy heuristic remains independent of the physical price units.
For a one-dimensional action space, the default target entropy is
The critic target is
Only true MDP termination is used in the bootstrap mask. Episode truncation at the configured horizon ends the rollout but does not force the bootstrap target to zero.
| Parameter | Default | Meaning |
|---|---|---|
horizon |
500 |
Steps per episode |
k |
10 |
Observation-history length |
m |
1 |
Competitor lag length |
p_min, p_max
|
1.0, 50.0
|
Agent price bounds |
unit_cost |
2.0 |
Per-unit cost |
adjustment_penalty |
0.1 |
Price-change penalty |
alpha0 |
50.0 |
Initial baseline market size |
alpha_growth |
5.0 |
Long-term growth strength |
alpha_seasonality |
0.1 |
Seasonal amplitude |
alpha_period |
52 |
Seasonal period |
beta |
2.0 |
Own-price demand sensitivity |
eta |
1.0 |
Competitor-price substitution effect |
demand_noise_std |
2.0 |
Demand-noise standard deviation |
competitor_intercept |
5.0 |
Competitor reaction intercept |
competitor_own_price_weight |
0.2 |
Response to the agent's lagged price |
competitor_lag_weight |
0.3 |
Competitor price persistence |
competitor_noise_std |
0.5 |
Competitor-noise standard deviation |
include_time_features |
false |
Hide explicit seasonal phase |
reward scale
|
0.01 |
Learning-reward scale |
| Parameter | Default |
|---|---|
Discount factor gamma |
0.99 |
Target update rate tau |
0.005 |
| Learning rate | 3e-4 |
| Hidden layers | [256, 256] |
| Batch size | 256 |
| Replay capacity | 1,000,000 |
| Initial entropy temperature | 0.2 |
| Automatic entropy tuning | enabled |
| Target entropy | -1 when unspecified |
| Device | automatic CUDA/CPU selection |
| Parameter | Default |
|---|---|
| Total environment steps | 1400 |
| Random-action warm-up | 200 |
| Update eligibility begins | step 100 |
| Update frequency | every step |
| Gradient updates per update step | 1 |
| Evaluation frequency | every 20 steps |
| Evaluation episodes | 5 |
| Checkpoint frequency | every 20 steps |
Because the default SAC batch size is 256, optimization cannot actually begin until the replay buffer contains at least 256 transitions, even though update_after is set to 100.
rl-project/
├── assets/
│ ├── training_plots.png # Training curves and SAC diagnostics
│ └── step_diagnostics.png # Learned pricing and demand rollout
├── configs/
│ ├── env.yaml # Market and reward configuration
│ ├── sac.yaml # SAC hyperparameters
│ └── experiment.yaml # Run, training, and output settings
├── src/
│ ├── agents/
│ │ ├── base_agent.py
│ │ ├── networks.py
│ │ └── sac_agent.py
│ ├── buffers/
│ │ └── replay_buffer.py
│ ├── environments/
│ │ ├── custom_env.py
│ │ ├── reward.py
│ │ └── wrappers.py
│ ├── evaluation/
│ │ ├── evaluator.py
│ │ ├── metrics.py
│ │ └── plots.py
│ ├── training/
│ │ ├── collector.py
│ │ ├── losses.py
│ │ └── trainer.py
│ ├── utils/
│ │ ├── checkpoint.py
│ │ ├── logger.py
│ │ └── seeding.py
│ ├── config.py
│ └── experiment.py
├── scripts/
│ ├── train.py
│ ├── evaluate.py
│ ├── sweep.py
│ └── visualize.py
├── tests/
├── notebooks/
│ └── analysis.ipynb
├── results/
│ ├── checkpoints/
│ ├── logs/
│ └── figures/
├── report.pdf
├── requirements.txt
└── pyproject.toml
Python 3.11+ is required by the project configuration.
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtOn Windows PowerShell:
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txtThe main dependencies are Gymnasium, NumPy, PyTorch, pandas, Matplotlib, PyYAML, and pytest.
All commands below should be run from the project root.
python scripts/train.pyThe CLI supports lightweight overrides without editing YAML files:
python scripts/train.py \
--seed 7 \
--run-name sac_pricing_seed7 \
--total-steps 5000For persistent experiment changes, edit:
configs/env.yamlfor the market and reward,configs/sac.yamlfor SAC hyperparameters,configs/experiment.yamlfor run and training settings.
python scripts/evaluate.py \
--checkpoint results/checkpoints/sac_pricing_seed0/agent_final.pt \
--episodes 10Evaluation uses deterministic actor actions and a fixed family of evaluation seeds so checkpoints can be compared under the same stochastic conditions.
python scripts/sweep.py --seeds 0 1 2 3 4Each seed is written to its own run directory.
python scripts/visualize.py --run-name sac_pricing_seed0This recreates the training and diagnostic-rollout figures from saved CSV logs.
jupyter notebook notebooks/analysis.ipynbThe notebook is intentionally analysis-only. Training remains in the CLI pipeline so experiments are easier to reproduce.
For a run named sac_pricing_seed0, the pipeline creates:
results/
├── checkpoints/
│ └── sac_pricing_seed0/
│ ├── agent_step_20.pt
│ ├── agent_step_40.pt
│ ├── ...
│ └── agent_final.pt
├── logs/
│ └── sac_pricing_seed0/
│ ├── training.csv
│ ├── rollout.csv
│ └── run_config.json
└── figures/
└── sac_pricing_seed0/
├── training.png
└── rollout.png
training.csv records periodic evaluation returns, profit, entropy temperature, critic loss, actor loss, and elapsed time. rollout.csv stores a full deterministic diagnostic trajectory containing agent price, competitor price, market size, demand, profit, adjustment cost, and scaled reward.
Checkpoints store network parameters, target critics, optimizer states, and the automatic entropy-temperature state. The current training pipeline does not serialize the replay buffer or expose a resume-training CLI, so checkpoints should be treated as complete agent snapshots rather than exact full-process restarts.
Run the test suite with:
pytest -qThe tests cover:
- environment API compliance and horizon truncation,
- action-bound enforcement,
- finite SAC update metrics,
- replay-buffer capacity and tensor shapes,
- checkpoint round trips,
- configuration loading.
The figures below summarize the 1,400-step experiment used for the project analysis. They show both the optimization behavior during SAC training and the learned pricing policy in a deterministic diagnostic rollout.
The evaluation return improves sharply once the replay buffer contains enough data for gradient updates, then settles near the best observed performance. At the same time, the automatically tuned entropy temperature decreases from roughly 0.20 to 0.144, gradually shifting the policy from stronger exploration toward more exploitative behavior. The critic loss also falls substantially from its early-training values.
After a short initial transient, the learned policy generally prices in the 10-13 range while adapting to a competitor that is typically around 9.5-11.5. The lower panel shows the non-stationary latent market baseline alongside realized demand: demand remains noisy, but its broad movement tracks the seasonal and growing market process that the policy must infer from observation history.
This is a controlled research simulator rather than a production pricing system. Important limitations include a linear demand model, one product, no inventory constraint, and a competitor with a fixed stochastic reaction rule.
Natural extensions include:
- recurrent policies such as LSTMs or GRUs for partial observability,
- explicit ablations with and without seasonal time features,
- multiple competitors,
- learning or strategic competitors and multi-agent RL,
- nonlinear or data-driven demand models,
- inventory and capacity constraints,
- distribution-shift and robustness experiments,
- broader hyperparameter and multi-seed evaluation.
The SAC implementation is based on the maximum-entropy actor-critic formulation introduced in:
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML 2018, arXiv:1801.01290.


