Conversation
Agent-Logs-Url: https://github.com/jhare96/reinforcement-learning/sessions/30751571-3415-43e8-b2c8-0fe2f3cd47b8 Co-authored-by: jhare96 <33518590+jhare96@users.noreply.github.com>
…, package exports Agent-Logs-Url: https://github.com/jhare96/reinforcement-learning/sessions/30751571-3415-43e8-b2c8-0fe2f3cd47b8 Co-authored-by: jhare96 <33518590+jhare96@users.noreply.github.com>
Agent-Logs-Url: https://github.com/jhare96/reinforcement-learning/sessions/5483f1bb-044e-4da1-93ec-41981b01934e Co-authored-by: jhare96 <33518590+jhare96@users.noreply.github.com>
… Python references to 3.10+ Agent-Logs-Url: https://github.com/jhare96/reinforcement-learning/sessions/5483f1bb-044e-4da1-93ec-41981b01934e Co-authored-by: jhare96 <33518590+jhare96@users.noreply.github.com>
Agent-Logs-Url: https://github.com/jhare96/reinforcement-learning/sessions/fae83cc0-9651-42a7-acf2-db80b3e50ef2 Co-authored-by: jhare96 <33518590+jhare96@users.noreply.github.com>
CI/CD
- New GitHub Actions workflows: ci.yml (lint + typecheck + pytest matrix
+ sdist/wheel build), docker.yml (build the existing Dockerfile),
docs.yml (build mkdocs and deploy to GitHub Pages on master).
Tests
- Added tests/ pytest suite covering rlib package metadata, the pure
numpy helpers in rlib.utils.utils, the polynomial scheduler, the
rlib.envs canonical API + registry, DummyBatchEnv, and the
lightweight env wrappers (53 tests, all passing).
Model base class hierarchy (rlib/networks/base.py)
- New abstract Model(torch.nn.Module, ABC) base providing:
* shared lr / lr_final / decay_steps / grad_clip / device state
* _build_optimiser() helper (optimiser + polynomial scheduler)
* _train_step(loss) helper (backward + clip + step + zero + schedule)
* value_loss() static helper (algorithm-agnostic 1/2 MSE)
* abstract evaluate() and backprop() — concrete signatures live on
each subclass since they're agent-specific
- New A2CModel(Model) intermediate that owns the A2C/A3C actor-critic
+ entropy loss, inherited by ActorCritic, ActorCritic_LSTM, and
Unreal_ActorCritic_LSTM.
- New PPOModel(Model) intermediate that owns the PPO clipped-policy
loss, inherited by PPO, PPOIntrinsic (RND), and DAAC.PolicyModel.
- Every trainable model in the library (ActorCritic{,_LSTM}, PPO,
PPOIntrinsic, RND, RANDAL, Curiosity, DAAC + Value/PolicyModel,
DQN, UnrealA2C{,2}, Unreal_ActorCritic_LSTM, VINCNN) now inherits
from Model and uses _build_optimiser/_train_step.
SyncMultiEnvTrainer
- model parameter typed as Model (was torch.nn.Module).
- _train_onestep / update_target / get_action / rollout converted
from @AbstractMethod to runtime NotImplementedError stubs so
trainers that only need _train_nstep don't need to redefine all
four to be instantiable.
Lint sweep (ruff)
- Removed all per-file ignore lists from pyproject.toml.
- Replaced every 'from rlib.* import *' with explicit imports
(eliminates ~169 F403/F405 errors).
- Auto-fixed unused imports/variables, mutable default args, ambiguous
variable names (l -> loss_value), trailing whitespace, printf-style
formatting, redundant collection calls, true/false comparisons,
ranges with unused loop vars, etc.
- Manually fixed the few non-auto-fixable issues (assert False, file
context managers, kernel_size=[a,b] -> tuple, pep604 union types).
- Whole repo is now ruff-clean and ruff-format-clean.
Mypy
- Dropped per-module strict overrides in favour of one consistent
project-wide config (check_untyped_defs=true,
disallow_untyped_defs=false). The remaining mypy errors are
concentrated in legacy 'def main()' demo functions inside agent
modules, which have been superseded by examples/.
Configs - New ModelConfig/A2CConfig/PPOConfig dataclasses in rlib/networks/model_config.py: collect lr, lr_final, decay_steps, grad_clip, device + family-specific fields (entropy_coeff, value_coeff for A2C; entropy_coeff, policy_clip for PPO) into immutable, asdict-able value objects. - New TrainerConfig dataclass in rlib/utils/trainer_config.py: collects all 17 SyncMultiEnvTrainer kwargs (train_mode, return_type, gamma, lambda_, total_steps, validate_freq, log_dir, ...). Validates train_mode and return_type via Literal types and __post_init__. Models - Model, A2CModel, PPOModel: __init__ now takes only config (legacy kwargs removed). Mirrors config fields onto self.lr/self.device etc. for ergonomics. - All 13 concrete model classes updated: ActorCritic, ActorCritic_LSTM, PPO, PPOIntrinsic, RND, RANDAL, Curiosity, DAAC, ValueModel, PolicyModel, DQN, UnrealA2C, UnrealA2C2, Unreal_ActorCritic_LSTM, VINCNN. Each now takes (... model-specific args ..., config, *, build_optimiser, optim, optim_args, **model_args) instead of duplicating the LR/coeff plumbing. Trainer - SyncMultiEnvTrainer.__init__ now takes only (envs, model, val_envs, config). All 10 concrete trainers updated: A2C, A2CLSTM_Trainer, PPOTrainer, RNDTrainer, RANDALTrainer, DAACTrainer, SyncDDQN, Curiosity_Trainer, UnrealTrainer (x2). Each declares only its agent-specific kwargs (num_epochs, num_minibatches, gamma_intr, ...) and forwards config to super. Examples - examples/cartpole_a2c.py, examples/atari_ppo.py, examples/montezuma_rnd.py: updated to use the config dataclasses. Tests - 21 new tests in tests/test_configs.py covering field defaults, immutability, asdict round-trip, validation errors, and Model integration. Total suite: 53 -> 74 tests, all passing.
Layout
- Every rlib/<Agent>/ now has:
model.py - the Model subclasses
trainer.py - the SyncMultiEnvTrainer subclass(es)
__init__.py - re-exports
And for the documented agents (A2C, PPO, RND), a __main__.py that
invokes the corresponding examples/ script via runpy, so
'python -m rlib.A2C', 'python -m rlib.PPO', 'python -m rlib.RND'
still work.
- Used 'git mv' for the larger half of each old file so rename
detection keeps the diff legible.
Renames
- rlib/A2C/A2C.py -> rlib/A2C/trainer.py (merged with A2C_lstm.py)
- rlib/A2C/A2C_lstm.py -> deleted (merged into trainer.py)
- rlib/A2C/ActorCritic.py -> rlib/A2C/model.py
- rlib/PPO/PPO.py -> rlib/PPO/trainer.py (model split out)
- rlib/RND/RND.py -> rlib/RND/model.py (trainer split out)
- rlib/RANDAL/RANDAL.py -> rlib/RANDAL/trainer.py (model split out)
- rlib/Curiosity/CuriosityA2C.py -> rlib/Curiosity/model.py (trainer split out)
- rlib/DAAC/DAAC.py -> rlib/DAAC/model.py (trainer split out)
- rlib/DDQN/SyncDQN.py -> rlib/DDQN/trainer.py (model split out)
- rlib/Unreal/UnrealA2C.py -> rlib/Unreal/lstm.py (the recurrent variant)
- rlib/Unreal/UnrealA2C2.py -> rlib/Unreal/feedforward.py
- rlib/VIN/VIN.py -> rlib/VIN/trainer.py (model split out)
Class renames
- rlib.A2C.A2C -> A2CTrainer (consistent *Trainer suffix)
- rlib.A2C.A2CLSTM_Trainer -> A2CLSTMTrainer
- rlib.Curiosity.Curiosity_Trainer -> CuriosityTrainer
- rlib.Unreal.UnrealA2C.UnrealTrainer -> UnrealLSTMTrainer (was clashing
with feedforward UnrealTrainer; the feedforward one keeps the name)
Trainers
- All trainers now type the model parameter directly via
'__init__(self, ..., model: <ConcreteModel>, ...)' instead of the
ugly class-level 'model: <ConcreteModel>' annotation.
Demo blocks
- Dropped every 'def main(env_id):' + '__main__' guard from agent
files (~1500 lines of dead, untested code superseded by examples/).
- The four documented agents (A2C, PPO, RND) get a one-line
__main__.py delegating to the canonical example so
'python -m rlib.<Agent>' still works.
Imports
- Updated rlib/Unreal/lstm.py and rlib/Unreal/feedforward.py to import
from the new rlib.A2C.model path.
- All __init__.py re-exports updated.
- examples/cartpole_a2c.py updated to use A2CTrainer.
Tests: 74 passing, ruff + format clean.
Configs
- New per-trainer dataclass subclasses of TrainerConfig in
rlib/utils/trainer_config.py:
PPOTrainerConfig (num_epochs, num_minibatches)
RNDTrainerConfig (gamma_intr, init_obs_steps,
num_epochs, num_minibatches)
RANDALTrainerConfig (extends RNDTrainerConfig: replay_length,
norm_pixel_reward)
DAACTrainerConfig (policy_epochs, value_epochs, num_minibatches)
DDQNTrainerConfig (epsilon_start/final/steps/test)
UnrealTrainerConfig (normalise_obs, replay_length)
All re-exported from rlib.utils.
Trainers
- The 6 trainers above now take their specific config subclass and
pull every extra hyperparameter from it. Their ctors collapse from
~10 kwargs to 4 (envs, model, val_envs, config).
- The 4 pass-through trainers (A2CTrainer, A2CLSTMTrainer,
CuriosityTrainer, UnrealLSTMTrainer) keep TrainerConfig.
Auto-logged hyperparameters
- New SyncMultiEnvTrainer._log_hyperparameters hook writes
<log_dir>/hyperparameters.txt at construction time using
dataclasses.asdict(config) | dataclasses.asdict(model.config),
prefixing model fields with 'model.'.
- Every trainer's hand-rolled hyper_paras dict (~15 lines x 8) gone.
Adding a new field to any *Config now shows up in the log
automatically.
Examples
- examples/atari_ppo.py uses PPOTrainerConfig
- examples/montezuma_rnd.py uses RNDTrainerConfig
Tests
- 14 new tests in tests/test_configs.py covering subclass inheritance,
field defaults, asdict round-trip, and an end-to-end check that a
real trainer construction writes the expected hyperparameters file.
- Total suite: 74 -> 88 tests, all passing.
Each *TrainerConfig subclass moves from rlib/utils/trainer_config.py to the rlib/<Agent>/trainer.py file that owns the corresponding trainer. The base TrainerConfig (and TrainMode/ReturnType aliases) stay in rlib.utils because they're algorithm-agnostic. Rationale: each *TrainerConfig is 1-to-1 with a single trainer class (unlike *ModelConfig which is shared across multiple models — A2CConfig is used by ActorCritic, ActorCritic_LSTM, Unreal_ActorCritic_LSTM, Curiosity), so co-location maximises locality of reasoning. Changes - Moved PPOTrainerConfig -> rlib/PPO/trainer.py - Moved RNDTrainerConfig -> rlib/RND/trainer.py - Moved RANDALTrainerConfig -> rlib/RANDAL/trainer.py (now imports RNDTrainerConfig from rlib.RND.trainer) - Moved DAACTrainerConfig -> rlib/DAAC/trainer.py - Moved DDQNTrainerConfig -> rlib/DDQN/trainer.py - Moved UnrealTrainerConfig -> rlib/Unreal/feedforward.py - Each rlib/<Agent>/__init__.py re-exports its config so users do 'from rlib.PPO import PPO, PPOTrainer, PPOTrainerConfig'. - rlib.utils now only re-exports TrainerConfig + Literal aliases. - examples/ and tests/ updated to use the new import paths. Tests: 88 passing, ruff + format clean. Net -34 LOC.
Layout
- New rlib/training/ subpackage holding everything related to the
training loop:
config.py - TrainerConfig + Literal aliases (was utils/trainer_config.py)
trainer.py - SyncMultiEnvTrainer (was utils/SyncMultiEnvTrainer.py)
returns.py - nstep_return / lambda_return / GAE + RETURN_FUNCTIONS
dispatch table (extracted from trainer + utils.utils)
validation.py - Validator Protocol + AsyncValidator + SyncValidator
+ make_validator factory
__init__.py - public re-exports
- rlib/utils/ now only holds genuine utility helpers (vec env runners,
env wrappers, schedulers, replay memory, numpy helpers).
- Lazy-loaded as rlib.training in rlib/__init__.py.
Trainer
- SyncMultiEnvTrainer.__init__ now stores self.validator (built
via make_validator from the val_envs argument) instead of the
bug-prone self.validate_func instance-method binding.
- The if/elif return_type cascade collapses to a single dispatch
through RETURN_FUNCTIONS[self.config.return_type].
- New _validation_score(render) hook lets recurrent agents (A2C-LSTM,
UNREAL-LSTM) override the validation flow without forking the whole
validation_summary method.
- Removed duplicate methods on the trainer: nstep_return / lambda_return
/ GAE / validate_async / _validate_async / validate_sync / fold_batch.
- Net trainer file shrinks ~470 -> ~310 lines.
Imports
- Every agent's trainer.py now imports from rlib.training instead of
rlib.utils.SyncMultiEnvTrainer / rlib.utils.trainer_config.
- examples/cartpole_a2c.py updated.
- rlib.utils.utils re-exports nstep_return/lambda_return/GAE from
rlib.training.returns for backwards compatibility (used by A3C
and the test suite).
Tests
- 11 new tests in tests/test_training.py covering Validator strategy
dispatch, both AsyncValidator and SyncValidator end-to-end, and the
RETURN_FUNCTIONS dispatch table.
- Test suite: 88 -> 99 passing.
Both files structurally belong in the env subpackage: * rlib/utils/VecEnv.py -> rlib/envs/vec_env.py (BatchEnv, DummyBatchEnv, ChunkEnv all subclass rlib.envs.RLVecEnv) * rlib/utils/wrappers.py -> rlib/envs/wrappers.py (every wrapper subclasses rlib.envs.RLEnvBase) Changes - git mv'd both files (filenames lower-cased to match envs/ convention). - Re-exported BatchEnv + DummyBatchEnv from rlib/envs/__init__.py alongside the existing RLEnv / RLEnvBase / RLVecEnv / make / wrap surface, so 'from rlib.envs import DummyBatchEnv' works. - Internal imports in vec_env.py and wrappers.py reach into rlib.envs.base / rlib.envs.registry directly rather than via rlib.envs (which would re-import them from the same package). - Updated 13 importers across rlib/, examples/, tests/ via sed. rlib/utils/ now contains only genuine utility helpers (numpy / torch helpers, schedulers, replay memory, play / random_agent / test scripts). Tests: 99 passing, ruff + format clean.
One script per (agent, env-class) pair from the dissertation
'Dealing with Sparse Rewards in Reinforcement Learning' (Hare 2019).
Hyperparameters are taken directly from the LaTeX source's tables
(tbl:A2C Atari, tbl:DDQN Atari, tbl:PPO Atari, tbl:RND Atari,
tbl:UNREAL Atari, tbl:RANDAL Atari, Control_PPO).
Layout
- examples/paper/common.py — shared MLP body, env factories
(CLASSIC_ENVS, ATARI_ENVS, NUM_WORKERS),
NatureCNN re-export.
- examples/paper/atari_<a>.py — A2C / DDQN / PPO / RND / UNREAL / RANDAL
on the three Atari benchmarks.
- examples/paper/classic_<a>.py — A2C / DDQN / PPO / RND / RANDAL on
Acrobot / CartPole / MountainCar.
(UNREAL skipped — pixel control N/A
for low-dim inputs, per paper.)
Each script:
* takes an env id as CLI arg (or 'all' to loop over the agent's envs);
* writes logs to logs/paper/<Agent>/<env_id>/;
* uses the per-trainer config subclass (PPOTrainerConfig,
RNDTrainerConfig, etc.) so dataclasses.asdict() captures the full
hyperparameter set in hyperparameters.txt automatically.
README.md gets a one-line pointer to the new directory.
Tests: 99 passing (no behaviour change), ruff clean.
- New rlib.training.returns.Returns enum: NSTEP / GAE / LAMBDA members wrap the underlying free functions via enum.member, so members are directly callable. The trainer's main loop now dispatches via self.config.returns(...) instead of looking up RETURN_FUNCTIONS by string key. - TrainMode is now a str-Enum (NSTEP / ONESTEP) rather than a Literal alias; the enum constructor takes care of the validation that __post_init__ used to do, so __post_init__ goes away. - Per-trainer subclasses (A2C, PPO, RND, RANDAL, Curiosity, DAAC, Unreal/lstm, VIN) drop their stale 'self.GAE' / 'self.nstep_return' / 'self.lambda_return' calls (which referenced methods that didn't exist anywhere on the trainer base class — latent bug) in favour of direct calls to the GAE / nstep_return / lambda_return free functions imported from rlib.training.returns. - Paper recipes migrated from return_type="GAE" to returns=Returns.GAE. - Tests rewritten against the enum API; 100/100 passing.
The Protocol form was redundant -- every type annotation in the
codebase already uses the ABC, so the structural-typing surface had
zero callers. Collapsing the two classes into one removes ~25 lines
of indirection and a Protocol/runtime_checkable import.
After this commit the env contract is just:
class RLEnv(ABC): # was RLEnvBase
# reset/step/close/observation_space/action_space + helpful
# __getattr__ delegation, unwrapped, context-manager defaults.
class RLVecEnv(ABC): # unchanged
# vectorised runners; merge_done/merge_info helpers.
- rlib/envs/registry.py and rlib/envs/adapters/ deleted entirely. The
whole 240-line backend-dispatch system existed only to support the
legacy gym package alongside Gymnasium; with that gone, the abstract
GymnasiumAdapter was a 44-line no-op pass-through. Gymnasium envs go
straight through the rlib wrappers and vec runners now.
- ApplePicker / ApplePickerDeterministic ported to the gymnasium API:
- subclasses gymnasium.Env (via `import gymnasium as gym`)
- reset(seed=None, options=None) -> (obs, info)
- step() returns the 5-tuple (obs, reward, terminated, truncated, info)
- explicit observation_space (was missing)
- Both registered with Gymnasium at import time so
`gymnasium.make("ApplePicker-v0")` works out of the box.
- Dropped the dead matplotlib + scipy.misc.imresize __main__ demo.
- rlib/envs/__init__.py collapsed to ~36 lines: re-exports gymnasium.make
directly, registers the two ApplePicker variants.
- rlib/envs/vec_env.py uses gymnasium.make directly (was rlib.envs.registry.make).
- rlib/envs/wrappers.py drops the _ensure_rlenv coercion helper; wrappers
trust they are getting a 5-tuple env.
- rlib/training/validation.py, rlib/utils/random_agent.py, rlib/utils/play.py
drop the wrap() indirection.
- tests/test_envs.py rewritten to test the new minimal surface (make
re-export, ApplePicker registration, RLVecEnv helpers, RLEnv ABC).
98/98 tests pass (was 100; the two registry-specific tests are gone
with the registry).
- rlib/networks/base.py -> rlib/agent.py (with class Model -> Agent).
The class is the policy + network bundle, not a generic torch
Module. Renaming clarifies what subclasses actually represent.
- rlib/networks/networks.py -> rlib/models.py (CNN / MLP / masked-RNN
building blocks). Top-level for discoverability.
- rlib/networks/model_config.py inlined: ModelConfig moves to
rlib/agent.py (same place as the Agent class that uses it);
A2CConfig and PPOConfig move to rlib/A2C/model.py and
rlib/PPO/model.py respectively, co-located with the agent classes
they configure (matches the pattern already used for per-trainer
configs).
- rlib/networks/ package directory removed entirely.
- All import sites updated:
from rlib.networks import Model -> from rlib.agent import Agent
from rlib.networks import ModelConfig -> from rlib.agent import ModelConfig
from rlib.networks import A2CConfig -> from rlib.A2C.model import A2CConfig
from rlib.networks import PPOConfig -> from rlib.PPO.model import PPOConfig
from rlib.networks.networks import X -> from rlib.models import X
- class X(Model) annotations / : Model / -> Model rewritten to Agent
across all agent files.
- rlib.A2C and rlib.PPO __init__ files re-export A2CConfig / PPOConfig.
- rlib/__init__.py lazy-submodule map: networks -> agent.
- tests/test_package.py parametrize list updated to match.
98/98 tests pass, ruff + format clean.
YAML configs are the constructor graph: any mapping with a
constructor: dotted.path.to.Class key is recursively instantiated by
importing the target and passing the remaining keys as kwargs.
Schema:
env:
constructor: rlib._cli.classic_envs # or atari_envs, custom callable
id: CartPole-v1
num_envs: 8
...
agent:
constructor: rlib.A2C.ActorCritic
model:
constructor: rlib.models.MLP
partial: true # body class, not instance
hidden_size: 64
input_size: ${input_shape} # interpolated from env bundle
...
trainer:
constructor: rlib.A2C.A2CTrainer
envs: ${train_envs}
val_envs: ${val_envs}
agent: ${agent}
config:
constructor: rlib.training.TrainerConfig
total_steps: 100_000
returns: GAE # str -> Returns enum (auto-coerced)
Conventions:
* constructor: dotted.path -- import + instantiate (recurse on kwargs).
* partial: true -- return functools.partial(target, **kwargs); used for
body classes (host constructor calls them itself with input_size).
* ${name} -- string-only interpolation against runtime namespace
(device, agent, plus env-factory bundle: train_envs/val_envs/
input_shape/action_size).
Usage:
python -m rlib.<Agent> path/to/config.yaml
python -m rlib.<Agent> path/to/config.yaml --set trainer.config.total_steps=1_000_000
Also:
* Added rlib.models.MLP (was duplicated across examples).
* Added clone_module helper for sibling networks (DDQN target net).
* PyYAML>=6.0 added to pyproject dependencies.
* 27 new tests in tests/test_cli.py covering instantiate / partial /
${...} interpolation / load_yaml + --set overrides / env factory /
clone_module.
125/125 tests pass.
- A2CTrainer.get_action / get_action_lstm: int(fastsample(policy)) ->
int(fastsample(policy).item()). The fastsample helper returns a
shape-(1,) ndarray; calling int() on it raises in NumPy >= 1.25.
- A2CTrainer._train_onestep: self.model.get_value(next_states) ->
self.model.evaluate(next_states)[1]. ActorCritic has no get_value
method (TF1 leftover). _train_onestep is only reached when train_mode
is ONESTEP, so this was a latent bug.
- PPO/RND/RANDAL/DAAC/DDQN get_action: return scalar Python int via
.item(). The validator passes single batched states and expects
scalar actions; ale_py rejects the (1,) ndarray that fastsample()
returned.
- DDQN trainer: self.num_steps -> self.nsteps. The attribute was
undefined; would crash on the first training rollout.
- RND/RANDAL trainers: state-normalisation shape mismatch.
`state_obs.update(next_states)` reduced only the time axis from a
(T, B, ...) rollout, leaving stats of shape (B, ...). The folded
batch in predictor_loss is (T*B, ...) -- broadcasting failed when
T*B != B. Fixed by:
1. init_state_obs reduces over the env axis too (mean(axis=0)).
2. _train_nstep folds (T, B, ...) -> (T*B, ...) before updating.
- VIN model: torch.sum(..., axis=1) -> dim=1 (torch canonical kwarg);
totorch(R).float().cuda() -> totorch(R, self.device).float() (was
hardcoded to CUDA, now respects configured device).
- NumpyReplayMemory: np.int -> np.int64 (np.int removed in NumPy 1.24).
The attribute on the trainer base class was already typed Agent (post Model->Agent rename in bf08cd6), but the variable was still called self.model. Trainer constructors took model=... and trainer code read self.model.X(). The rename only happened on the *agent class* side. This commit completes the rename for the trainer side: - SyncMultiEnvTrainer.__init__: model: Agent -> agent: Agent; self.model = model -> self.agent = agent. - Per-trainer subclass ctors: model: <SubclassType> -> agent: ... (A2C, PPO, RND, DDQN, DAAC, Curiosity, RANDAL, VIN, Unreal/feedforward, Unreal/lstm). - DDQN: target_model: DQN -> target_agent: DQN; self.target_model -> self.target_agent. - All call sites in trainer code: self.model.X() -> self.agent.X(). - Hyperparameter log prefix: model.{k} -> agent.{k}. - Examples and paper recipes updated: local var renamed model -> agent and target_model -> target_agent (DDQN); kwarg model=... -> agent=... in trainer constructions. (Agent constructor kwargs like target_model=PredictorCNN stay -- those are real Agent ctor params.) - Trainer.load() rewritten: dropped stale log_scalars/gpu_growth/ **attrs kwargs that no longer match the dataclass-config trainer signature; renamed the model parameter to agent and pass via the new TrainerConfig path. Crucially: self.model on the *Agent* classes (the body network attribute on ActorCritic / PPO / RND etc.) is unchanged. That is the real "model" -- the torch.nn.Module the agent wraps. Only the trainer-side reference to its agent is renamed. 125/125 tests pass, ruff + format clean.
- requires-python >=3.11 (was >=3.10): rlib.training.returns uses enum.member, added in 3.11. The alternative (staticmethod-wrapped callables on the enum) is rejected by Python enum metaclass since staticmethod is treated as a method descriptor, not a member value. - target-version = py311 in [tool.ruff], python_version = 3.11 in [tool.mypy]. - CI matrix dropped 3.10; runs on 3.11 and 3.12. - TrainMode now subclasses enum.StrEnum (3.11+) directly instead of (str, enum.Enum) — same behaviour, simpler. - Each trainer subclass got a class-level agent: <ConcreteAgent> annotation. Without it, mypy sees the base trainer is agent: Agent and Agent inherits from torch.nn.Module — and nn.Module.__getattr__ returns Tensor | Module, so calls like self.agent.intrinsic_reward() are flagged as "Tensor not callable". Narrowing the attribute on each subclass fixes the entire cluster. - Curiosity trainer: dropped self.state_mean = None / self.state_std = None (then later read .shape on them — would crash). The first RollingObs.update call sets them. - A2C, Curiosity, Unreal/lstm: replaced the TF1 leftover self.saver.save(self.sess, ...) with self.save_model(s). - A2C-LSTM and UNREAL-LSTM validate_sync: added `assert not isinstance(env, list)` so mypy can narrow val_envs (the dispatch in _validation_score guarantees this). - VIN trainer: stored model_dir, added self.s counter, added save() + no-op update_target() methods. The training loop calls them unconditionally when freq > 0; without these the standalone trainer would AttributeError as soon as save_freq > 0 was set. - ReplayMemory.py: dropped FrameBuffer, replayMemory (lower-case), stack_frames, preprocess_frame, and the standalone main() — all depended on scipy.misc.imresize (removed in SciPy 1.3, ~2019). Only NumpyReplayMemory remains; the file is now 64 lines (was 200). Result: 125/125 tests pass, ruff + format clean, mypy down from 95 to 42 errors. Most remaining are spurious nn.Module __getattr__ narrowing or numpy-loss-vs-int variable typing in the larger trainers; not blocking runtime.
- mkdocs --strict failed because nav: ../CHANGELOG.md and the
index.md changelog link both pointed outside docs_dir, which
strict mode rejects. Switched to a thin docs/changelog.md that
pulls the top-level CHANGELOG.md in via pymdownx.snippets:
docs/changelog.md # one-liner: ---8<--- "CHANGELOG.md"
mkdocs.yml # nav now references changelog.md
# snippets ext + base_path = [., docs]
- A2C/model.py: ActorCritic_LSTM.evaluate hidden parameter now typed
tuple[np.ndarray, np.ndarray] | None (was np.ndarray | None) and
the converted tensor versions get distinct names (hidden_t /
hidden_out / hidden_np) so type-narrowing works. Restored the
missing tonumpy_many(*hidden_out) on the way back.
- Unreal/lstm.py: dropped the dead AtariEnv__ helper (~38 lines).
- utils/utils.py: Welfords_algorithm.__init__ widened mean to
float | np.ndarray (was inferred int from default 0); test passes
np.zeros(4).
- tests/test_cli.py: assert isinstance(clone, torch.nn.Linear)
narrows clone_module return for type-checkers.
mypy on tests/ is clean (was 2 errors). 125/125 unit tests pass.
The lone remaining VS Code GitHub Actions warning on docs.yml
("Value github-pages is not valid") is a false positive — the
github-pages environment is auto-managed by actions/configure-pages
and does not need to exist as a checked-in env.
- git mv all 11 paper recipe scripts into examples/paper/scripts/
(history preserved). Added scripts/__init__.py.
- New examples/paper/configs/ with 11 matching YAML configs that
reproduce each scripts hyperparameters via the rlib._cli runner:
atari_a2c.yaml ALE/SpaceInvaders-v5
atari_ddqn.yaml ALE/SpaceInvaders-v5
atari_ppo.yaml ALE/SpaceInvaders-v5
atari_randal.yaml ALE/MontezumaRevenge-v5
atari_rnd.yaml ALE/MontezumaRevenge-v5
atari_unreal.yaml ALE/MontezumaRevenge-v5
classic_a2c.yaml CartPole-v1
classic_ddqn.yaml CartPole-v1
classic_ppo.yaml CartPole-v1
classic_randal.yaml MountainCar-v0
classic_rnd.yaml MountainCar-v0
- Each YAML maps the script directly: same lr / nsteps / num_envs /
total_steps / aux-loss coefficients / etc. Atari configs use
fire_reset: true (paper preprocessing) and the canonical ALE/<Game>-v5
ids (the legacy *Deterministic-v4 ids no longer ship with ale-py 0.10+).
- Updated the moved scripts: from examples.paper.common ->
from examples.paper.scripts.common; docstring `python ...` examples
point at the new path.
- README rewrite: side-by-side YAML vs script tables, override examples,
Atari sweep snippet.
- Smoke run of `python -m rlib.A2C examples/paper/configs/classic_a2c.yaml`
trains end-to-end (validation score 51 -> 24 across 2 updates).
125/125 tests pass, ruff + format clean.
- New rlib.training.trainer.SyncMultiEnvTrainer._progress(iterator, num_updates): wraps a training-loop range with a tqdm bar (desc=ClassName[env_id], unit=update). Bar is stored on self._pbar so validation_summary attaches the latest score / loss / fps as postfix metrics. - Replaced the for-loop in every trainer (_train_nstep / _train_onestep) with self._progress(...) — base trainer + A2C (n-step + 1-step), A2C-LSTM, PPO, RND, RANDAL, Curiosity, DAAC, Unreal feedforward + LSTM, VIN. - validation_summary and saved-model logs now use tqdm.write(...) so they no longer break the bar. - Added tqdm>=4.60 to project dependencies. 125/125 tests pass; smoke run of classic_a2c.yaml shows live bar updating.
- New top-level Makefile with targets that mirror .github/workflows/ci.yml
(and docs.yml). Run `make ci` locally before pushing to catch the same
failures the pipeline does. Targets: install, lint, format, format-check,
typecheck, test, build, docs, ci, clean. TYPECHECK_PATHS mirrors the env
var in ci.yml so the local + CI mypy invocations stay in lockstep.
- Fix the mypy errors that triggered this:
* rlib/envs/apple_picker.py: annotate item_locs as dict[int,dict[int,int]],
widen agent_loc to tuple|list, annotate info: dict.
* rlib/envs/vec_env.py: rename the Callable returned by _send_step to `recv`
so reassigning `results = list(...)` no longer confuses mypy; rename the
DummyBatchEnv.step zip-unpack locals (obs_b/rewards_b/done_b/infos_b)
so the per-env scalar `done: bool` and `info: dict` annotations are not
shadowed by tuples.
* rlib/envs/wrappers.py: ToTorchEnv.step deliberately overrides the
Tensor return shape; place `# type: ignore[override]` on the def-line
so mypy actually consumes it.
`make ci` (lint + format-check + typecheck + test + build): all green,
125/125 tests pass.
The deployed site (https://jhare96.github.io/reinforcement-learning/) only had hand-written overview pages and no per-class API docs. Add an auto-generated API reference driven by mkdocstrings[python] so every public class / function gets a page sourced directly from its docstrings + signatures. - pyproject.toml: add mkdocstrings[python]>=0.24 to the docs extra. - mkdocs.yml: register the mkdocstrings plugin (Google docstring style, source linked, members in source order, signature on its own line, private members filtered) and add an API reference nav section covering: agent base, every agent package (A2C / DDQN / PPO / RND / RANDAL / Curiosity / UNREAL / DAAC / VIN), training, envs, models, utils. - docs/api/*.md: 14 new pages, each is a one-line `::: rlib.<module>` stub that mkdocstrings expands into the full reference. - Doc fixes uncovered by `mkdocs --strict`: * rlib/training/trainer.py SyncMultiEnvTrainer.__init__ docstring referenced the obsolete `model:` parameter (now `agent:`). * rlib/models.py MaskedLSTMBlock.forward used `name - description` which Google-style griffe could not parse; switched to `name: description`. `make ci` + `make docs` both green.
The live site at https://jhare96.github.io/reinforcement-learning/ was missing the API reference and the README content because: 1. docs/index.md was a hand-written stub instead of pulling in README.md. 2. .github/workflows/docs.yml only deployed on push to master, but all of the recent v3 work (mkdocstrings API reference, updated examples, paper configs) lives on the v3 branch. Changes: - docs/index.md: replace the hand-written quick-links section with a pymdownx.snippets include of README.md, so the docs home renders the full project README. The Design overview ASCII diagram is kept below. - .github/workflows/docs.yml: deploy job now also fires on push to v3 (still gated to push events only — PRs still just upload the artefact). - README.md: rewrite a handful of repo-relative links (CONTRIBUTING.md, LICENSE, NOTICE, pyproject.toml, examples/cartpole_a2c.py) as absolute https://github.com/... URLs so they resolve correctly when the README is rendered as the docs landing page (mkdocs --strict was rejecting them as broken). `make docs` builds clean in --strict; `make ci` green.
The CHANGELOG had two pre-release sections (the historical 3.0.0 description
of the gym->gymnasium shim, and an unreleased 3.1.0 describing the rlib.envs
rework) that no longer match what is actually on this branch. No 3.0.0 tag
has been cut yet, so collapse them into one [3.0.0] - Unreleased entry that
documents the full set of changes since v2:
- Packaging / licensing / tooling (Apache 2.0, PEP 621, Dockerfile,
py.typed, Makefile, pre-commit, GitHub Actions CI + Docs Pages deploy).
- Environment layer (rlib.envs: RLEnv / RLVecEnv ABCs, BatchEnv /
DummyBatchEnv / ChunkEnv, wrappers, ApplePicker port).
- Training infrastructure (rlib.training: SyncMultiEnvTrainer,
TrainerConfig + per-agent subclasses, TrainMode StrEnum,
Returns enum-of-functions, Validator, tqdm progress bars,
auto-logged hyperparameters).
- Agent layer (rlib.agent.Agent + ModelConfig, model.py / trainer.py
split, rlib.models replacing rlib.networks).
- CLI (rlib._cli YAML runner with constructor / partial / interpolation
support, per-agent __main__ shims).
- Examples (examples/ + examples/paper/{scripts,configs}).
- Docs site (mkdocs-material + mkdocstrings API reference).
- Removals (legacy gym, rlib.utils.gym_compat, rlib.networks,
rlib.utils.SyncMultiEnvTrainer, in-module demo blocks).
- Migration notes block with a diff of the most common import / call
changes.
Drive-by:
- rlib/__init__.py: __version__ 3.1.0 -> 3.0.0 (no 3.0.0 release tag yet).
- README.md: refresh quickstart for the YAML CLI + Python config-only
trainer/agent API; bump python badge to 3.11; correct repo layout to
reflect rlib.envs / rlib.training / rlib.models / agent.py / _cli.py.
- docs/agents.md, environments.md, wrappers.md: update class names
(A2CLSTMTrainer, CuriosityTrainer), kill references to the deleted
rlib.utils.SyncMultiEnvTrainer / rlib.utils.VecEnv / rlib.utils.wrappers
/ rlib.envs.adapters / RLEnvBase / register_backend, and switch the
cross-links to the in-site API reference pages.
`make ci` (lint + format-check + typecheck + test + build) green.
`make docs` --strict builds clean. Wheel/sdist now build as rlib-3.0.0.
There was a problem hiding this comment.
Pull request overview
This PR bumps rlib to a “V3” layout by adding CI/docs/Docker automation and extensive documentation/examples, while also introducing significant internal refactors (new rlib.envs contract, rlib.training package, and updated agent/trainer modules).
Changes:
- Added GitHub Actions workflows for lint/typecheck/tests/build, docs build/deploy, and Docker smoke builds.
- Added MkDocs-based documentation, contribution/process templates, and a changelog/notice.
- Refactored core library structure (env API, training/returns/validation utilities, agent modules) and added a new test suite + examples/paper reproduction configs.
Reviewed changes
Copilot reviewed 148 out of 153 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
tests/conftest.py |
Global pytest env defaults for determinism/CPU. |
tests/test_envs.py |
Tests for rlib.envs public contract + ApplePicker registration. |
tests/test_package.py |
Tests for package version + lazy submodule imports. |
tests/test_schedulers.py |
Tests for polynomial LR scheduler helper. |
tests/test_training.py |
Tests for training validators + return estimators enum. |
tests/test_vec_env.py |
Tests for vector env runner behavior (DummyBatchEnv). |
tests/test_wrappers.py |
Tests for env wrapper behavior (reward clipping, time limit, etc.). |
setup.py |
Simplified setuptools shim relying on pyproject.toml. |
rlib/__init__.py |
Version export + lazy submodule loader. |
rlib/py.typed |
Marks package as PEP 561 typed. |
rlib/agent.py |
Introduces shared Agent base + ModelConfig and optimizer/scheduler plumbing. |
rlib/training/__init__.py |
Training package re-exports (trainer/config/returns/validation). |
rlib/training/config.py |
Frozen trainer config + TrainMode. |
rlib/training/returns.py |
Return/advantage estimators + Returns enum dispatch. |
rlib/training/validation.py |
Async/sync validation strategies + factory. |
rlib/utils/__init__.py |
Utility package docstring/positioning. |
rlib/utils/schedulers.py |
Typed polynomial LR scheduler helper. |
rlib/utils/utils.py |
Utility formatting/typing + returns functions re-export for backwards compat. |
rlib/utils/play.py |
Updated interactive play helper to new env API. |
rlib/utils/random_agent.py |
Updated random-agent runner to new env API. |
rlib/utils/VecEnv.py |
Removes legacy vec-env implementation. |
rlib/envs/__init__.py |
Canonical env subpackage + ApplePicker registration + gymnasium.make re-export. |
rlib/envs/base.py |
Defines RLEnv and RLVecEnv canonical contracts and merge helpers. |
rlib/envs/apple_picker.py |
Adds/updates ApplePicker env to Gymnasium 5-tuple API. |
rlib/VIN/model.py |
Adds VIN model updated to shared Agent/ModelConfig pattern. |
rlib/VIN/__init__.py |
VIN submodule exports. |
rlib/A3C/A3C.py |
Edits legacy A3C script to updated utils imports/formatting. |
rlib/A3C/__init__.py |
A3C package marker. |
rlib/A2C/ActorCritic.py |
Removes legacy A2C ActorCritic implementation. |
rlib/A2C/A2C.py |
Removes legacy A2C trainer script. |
rlib/A2C/__init__.py |
New A2C package exports. |
rlib/A2C/__main__.py |
Adds YAML-driven CLI entry point. |
rlib/PPO/model.py |
Adds PPO base/model classes + config typing/docs. |
rlib/PPO/trainer.py |
Adds PPO trainer implementation on new training base. |
rlib/PPO/__init__.py |
PPO submodule exports. |
rlib/PPO/__main__.py |
Adds YAML-driven CLI entry point. |
rlib/DDQN/model.py |
Adds DDQN/DQN model on new Agent base. |
rlib/DDQN/trainer.py |
Adds synchronous DDQN trainer on new training base. |
rlib/DDQN/__init__.py |
DDQN submodule exports. |
rlib/DDQN/__main__.py |
Adds YAML-driven CLI entry point. |
rlib/DAAC/model.py |
Adds DAAC composite agent implementation. |
rlib/DAAC/trainer.py |
Adds DAAC trainer on new training base. |
rlib/DAAC/__init__.py |
DAAC submodule exports. |
rlib/DAAC/__main__.py |
Adds YAML-driven CLI entry point. |
rlib/RND/__init__.py |
RND submodule exports. |
rlib/RND/__main__.py |
Adds YAML-driven CLI entry point. |
rlib/RANDAL/__init__.py |
RANDAL submodule exports. |
rlib/RANDAL/__main__.py |
Adds YAML-driven CLI entry point. |
rlib/Curiosity/model.py |
Adds Curiosity agent implementation (ICM). |
rlib/Curiosity/trainer.py |
Adds Curiosity trainer on new training base. |
rlib/Curiosity/__init__.py |
Curiosity submodule exports. |
rlib/Curiosity/__main__.py |
Adds YAML-driven CLI entry point. |
rlib/Unreal/__init__.py |
Unreal submodule exports. |
rlib/Unreal/__main__.py |
Adds YAML-driven CLI entry point. |
rlib/.vscode/settings.json |
Removes editor-specific Python path setting. |
requirements.txt |
Adds pip-installable dependency list for Docker/users. |
pyproject.toml |
Modern packaging metadata + ruff/mypy/pytest configuration. |
NOTICE |
Adds attribution/third-party notice (Baselines MIT excerpt). |
Makefile |
Adds local targets mirroring CI checks (lint/type/test/docs/build). |
Dockerfile |
Adds CPU-only Docker image build + smoke test defaults. |
.gitignore |
Adjusts ignore rules (including allowing requirements*.txt). |
.github/workflows/ci.yml |
CI: ruff, mypy, pytest matrix, build+twine checks. |
.github/workflows/docs.yml |
Docs build + GitHub Pages deploy pipeline. |
.github/workflows/docker.yml |
Docker build + image smoke test pipeline. |
.github/PULL_REQUEST_TEMPLATE.md |
PR template for consistent change/test reporting. |
.github/ISSUE_TEMPLATE/bug_report.md |
Bug report template. |
.github/ISSUE_TEMPLATE/feature_request.md |
Feature request template. |
mkdocs.yml |
MkDocs Material site configuration + nav and mkdocstrings. |
docs/index.md |
Docs landing page including design overview. |
docs/agents.md |
Agent overview + common training patterns. |
docs/environments.md |
Env contract/usage docs (5-tuple API + vec env runners). |
docs/wrappers.md |
Wrapper reference docs. |
docs/changelog.md |
Includes top-level CHANGELOG in docs site. |
docs/api/agent.md |
mkdocstrings API stub for rlib.agent. |
docs/api/a2c.md |
mkdocstrings API stub for rlib.A2C. |
docs/api/curiosity.md |
mkdocstrings API stub for rlib.Curiosity. |
docs/api/daac.md |
mkdocstrings API stub for rlib.DAAC. |
docs/api/ddqn.md |
mkdocstrings API stub for rlib.DDQN. |
docs/api/envs.md |
mkdocstrings API stub for rlib.envs. |
docs/api/models.md |
mkdocstrings API stub for rlib.models. |
docs/api/ppo.md |
mkdocstrings API stub for rlib.PPO. |
docs/api/randal.md |
mkdocstrings API stub for rlib.RANDAL. |
docs/api/rnd.md |
mkdocstrings API stub for rlib.RND. |
docs/api/training.md |
mkdocstrings API stub for rlib.training. |
docs/api/unreal.md |
mkdocstrings API stub for rlib.Unreal. |
docs/api/utils.md |
mkdocstrings API stub for rlib.utils.*. |
docs/api/vin.md |
mkdocstrings API stub for rlib.VIN. |
examples/README.md |
Documents runnable example scripts. |
examples/cartpole_a2c.py |
Minimal CartPole A2C example using new APIs. |
examples/atari_ppo.py |
Minimal Atari PPO example using new APIs. |
examples/montezuma_rnd.py |
Minimal Montezuma RND example using new APIs. |
examples/paper/README.md |
Paper reproduction guide (YAML vs scripts). |
examples/paper/__init__.py |
Paper package docs/description. |
examples/paper/scripts/common.py |
Shared env factories + network bodies for reproduction runs. |
examples/paper/scripts/classic_a2c.py |
Paper classic-control A2C script. |
examples/paper/scripts/classic_ddqn.py |
Paper classic-control DDQN script. |
examples/paper/scripts/classic_ppo.py |
Paper classic-control PPO script. |
examples/paper/scripts/classic_rnd.py |
Paper classic-control RND script. |
examples/paper/scripts/classic_randal.py |
Paper classic-control RANDAL script. |
examples/paper/scripts/atari_a2c.py |
Paper Atari A2C script. |
examples/paper/scripts/atari_ddqn.py |
Paper Atari DDQN script. |
examples/paper/scripts/atari_ppo.py |
Paper Atari PPO script. |
examples/paper/scripts/atari_rnd.py |
Paper Atari RND script. |
examples/paper/scripts/atari_randal.py |
Paper Atari RANDAL script. |
examples/paper/scripts/atari_unreal.py |
Paper Atari UNREAL script. |
examples/paper/configs/classic_a2c.yaml |
YAML config for paper classic A2C run. |
examples/paper/configs/classic_ddqn.yaml |
YAML config for paper classic DDQN run. |
examples/paper/configs/classic_ppo.yaml |
YAML config for paper classic PPO run. |
examples/paper/configs/classic_rnd.yaml |
YAML config for paper classic RND run. |
examples/paper/configs/classic_randal.yaml |
YAML config for paper classic RANDAL run. |
examples/paper/configs/atari_a2c.yaml |
YAML config for paper Atari A2C run. |
examples/paper/configs/atari_ddqn.yaml |
YAML config for paper Atari DDQN run. |
examples/paper/configs/atari_ppo.yaml |
YAML config for paper Atari PPO run. |
examples/paper/configs/atari_rnd.yaml |
YAML config for paper Atari RND run. |
examples/paper/configs/atari_randal.yaml |
YAML config for paper Atari RANDAL run. |
examples/paper/configs/atari_unreal.yaml |
YAML config for paper Atari UNREAL run. |
CONTRIBUTING.md |
Contributor guide and development workflow. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+140
to
+142
| if render: | ||
| with self._lock: | ||
| env.close() |
Comment on lines
+54
to
+59
| if x in self.item_locs: | ||
| if y in self.item_locs: | ||
| pass | ||
| else: | ||
| self.item_locs[x][y] = 1 | ||
| objects += 1 |
Comment on lines
+19
to
+23
| color = 1 if distance <= r**2 else 0 # np.clip(r - distance, 0, 1) | ||
| try: | ||
| grid[i, j, 2] = color | ||
| print('i,j', (i, j)) | ||
| except IndexError: |
Comment on lines
+37
to
+41
| # Probe a single env to discover input shape and action space. | ||
| probe = AtariEnv(gym.make(env_id), k=4, episodic=False, reset=False, clip_reward=False) | ||
| input_shape = probe.reset().shape | ||
| action_size = probe.action_space.n | ||
| probe.close() |
Comment on lines
+97
to
+99
| def get_action(self, states): | ||
| policies, values = self.agent.evaluate(states) | ||
| return int(fastsample(policies).item()) |
Comment on lines
+28
to
+55
| ## Coding style | ||
|
|
||
| - Target **Python 3.10+**. | ||
| - Match the surrounding code style. The codebase is gradually being typed — | ||
| please add type hints (PEP 604 `X | Y` union syntax, built-in generics | ||
| like `list[int]`/`dict[str, Any]`) to any new public function. | ||
| - Keep changes focused: one feature or fix per PR. | ||
| - Avoid adding new top-level dependencies unless strictly necessary; prefer | ||
| putting heavy or environment-specific deps behind an optional extra in | ||
| `pyproject.toml`. | ||
|
|
||
| ## Environment abstraction | ||
|
|
||
| When touching wrappers, vectorised env runners, or adding support for a new | ||
| gym-like backend (`dm_env`, PettingZoo, EnvPool, an in-house simulator, ...), | ||
| build on the [`rlib.envs`](rlib/envs/) package: | ||
|
|
||
| - Subclass [`RLEnvBase`](rlib/envs/base.py) (or, for protocol-only typing, | ||
| use the `RLEnv` Protocol) and implement the modern 5-tuple | ||
| `step(action) -> (obs, reward, terminated, truncated, info)` and | ||
| `reset(*, seed=None, options=None) -> (obs, info)`. | ||
| - For a new backend, drop a small adapter file under | ||
| [`rlib/envs/adapters/`](rlib/envs/adapters/) and register it with | ||
| `register_backend(predicate, adapter_cls)` so `rlib.envs.make` / | ||
| `rlib.envs.wrap` will auto-pick it up. | ||
|
|
||
| The legacy `rlib.utils.gym_compat` shim has been removed; use | ||
| `rlib.envs.wrap` (or just `import gymnasium as gym`) in new code. |
Comment on lines
+71
to
+72
| - Subclass `rlib.utils.SyncMultiEnvTrainer.SyncMultiEnvTrainer` (or document | ||
| why a custom trainer is needed). |
Comment on lines
+74
to
77
| rl_env = env | ||
| rl_env.reset() | ||
| rendered = env.render(mode='rgb_array') | ||
|
|
Comment on lines
+19
to
+37
| __version__ = "3.0.0" | ||
|
|
||
| # Mapping of attribute name -> dotted submodule path | ||
| _LAZY_SUBMODULES = { | ||
| "A2C": "rlib.A2C", | ||
| "A3C": "rlib.A3C", | ||
| "PPO": "rlib.PPO", | ||
| "DDQN": "rlib.DDQN", | ||
| "RND": "rlib.RND", | ||
| "RANDAL": "rlib.RANDAL", | ||
| "Curiosity": "rlib.Curiosity", | ||
| "Unreal": "rlib.Unreal", | ||
| "DAAC": "rlib.DAAC", | ||
| "VIN": "rlib.VIN", | ||
| "envs": "rlib.envs", | ||
| "agent": "rlib.agent", | ||
| "training": "rlib.training", | ||
| "utils": "rlib.utils", | ||
| } |
Comment on lines
+10
to
+36
| requires-python = ">=3.11" | ||
| license = { file = "LICENSE" } | ||
| authors = [ | ||
| { name = "Joshua Hare" }, | ||
| ] | ||
| keywords = [ | ||
| "reinforcement-learning", | ||
| "deep-learning", | ||
| "pytorch", | ||
| "ppo", | ||
| "a2c", | ||
| "ddqn", | ||
| "rnd", | ||
| "unreal", | ||
| "sparse-rewards", | ||
| ] | ||
| classifiers = [ | ||
| "Development Status :: 4 - Beta", | ||
| "Intended Audience :: Developers", | ||
| "Intended Audience :: Science/Research", | ||
| "License :: OSI Approved :: Apache Software License", | ||
| "Operating System :: OS Independent", | ||
| "Programming Language :: Python :: 3", | ||
| "Programming Language :: Python :: 3 :: Only", | ||
| "Programming Language :: Python :: 3.10", | ||
| "Programming Language :: Python :: 3.11", | ||
| "Programming Language :: Python :: 3.12", |
Bug fixes:
- rlib/envs/apple_picker.py:
* draw_circle: drop the debug print() inside the pixel loop.
* generate_random_locs: fix the membership check; previously it
tested `if y in self.item_locs` (an x-keyed dict), almost always
False, so duplicate (x, y) pairs could be counted as new objects.
Switched to `y in self.item_locs[x]`.
* Switch the IndexError swallow to contextlib.suppress (ruff SIM105).
- examples/atari_ppo.py: Gymnasiums reset returns (obs, info), so
`probe.reset().shape` raised AttributeError. Use `probe.reset()[0].shape`.
- rlib/training/validation.py: AsyncValidator no longer calls env.close()
at the end of a render-true run; the env is owned by the caller / the
trainer reuses it across validations.
Vec-validator support:
- rlib/{A2C,PPO,DAAC,RND,RANDAL,Curiosity,DDQN}/trainer.py get_action
used to do `int(fastsample(...).item())`, which only works for the
AsyncValidators single-env path. SyncValidator passes a batched
states array of shape (num_envs, ...) and expects an action per env.
Each get_action now returns the full per-env action array when the
input is batched, and keeps the int fast-path for the single-env case
(`states.shape[0] == 1`). DDQN and Curiosity get the same treatment
even though they were not flagged, because the same code paths apply.
Doc / packaging fixes:
- pyproject.toml: drop the stale `Programming Language :: Python :: 3.10`
classifier so Trove matches `requires-python = ">=3.11"`.
- CONTRIBUTING.md: bump the supported Python version to 3.11+; rewrite
the env-abstraction section to describe the actual public API
(`RLEnv` ABC, `RLVecEnv` ABC, `merge_done`/`merge_info`,
`gymnasium.make`); the previous text referenced the long-deleted
`RLEnvBase`, `rlib.envs/adapters/`, `register_backend`,
`rlib.envs.wrap` and `rlib.utils.gym_compat`. Fix the "Adding a new
agent" section to point at `rlib.training.SyncMultiEnvTrainer` and
describe the model.py / trainer.py split + YAML config convention.
- docs/index.md: redraw the design-overview diagram around the current
module layout (rlib.envs.wrappers / BatchEnv / DummyBatchEnv / RLEnv /
RLVecEnv / rlib.training.SyncMultiEnvTrainer / rlib.agent.Agent),
and replace the obsolete "compat shim" paragraph with the correct
description of `RLVecEnv.merge_done` / `merge_info` as the single
5->4 tuple boundary.
`make ci` (lint + format + typecheck + 125 tests + build + twine) green.
`make docs --strict` builds clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces comprehensive project infrastructure and documentation improvements to support development, contribution, and maintenance of the
rliblibrary. The changes include the addition of GitHub Actions workflows for CI, documentation, and Docker builds, new issue and pull request templates, a detailed changelog, and a contributor guide.Key changes are as follows:
Continuous Integration and Automation:
.github/workflows/ci.ymlto run linting (ruff), type checking (mypy), testing (pytest), and build distribution artefacts on pushes and pull requests to main branches..github/workflows/docs.ymlto automate documentation builds and deploy to GitHub Pages on relevant changes..github/workflows/docker.ymlfor building and smoke-testing Docker images on code or dependency changes.Documentation and Contribution:
CONTRIBUTING.mdwith clear guidelines for reporting issues, setting up the development environment, code style, environment abstraction, submitting pull requests, and adding new agents.CHANGELOG.mdwith a detailed, versioned record of all notable changes, following Keep a Changelog and Semantic Versioning conventions.Issue and PR Templates:
.github/ISSUE_TEMPLATE/bug_report.md) and feature requests (.github/ISSUE_TEMPLATE/feature_request.md) to standardize incoming issues. [1] [2].github/PULL_REQUEST_TEMPLATE.md) to guide contributors in documenting their changes and testing.