SheepRL's DreamerV3 plays Streets Of Rage.

[Disastorm]

I used SheepRL to train a dreamerV3 model to play the last 4 stages of Streets of Rage. If anyone is interested here is a link to my youtube video at the timestamp of the playthrough ( before this timestamp, SB3's PPO was used to train the first 4 stages ).

Each of the 4 final stages has their own dreamerV3 model.
The final playthrough is done on specific seeds so that performance is always the same. However, at least from my experience, the dreamerV3 models perform very similarly across different seeds.

https://youtu.be/gpRdGwSonoo?t=1515

[belerico]

Hi @Disastorm, i'm super happy that you were able to train Dreamer-V3 on your game. Right now we're working to improve it even better! Would you mind to open a PR where you add Streets of Rage to SheepRL so that everyone can test also other agents on that?

[Disastorm]

Sorry its a bit hacked together so don't really want to PR. I can post here my env and if you want you can implement it more properly.

I changed the below sheeprl file to pass the vector_env_idx to the env.

diff --git a/sheeprl/utils/env.py b/sheeprl/utils/env.py
index 14d5710..acba5b7 100644
--- a/sheeprl/utils/env.py
+++ b/sheeprl/utils/env.py
@@ -69,6 +69,8 @@ def make_env(
             instantiate_kwargs["seed"] = seed
         if "rank" in cfg.env.wrapper:
             instantiate_kwargs["rank"] = rank + vector_env_idx
+        # if "index" in cfg.env.wrapper:
+        instantiate_kwargs["index"] = vector_env_idx
         env = hydra.utils.instantiate(cfg.env.wrapper, **instantiate_kwargs, _convert_="all")

Things to note about my env:

• get_env function should return your stable retro gym env. I have some of my custom wrappers in there that are not included here.
• I think some env constructor params such as screen_size are not used. Also for some reason when I actually sized my screen properly,
  the transform_obs TransformObservation resizer in sheeprl was failing, so I just kept my screen size big and let the transform_obs resizer resize it which works.
• loadstate variable in env yaml can be comma separated to train different game states in parallel environments. This is also what is using the above vector_env_idx.
• I trained by using 64x64 RGB images. I tried other settings including 128x128 and monochrome and I felt this was the best.
• I have a dict called act_dict which maps all possible Sega Genesis inputs into 4 values, first value is Dpad 9 directions, and last 3 values are binary buttons. This was done to get a Discrete action space ( I couldn't get MultiBinary to work with SheepRL DreamerV3). This dict can actually be moved into its own custom Gym Wrapper instead of having this here.
• My custom FrameSkipWrapper skips 4 frames so I kept sheeprl's action_repeat = 1.

my env retro.py file is below:

from __future__ import annotations

from typing import Any, Dict, List, Optional, Sequence, SupportsFloat, Tuple, Union

import gymnasium as gym
import numpy as np
from gymnasium import spaces
from gymnasium.core import RenderFrame
import retro

# the below import is my custom gym wrappers so replace it with your own
from frame_skip_wrapper import FrameSkipWrapper, GenesisMultiDiscreteWrapper,  QuantizedWarpFrame


global_env = None

# this should return your retro gym env
def get_env(current_rom='StreetsOfRage-Genesis', state_name='1Player.Blaze.Round5.Normal', rend_mode='rgb_array'):
        global global_env
        if global_env is not None:
            global_env.close()
        
        print("loading state " + state_name)
        env = retro.make(current_rom, state_name, render_mode=rend_mode)
        
        env = QuantizedWarpFrame(env,num_colors=None, greyscale = False, resize=False)

        clipping_rewards = False
        if not clipping_rewards:
            print("not clipping rewards")
        env = FrameSkipWrapper(env, skip=4, clip_reward=clipping_rewards, use_action_queue=False)

        env = GenesisMultiDiscreteWrapper(env)

        global_env = env
        return env

class RetroSheepWrapper(gym.Wrapper):
    def __init__(self, rom: str, loadstate: str, screen_size: Sequence[int, int] | int, index: int | None = 0, seed: int | None = None) -> None:
        state_names = loadstate.split(',')

        print("state names " + repr(state_names) + " index: " + str(index) + " seed: " + str(seed))

        state_names_idx = index % len(state_names)

        env = get_env(current_rom=rom, state_name=state_names[state_names_idx]);
        super().__init__(env)
        self.env = env

        self.observation_space = spaces.Dict(
            {
                "rgb": spaces.Box(
                    self.env.observation_space.low,
                    self.env.observation_space.high,
                    self.env.observation_space.shape,
                    self.env.observation_space.dtype,
                )
            }
        )
        self.action_space = spaces.Discrete(72)

        self.reward_range = self.env.reward_range or (-np.inf, np.inf)
        self.observation_space.seed(seed)
        self.action_space.seed(seed)
        # render
        self._render_mode: str = "rgb_array"
        # metadata
        self._metadata = {"render_fps": 30}

    act_dict = [   # 72 variations for Genesis. This should be moved into a Gym Wrapper.
        [0,0,0,0], # noop
        [1,0,0,0], # up
        [2,0,0,0], # down
        [3,0,0,0], # left
        [4,0,0,0], # right
        [5,0,0,0], # up - left
        [6,0,0,0], # up - right
        [7,0,0,0], # down - left
        [8,0,0,0], # down - right

        [0,1,0,0], # B
        [0,0,1,0], # A
        [0,0,0,1], # C
        [0,1,1,0], # BA
        [0,1,0,1], # BC
        [0,0,1,1], # AC
        [0,1,1,1], # BAC

        # begin variations:

        [1,1,0,0], # B
        [1,0,1,0], # A
        [1,0,0,1], # C
        [1,1,1,0], # BA
        [1,1,0,1], # BC
        [1,0,1,1], # AC
        [1,1,1,1], # BAC

        [2,1,0,0], # B
        [2,0,1,0], # A
        [2,0,0,1], # C
        [2,1,1,0], # BA
        [2,1,0,1], # BC
        [2,0,1,1], # AC
        [2,1,1,1], # BAC

        [3,1,0,0], # B
        [3,0,1,0], # A
        [3,0,0,1], # C
        [3,1,1,0], # BA
        [3,1,0,1], # BC
        [3,0,1,1], # AC
        [3,1,1,1], # BAC

        [4,1,0,0], # B
        [4,0,1,0], # A
        [4,0,0,1], # C
        [4,1,1,0], # BA
        [4,1,0,1], # BC
        [4,0,1,1], # AC
        [4,1,1,1], # BAC

        [5,1,0,0], # B
        [5,0,1,0], # A
        [5,0,0,1], # C
        [5,1,1,0], # BA
        [5,1,0,1], # BC
        [5,0,1,1], # AC
        [5,1,1,1], # BAC

        [6,1,0,0], # B
        [6,0,1,0], # A
        [6,0,0,1], # C
        [6,1,1,0], # BA
        [6,1,0,1], # BC
        [6,0,1,1], # AC
        [6,1,1,1], # BAC

        [7,1,0,0], # B
        [7,0,1,0], # A
        [7,0,0,1], # C
        [7,1,1,0], # BA
        [7,1,0,1], # BC
        [7,0,1,1], # AC
        [7,1,1,1], # BAC

        [8,1,0,0], # B
        [8,0,1,0], # A
        [8,0,0,1], # C
        [8,1,1,0], # BA
        [8,1,0,1], # BC
        [8,0,1,1], # AC
        [8,1,1,1], # BAC

    ]
    def _action(self, action):
        # print("action: " + repr(action))
        return self.act_dict[action]

    @property
    def render_mode(self) -> str | None:
        return self._render_mode

    def _convert_obs(self, obs: np.ndarray) -> Dict[str, np.ndarray]:
        return {"rgb": obs}

    def step(self, action: Any) -> Tuple[Any, SupportsFloat, bool, bool, Dict[str, Any]]:
        obs, reward, done, truncated, info = self.env.step(self._action(action))
        return self._convert_obs(obs), reward, done, truncated, info

    def reset(
        self, *, seed: Optional[int] = None, options: Optional[Dict[str, Any]] = None
    ) -> Tuple[Any, Dict[str, Any]]:
        obs, info = self.env.reset(seed=seed, options=options)
        return self._convert_obs(obs), info

    def render(self) -> Optional[Union[RenderFrame, List[RenderFrame]]]:
        return self.env.render()

    def close(self) -> None:
        return

my env yaml looks something like this:

defaults:
  - default
  - _self_

# Override from `default` config
id: StreetsOfRage-Genesis
action_repeat: 1
capture_video: False
num_envs: 4
from_vectors: False
screen_size: 64
render_mode: rgb_array
grayscale: False

# Wrapper to be instantiated
wrapper:
  _target_: sheeprl.envs.retro.RetroSheepWrapper
  rom: StreetsOfRage-Genesis
  loadstate: 1Player.Blaze.Round8.Normal
  # loadstate: 1Player.Blaze.Round8.Normal,stage5-boss-2lives
  screen_size: 64
  seed: ${seed}

[belerico]

Thank you @Disastorm! We are in the middle of fixing some stuff about dreamer-v3: when we're done I'll notify you, so maybe you can rerun the experiment and tell us if anything improves.
Thank you