Library Conversion: Open AI Baselines

[dynamicwebpaige]

OpenAI Baselines is a set of high-quality implementations of reinforcement learning algorithms. These algorithms make it easier for the research community to replicate, refine, and identify new ideas, and creates solid baselines to build research on top of.

The purpose of this ticket will be to migrate the OpenAI Baselines library to the TF 2.0 API.

• Convert OpenAI Baselines library.
• Run unit tests.
• Ensure processes are well understood.

For more information on OpenAI baselines, check out their blog post here - and be sure to amble around their website, as well as the OpenAI gym!

[navneet-nmk]

I believe even google dopamine needs an update to tf 2.0

[ffahleraz]

Hi I have previously worked on some projects with OpenAI Baselines, can I have a try on this?

[tomerk]

Adding @tanzhenyu as an assignee here because he's been taking a lead on this & on coordinating with the OpenAI team. Zhenyu, do you have any part of the migration in mind that you feel open source contributors would be able to assist with right now?

[tanzhenyu]

Sorry for the delay. Yes it'd be great to get some help from external contributors. @ffahleraz would you be able to help on one or two algos?

[josiahbjorgaard]

@tanzhenyu I'd be happy to assist as well! I could help with an algorithm or two. Would that be useful?

[tanzhenyu]

@josiahbjorgaard Yep that'd be very helpful. The discussion with OpenAI is to first port things to my own repo. I have already done so with PPO/A2C/DDPG, and working on DQN right now. We can use help on 1) GAIL, 2) ACER, 3) ACKTR, 4) HER, 5) TRPO.
Here's my repo: https://github.com/tanzhenyu/baselines-tf2. You can branch and make PR directly, and once everything is finalized we can port to OpenAI master branch.
I have written most util methods that are required, so hopefully the work required for each algo is stand-alone. @josiahbjorgaard @ffahleraz Let me know what algos you want to grab.

[josiahbjorgaard]

@tanzhenyu I'll start on ACER

[8bitmp3]

I believe even google dopamine needs an update to tf 2.0

TF-Agents is prioritized over Dopamine - @dynamicwebpaige.

If anyone is interested, there's also Stable Baselines - a TensorFlow-based fork of OpenAI's Baselines: https://github.com/Stable-Baselines-Team/stable-baselines with ACER, A2C, ACKTR, HER, PPO2, SAC etc. Full blog post here.

[schneimo]

@tanzhenyu I will start on ACKTR, if no one works already on this.

[8bitmp3]

@tanzhenyu I will start on ACKTR, if no one works already on this.

Go for it @MoritzTaylor - handing the ownership over to you - you may be a lot quicker 👍

[tanzhenyu]

@MoritzTaylor Sounds good. Have fun and let me know how I could help!

[seungjaeryanlee]

I will start working on GAIL! 🚀

[tanzhenyu]

@MoritzTaylor @josiahbjorgaard @seungjaeryanlee
Gently ping to check the status on your end :-)

Some update from my side: I have merged the current local repo into baselines tf2 branch, you could either sync from there or sync from my repo (since I will probably continue working on hindsight experience replay).

And let me know how I can help!

[schneimo]

@tanzhenyu
I have converted most of the files. The only thing I still need to convert is the Kronecker Factor Optimizer, which is a little bit harder. After I finished to convert the optimizer I want to do some testing. I'll try to complete the conversion in the next few days.

I sync already from your repo to be up to date with your commits.

[tanzhenyu]

@MoritzTaylor Ah yes, KFAC is a great optimizer and unfortunately we don't include them in TF core.

[tanzhenyu]

@josiahbjorgaard @seungjaeryanlee Gently ping and let me know if you need additional help.

[josiahbjorgaard]

@tanzhenyu Somebody else should take over mine. I haven't had a chance to get anywhere with it.

[tanzhenyu]

@josiahbjorgaard Alright.

[araffin]

I believe even google dopamine needs an update to tf 2.0

TF-Agents is prioritized over Dopamine - @dynamicwebpaige.

If anyone is interested, there's also Stable Baselines - a TensorFlow-based fork of OpenAI's Baselines: https://github.com/Stable-Baselines-Team/stable-baselines with ACER, A2C, ACKTR, HER, PPO2, SAC etc. Full blog post here.

Yes, we would like also to support tf 2.0 in the future for stable baselines.
In short, it is OpenAI baselines with a nice API (scikit learn like) + documentation + new algorithms (SAC, TD3)
The main repo url: https://github.com/hill-a/stable-baselines
Documentation: https://stable-baselines.readthedocs.io/en/master/

Corresponding issue: hill-a/stable-baselines#366

@dynamicwebpaige Should I open a new issue for that?

[avnishn]

@tanzhenyu if there are any open issues that a community contributor can help with, I'd be more than happy to help!

[tanzhenyu]

@avnishn can you help with converting ACER to TF2?

[avnishn]

Sure thing! Is there a date that you need it by? @tanzhenyu

[tanzhenyu]

@avnishn Probably in a month or two. I don't have much bandwidth to make this at the moment, but ACER should be pretty straightforward compared to other algs. Let me know if you run into any issues.

[schneimo]

@tanzhenyu
Sorry that I did not finished already, even I said last time, that I am finish soon. Fortunately I have more time now.
I reach out because I want to ask if it is possible to rewrite KFAC from ground up, since it is not as obvious to transform the TF1 code of it to TF2 as I thought? I think rewriting it, could be easier here, since I've tried to preserve most of the main code so far.

[tanzhenyu]

@MoritzTaylor Yes that makes sense to rewrite while maintaining most of the main logic.

[siyavash]

@tanzhenyu It seems nobody has taken TRPO, I'd be happy to help with that or any other alogs.

[tanzhenyu]

@tanzhenyu It seems nobody has taken TRPO, I'd be happy to help with that or any other alogs.

Here.

[siyavash]

@tanzhenyu it seems TRPO is rewritten. If there are any open issues that I can help with, I'd be very happy to help!

[GIS-PuppetMaster]

@tanzhenyu
对不起，我还没完成，即使我上次说过，我也快完成了。幸运的是我现在有更多时间。
我伸出援手是因为我想问是否有可能从头开始重写KFAC，因为将其TF1代码转换为TF2并不像我想的那么明显？我认为在这里重写它可能会更容易，因为到目前为止我一直尝试保留大多数主要代码。

Have you finished KFAC yet? I cloned your KFAC.py and got an error
AttributeError: 'KfacOptimizer' object has no attribute '_update_stats_op'

[nbro]

@araffin It seems that you don't have big plans to upgrade stable-baselines to TF 2, although you already started another Github project, but you are apparently investing your time more on stable-baselines-3 with PyTorch. TF 2.0 is more widely used than PyTorch, so I feel this is somehow a mistake.

[araffin]

@araffin It seems that you don't have big plans to upgrade stable-baselines to TF 2, although you already started another Github project, but you are apparently investing your time more on stable-baselines-3 with PyTorch. TF 2.0 is more widely used than PyTorch, so I feel this is somehow a mistake.

i already replied to you in the SB issue, but for others the answers about our choice to use PyTorch are in hill-a/stable-baselines#366 and hill-a/stable-baselines#733

[tanzhenyu]

I'm gonna close this issue, given the OpenAI Baseline conversions is complete under branch tf-2. We don't have intermediate plans to support StableBaselines.