Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Continous CQL loss logging and aligning with discrete logging #317

Closed
wants to merge 21 commits into from
Closed

Continous CQL loss logging and aligning with discrete logging #317

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
4a6edc9
tracking of cql regularisation for continuous cql
joshuaspear Aug 4, 2023
5b7185c
updated for linting and formatting
joshuaspear Aug 4, 2023
7fd0a37
overwriting dr3 pull and aligning cql logging
joshuaspear Aug 10, 2023
a52651e
updated formatting
joshuaspear Aug 10, 2023
a46d73f
update gitignore
joshuaspear Aug 10, 2023
7255159
Fix custom network docs
takuseno Aug 11, 2023
294544d
Fix typo (#321)
asmith26 Aug 11, 2023
b8263d4
Add TPU example notebook
takuseno Aug 12, 2023
3ec25d7
Simpify Q functions with forwarders
takuseno Aug 14, 2023
bae6777
Add Checkpointer
takuseno Aug 14, 2023
817810d
Refactor models with Modules
takuseno Aug 14, 2023
4ba297f
Refactor reset_optimizer_states
takuseno Aug 14, 2023
2d730ef
Refactor to support torch.compile
takuseno Aug 15, 2023
9d4f928
Move update logics to impl
takuseno Aug 23, 2023
0874990
merging refactor_loss branch and resolved conflicts with cql and cql_…
joshuaspear Aug 29, 2023
2846d20
corrected formatting
joshuaspear Aug 29, 2023
8e5aec8
generalised loss dataclass
joshuaspear Aug 29, 2023
fa90b6c
added method returns
joshuaspear Aug 29, 2023
b1c1e62
updated formatting
joshuaspear Aug 29, 2023
5bd59a1
updated bcq loss
joshuaspear Aug 29, 2023
80b9579
updated formatting
joshuaspear Aug 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ docs/d3rlpy*.rst
docs/modules.rst
docs/references/generated
coverage.xml
.coverage
.coverage*
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change necessary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When running tests, I seem to get .coverage... files with references to my local system. Maybe it's the way I'm running the test?

.mypy_cache
.ipynb_checkpoints
build
Expand Down
3 changes: 2 additions & 1 deletion d3rlpy/algos/qlearning/cql.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,8 +206,9 @@ def inner_update(self, batch: TorchMiniBatch) -> Dict[str, float]:
alpha_loss, alpha = self._impl.update_alpha(batch)
metrics.update({"alpha_loss": alpha_loss, "alpha": alpha})

critic_loss = self._impl.update_critic(batch)
critic_loss, conservative_loss = self._impl.update_critic(batch)
metrics.update({"critic_loss": critic_loss})
metrics.update({"conservative_loss": conservative_loss})

actor_loss = self._impl.update_actor(batch)
metrics.update({"actor_loss": actor_loss})
Expand Down
21 changes: 19 additions & 2 deletions d3rlpy/algos/qlearning/torch/cql_impl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,23 @@ def compute_critic_loss(
conservative_loss = self._compute_conservative_loss(
batch.observations, batch.actions, batch.next_observations
)
return loss + conservative_loss
return loss + conservative_loss, conservative_loss

@train_api
def update_critic(self, batch: TorchMiniBatch) -> np.array:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is altering a signature of update_critic, which is def update_critic(self, batch: TorchMiniBatch) -> float:. Thus, we actually need to refactor these methods first. I'll take the first pass to return Dict[str, float]. Then, you can make changes on top of it. Sorry for the inconvenience. I'll let you know when it's done.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The interface has been updated in this commit: 67723be . Please check.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@takuseno - I'm sorry for the delay! All looks good to me except my one comment on the AWAC d3rlpy/algos/qlearning/awac.py file. Cheers

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@takuseno is there anything else you need from me at all for the PR? :)

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joshuaspear Thanks for the contribution. I've changed the target branch to refactor_loss because I've made some new changes to master branch. I'm seeing conflicts between your PR and refactor_loss branch. Could you resolve it? Here is an example instruction to merge refactor_loss branch to your branch.

$ git fetch upstream
$ git checkout master
$ git merge upstream/refactor_loss

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@takuseno no probs - will do

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@takuseno have merged the branches - I included a couple more data classes for the loss outputs FYI

self._critic_optim.zero_grad()

q_tpn = self.compute_target(batch)

loss, cql_loss = self.compute_critic_loss(batch, q_tpn)

loss.backward()
self._critic_optim.step()

critic_loss = float(loss.cpu().detach().numpy())
cql_loss = float(cql_loss.cpu().detach().numpy())
res = np.array([critic_loss, cql_loss])
return res

@train_api
def update_alpha(self, batch: TorchMiniBatch) -> Tuple[float, float]:
Expand Down Expand Up @@ -221,7 +237,8 @@ def compute_loss(
conservative_loss = self._compute_conservative_loss(
batch.observations, batch.actions.long()
)
return loss + self._alpha * conservative_loss, conservative_loss
cql_loss = self._alpha * conservative_loss
return loss + cql_loss, cql_loss

def _compute_conservative_loss(
self, obs_t: torch.Tensor, act_t: torch.Tensor
Expand Down
Loading