Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Paths and file management #61

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Conversation

alex404
Copy link
Contributor

@alex404 alex404 commented Oct 31, 2024

So this does quite a fair bit of refactoring.

  • I setup the analyses in statistics.py to return dataclass instances to make things more manageable and readable.
  • I also made a push to a more pathlib style of directory management, in particular in the runner submodules.
  • Generally going for a style where the at the top of the framework command (e.g. intialize, analyze or train) all necessary variables are extracted out of the DictConfig.
  • Saving analysis results to disk was the original push for this, and that's done too.

I definitely wasn't perfectly complete or consistent with how I did this, but I think things are significantly better organized than they were before. Ideally this provides a good reference for how to factor/refactor such code in other parts of the library. Definitely open to feedback on review.x

Resolves #43

@alex404
Copy link
Contributor Author

alex404 commented Oct 31, 2024

This probably seems more complicated than it is. The bulk of the changes are moving blocks of arguments around into dataclasses.

@alex404
Copy link
Contributor Author

alex404 commented Oct 31, 2024

Actually hold on just discovered a bug

@alex404
Copy link
Contributor Author

alex404 commented Oct 31, 2024

Okay seems to be fine now

Copy link
Contributor

@fabioseel fabioseel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems mostly fine, just want to hear your comment thoughts on my comment in plot (moving the inner function out to module scope)
and in statistics (using the defined dataclasses already within the functions)

fig, axs = plt.subplots(num_transforms, 1, figsize=(20, 5 * num_transforms))
if num_transforms == 1:
axs = [axs]

transform_index = 0

def make_image_grid(arrays: List[FloatArray], nrow: int) -> FloatArray:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from looking at it, I think this function might be worth moving out of the local scope (but perhaps private to the module)?
Or do you think it's only of value within this method?

estimates: List[Tuple[FloatArray, int]]


@dataclass
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, I like the approach of typing this more strictly here
But do we need this 'ReconstructionStatistics'?
I feel like it would be enough to have a tuple returned / as a parameter where needed 🤔

@@ -34,7 +102,7 @@ def transform_base_images(
src, _ = base_dataset[np.random.randint(base_len)]
images.append(src)

results: Dict[str, Dict[str, Dict[float, List[Tensor]]]] = {
results: Dict[str, Dict[str, Dict[float, List[FloatArray]]]] = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use TransformStatistics already here?
Defining this results dict first this way makes making changes to this later harder, as also the TransformStatistics type will then have to be adjusted (same in other functions)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

really like the move towards a more traceable config system :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve directory and file management
2 participants