Tuning - Augmentation Subsets Support

[klemen1999]

Adds ability to randomly select subset of augmentations each tuning run. It can be defined in the config like this:

tuner:
  params:
    trainer.preprocessing.augmentations_subset: [["Defocus", "Sharpen", "Flip"], 2] # randomly selects 2 augs to be active in the run

Note: Currently this is only supported for augmentations.

[github-actions]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
4900	3792	77%	0%	🟢

New Files

No new covered files...

Modified Files

File	Coverage	Status
luxonis_train/core/tuner.py	80%	🟢
luxonis_train/utils/config.py	95%	🟢
TOTAL	88%	🟢

updated for commit: 4abe9f8 by action🐍

[github-actions]

Test Results

  4 files    4 suites   1h 0m 32s ⏱️
 68 tests  43 ✅  25 💤 0 ❌
272 runs  172 ✅ 100 💤 0 ❌

Results for commit 4abe9f8.

♻️ This comment has been updated with latest results.

[kozlov721]

LGTM

[tersekmatija]

Won't have time to review the code, just a conceptual question: What is the behavior given we now have both is_active and subset option for augmentations? Do inactive augmentations become active?

It would be good to document this somewhere.

[klemen1999]

Won't have time to review the code, just a conceptual question: What is the behavior given we now have both is_active and subset option for augmentations? Do inactive augmentations become active?

It would be good to document this somewhere.

The _subset sets is_active to True for those augs in the set that are chosen and False for other ones in the set. Other augs not specified in the Tuner set (but defined in the trainer preprocessing block) are left as they are. I've added additional clarification in the README.

[kozlov721]

Can we also tune the size of the selected subset?

[klemen1999]

Can we also tune the size of the selected subset?

Not really. The way tuner params work is they override existing Config params each trial. Since Size of the subset is not a Config param we can't override it. Quick solution could be to check for special tuner param key an then generate random int in desired range to use as subset size but this solution seems a bit dirty. Do you have something nicer in mind?

[kozlov721]

How are we preventing it from selecting a subset that was already selected in a previous run?

[kozlov721]

For this question I assume that typically we would use the augmentations tuning with all the other tunings disabled to get purely augmentation results. Would that be correct?

[klemen1999]

Good point, there isn't really any prevention in place for selecting same subset in subsequent runs. We could change it to first get all possible combinations and then loop through them each run. But then number of trials should also be set to number of combinations so we go through all of them. I guess it depends on what would be the typical usecase for this aug subset sampling.

As you mentioned below, perhaps looping over whole powerset is a more sensible impelmentation since normally we shouldn't be limited by the number of augmentations used and instead use all of those that produce better results. But to see if augmentation works you normally have to train the model for more epochs (at the beginning hard augs could produce worse results but with time perhaps those are the ones that actually improve acc on test set) and going over whole powerset could take a while (maybe add ability to limit minimum size of the subset to prune smaller ones).
CC: @tersekmatija on what would be the intended usecase of aug subsampling.

[kozlov721]

When I'm thinking more about the augmentation tuning, I think this will actually be more complex and will need more optimization. For now, I can think of a few optimizations if we'll work with some assumptions about the augmentations' effect on the training.

My idea: I think we can assume the order of augmentations should not (reasonably) matter, so if augmentation $A$ increases the model's performance, it doesn't matter where in the pipeline it's placed so we can lock it in place and continue only with the remaining augmentations and one-smaller subset size.
This would decrease the number of possibilities from $n^k$ to $n \choose k$ ($n$ being the number of all augmentations and $k$ being the subset size), but that's still way too many for any reasonable usage. We support ~80 augmentations and even with a subset of size only 5, there's 22 milion combinations to go through.

However, as an extension of the above, if augmentation $A$ did not increase the performance, we can discard it for the rest of the tuning. That's because if $A$ didn't improve, a different augmentation $B$ did improve, and the order doesn't matter, then $A \circ B \simeq B \simeq B \circ A$, so there's no need to try $A$ ever again.
This would bring the number of combinations all the way down to $n$, even with tunable subset size.

More digestible in code:

best_augmentations = []
for a in all_augmentations:
    if improves(a):
        best_augmentations.append(a)
return best_augmentations

Someone would need to double-check my math on this though (and test whether the assumption even holds in the first place).

[tersekmatija]

Math looks correct :) The challenge here is if you use more augmentations at once. If you use a and b and they improve the accuracy, you don't know whether contribution was made by a or b. This means that in any case you need to run at least n+1 (where n=80) combinations to find out which show improvement and which do not. Furthermore, I think the challenge then might be that single augmentations improve the performance, but a combination of them decreases it (image gets too corrupted to be useful).

I still think it should be up to the user to define which augmentations are reasonable, and then merely test a few combinations to check whether they are useful or not. I think in the above scenario it's also hard to define what k is?

[kozlov721]

Can we also tune the size of the selected subset?

Not really. The way tuner params work is they override existing Config params each trial. Since Size of the subset is not a Config param we can't override it. Quick solution could be to check for special tuner param key an then generate random int in desired range to use as subset size but this solution seems a bit dirty. Do you have something nicer in mind?

I think instead of tuning the subset size, we can add another option like _powerset that would go through all possible subsets. This might quickly get too resource intensive though, so not sure if it even makes sense to implement.

[tersekmatija]

Yeah I'd skip for now. You can still achieve this with active flag on aug if you wanted to?