Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow training a model on multiple annotations #8071

Merged
merged 22 commits into from
Sep 25, 2024
Merged
Changes from 3 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
b6aefe3
implement train-model for multiple annotations (WIP)
philippotto Sep 9, 2024
618cd38
further refactoring of train model form
philippotto Sep 11, 2024
8a8a0cc
make onFinish work for multi-annotation training
philippotto Sep 11, 2024
89d1c78
Merge branch 'master' of github.com:scalableminds/webknossos into mul…
philippotto Sep 11, 2024
7e4e3ea
clean up
philippotto Sep 11, 2024
a927a7e
fix annotationId and csv validation
philippotto Sep 11, 2024
9167b0e
fix training model from annotation view
philippotto Sep 11, 2024
79f859d
Merge branch 'master' into multi-anno-training
philippotto Sep 12, 2024
732a22f
clean up
philippotto Sep 12, 2024
b405d11
Merge branch 'master' of github.com:scalableminds/webknossos into mul…
Sep 23, 2024
6eccb7e
WIP: Apply feedback
Sep 23, 2024
4926361
make bbox checking work while not refetching tracings
Sep 24, 2024
77e0439
some cleanup
Sep 24, 2024
f3b4c16
apply missing pr feedback
Sep 24, 2024
f748d6b
Merge branch 'master' of github.com:scalableminds/webknossos into mul…
Sep 24, 2024
66ecfef
improve ai modal csv input parsing
Sep 24, 2024
e2546b9
Update frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx
MichaelBuessemeyer Sep 24, 2024
acc2b2d
apply feedback
Sep 24, 2024
463de3c
Merge branch 'multi-anno-training' of github.com:scalableminds/webkno…
Sep 24, 2024
75aa6b1
Update frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx
MichaelBuessemeyer Sep 25, 2024
6e378a3
Merge branch 'master' into multi-anno-training
MichaelBuessemeyer Sep 25, 2024
1045a03
Merge branch 'master' into multi-anno-training
MichaelBuessemeyer Sep 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 21 additions & 11 deletions frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,13 @@ import { convertUserBoundingBoxesFromServerToFrontend } from "oxalis/model/reduc
const { TextArea } = Input;
const FormItem = Form.Item;

// This type is used for GenericAnnotation = HybridTracing | APIAnnotation as in case of multi annotation training,
// only the APIAnnotations of the given annotations to train on are loaded from the backend.
// Thus, the code needs to handle both HybridTracing | APIAnnotation where APIAnnotation is missing some information.
// Therefore, volumeTracings with the matching volumeTracingResolutions are needed to get more details on each volume annotation layer and its resolutions.
// The userBoundingBoxes are needed for checking for equal bounding box sizes. As training on fallback data is supported and an annotation is not required to have VolumeTracings,
// it is necessary to save userBoundingBoxes separately and not load them from volumeTracings entries to support skeleton only annotations.
// Note that a copy the userBoundingBoxes is included in each volume and skeleton tracing of an annotation. Thus, it doesn't matter from which the userBoundingBoxes are taken.
MichaelBuessemeyer marked this conversation as resolved.
Show resolved Hide resolved
export type AnnotationInfoForAIJob<GenericAnnotation> = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please note that I also added this comment. It hopefully explains why all these additional properties (volumeTracings, userBoundingBoxes, volumeTracingResolutions) are needed. Please tell me your opinion on whether you understand what I want to express and maybe on wording improvements 🙏

annotation: GenericAnnotation;
dataset: APIDataset;
Expand Down Expand Up @@ -217,8 +224,9 @@ export function TrainAiModelTab<GenericAnnotation extends APIAnnotation | Hybrid
);

const { areSomeAnnotationsInvalid, invalidAnnotationsReason } =
areAllAnnotationsInvalid(annotationInfos);
const { areSomeBBoxesInvalid, invalidBBoxesReason } = areBoundingBoxesInvalid(userBoundingBoxes);
areInvalidAnnotationsIncluded(annotationInfos);
const { areSomeBBoxesInvalid, invalidBBoxesReason } =
areInvalidBoundingBoxesIncluded(userBoundingBoxes);
const invalidReasons = [invalidAnnotationsReason, invalidBBoxesReason]
.filter((reason) => reason)
.join("\n");
Expand All @@ -234,12 +242,14 @@ export function TrainAiModelTab<GenericAnnotation extends APIAnnotation | Hybrid
<AiModelNameFormItem />
<AiModelCategoryFormItem />

{annotationInfos.map(({ annotation, dataset }, idx) => {
{annotationInfos.map(({ annotation, dataset, volumeTracings }, idx) => {
const segmentationLayerNames = _.uniq([
// Only consider the layers that are not volume layers (these don't have a tracing id).
// Only consider the layers that are not volume layers (these aren't a fallback layer in one of the volume tracings).
// Add actual volume layers below.
...getSegmentationLayers(dataset)
.filter((layer) => layer.tracingId == null)
.filter(
(layer) => !volumeTracings.find((tracing) => tracing.fallbackLayer === layer.name),
)
.map((layer) => layer.name),
// Add volume layers here.
...annotation.annotationLayers
Expand Down Expand Up @@ -375,7 +385,7 @@ export function CollapsibleWorkflowYamlEditor({
);
}

function areAllAnnotationsInvalid<T extends HybridTracing | APIAnnotation>(
function areInvalidAnnotationsIncluded<T extends HybridTracing | APIAnnotation>(
annotationsWithDatasets: Array<AnnotationInfoForAIJob<T>>,
): {
areSomeAnnotationsInvalid: boolean;
Expand Down Expand Up @@ -404,7 +414,7 @@ function areAllAnnotationsInvalid<T extends HybridTracing | APIAnnotation>(
return { areSomeAnnotationsInvalid: false, invalidAnnotationsReason: null };
}

function areBoundingBoxesInvalid(userBoundingBoxes: UserBoundingBox[]): {
function areInvalidBoundingBoxesIncluded(userBoundingBoxes: UserBoundingBox[]): {
areSomeBBoxesInvalid: boolean;
invalidBBoxesReason: string | null;
} {
Expand Down Expand Up @@ -476,10 +486,10 @@ function AnnotationsCsvInput({
getTracingForAnnotationType(annotation, layer) as Promise<ServerVolumeTracing>,
),
);
// TODO: make bboxs a member again
const volumeTracings = volumeServerTracings.map((tracing) =>
serverVolumeToClientVolumeTracing(tracing),
);
// A copy of the user bounding boxes of an annotation is saved in every tracing. In case no volume tracing exists, the skeleton tracing is checked.
let userBoundingBoxes = volumeTracings[0]?.userBoundingBoxes;
if (!userBoundingBoxes) {
const skeletonLayer = annotation.annotationLayers.find(
Expand All @@ -491,12 +501,12 @@ function AnnotationsCsvInput({
skeletonTracing.userBoundingBoxes,
);
} else {
new Error(`Annotation ${annotation.id} has neither a volume nor a skeleton layer`);
throw new Error(
`Annotation ${annotation.id} has neither a volume nor a skeleton layer`,
);
}
}

console.log(volumeTracings); // TODOM remove

return {
annotation,
dataset,
Expand Down