Add modality info and transforms for raw RGB, tokens, video descriptions, video transcripts, and video bounding boxes. #1

kdu4108 · 2024-06-21T11:43:29Z

Goal: Inputs from each modality, after being loaded from their paths, need to be transformed so that they can be used downstream for the model. These transforms are defined in modality_transforms.py. modality_info.MODALITY_TRANSFORMS defines the mapping from different modality IDs (e.g., "tok_rgb") to its correspnding transformation. In this PR, we need to define new transformations for each video modality according to be represented as we have described in https://docs.google.com/presentation/d/1AY3QV1N_hoi9aXI1r8QTqrNmDK9LyorgJDQMPWb8hBo/edit#slide=id.g2e696416940_0_144.

Summary: So far, I've implemented but NOT tested (like at all, because I haven't done any of the data setup yet :/ this is entirely just me spitballing on what the code should maybe look like and probably has a million little runtime errors, sorry :(( )

(a) the modality info mapping (I feel pretty good about this, but @garjania please take a look to see if that looks okay!) and
(b) tentative implementations for transforms for VideoTokTransform, VideoDescriptionTransform, VideoTranscriptTransform, and VideoDetTransform. Of these, I feel best about VideoDetTransform and VideoTokTransform. I have several TODOs in the code indicating questions. Most of these revolve around what kind of data augmentations (image_augment function) we should keep for the different video transforms. Also the input format and desired output shapes for some of these are not entirely clear to me (i.e., should the outputs already be unrolled from (num_frames, num_tokens_per_frame) to (num_frames * num_tokens_per_frame,)?), so any input there would be great!

Remaining TODOs: Implement the raw RGB, clarify/confirm the questions, update the implementation accordingly, implement runtime fixes and test to make sure the transforms can all run correctly.
Reviewers: @garjania @vesteinn @yahya

kdu4108 · 2024-06-21T11:45:19Z

Oops, I apologize for the extra files. Everythign except modality_transforms and modality_info.py should just be comments, so feel free to ignore them.

kdu4108 · 2024-07-03T13:19:42Z

fourm/data/modality_transforms.py

+    """
+
+    # raise NotImplementedError("I'm not ")
+    def load(self, path):


Note: webd formats are not loaded using this function. This is only applicable for non-webd data storage formats (Ali).

ml-4m/fourm/data/unified_datasets.py

Line 370 in 4c2c9a5

datapipe = wds.DataPipeline(

- the pipeline for loading webd and running the preprocess/augment/postprocess.

ml-4m/fourm/data/unified_datasets.py

Line 162 in 4c2c9a5

def wds_decoder(key, value):

is the thing that actually loads the specific files from the webd tars

All modalities should use webd and be compressed into webd/tar files.

This means the load function will (probably) never be called for webd/our video-based modalities

kdu4108 · 2024-07-03T13:28:07Z

fourm/data/modality_transforms.py

+class VideoTokTransform(AbstractTransform):
+    """
+    Assume input tokens is an ndarray of shape (num_frames, num_tokens_per_frame).
+    Transform the tokens to a torch tensor of shape (num_frames * num_tokens,).


Maybe don't flatten it here because this will make it a little bit more annoying to add the temporal embedding. Instead we can unflatten after adding the temporal embedding.

kdu4108 · 2024-07-03T13:28:59Z

fourm/data/modality_transforms.py

+    def preprocess(self, sample):
+        return sample
+
+    def image_augment(


Role of this: do the augmentation during the tokenization step here

TokTransform's v input is actually a list of tokenized inputs which are different augmentations of the same input. So that transform requires an index to select one of the augmented inputs.

So here, we should assume no augmentations but keep the input v as a list of augmentations. but in this case since there's none just get v[0] always.

kdu4108 · 2024-07-03T13:49:26Z

fourm/data/modality_transforms.py

+    def image_augment(
+        self,
+        bboxes_per_frame: List[List[Tuple]],
+        crop_coords: Tuple,


No rand_aug_idx for this because the other parameters actually specify which aug to apply

so we have the same crop for each frame already!

kdu4108 · 2024-07-03T13:52:24Z

fourm/data/modality_transforms.py

+            The innermost tuple is the actual representation of a bounding box instance (of the form [xmin, ymin, xmax, ymax, class_name, score]).
+        """
+        instances = [frame["instances"] for frame in sample]
+        return self.convert_detection_instance(instances)


bugfix: wrap this in a list

kdu4108 · 2024-07-03T13:55:44Z

fourm/data/modality_transforms.py

+        for i, bboxes in enumerate(bboxes_per_frame):
+            bboxes_str = self.convert_bboxes_to_string(bboxes_per_frame)
+            output_str += bboxes_str
+            output_str += f"<frame_{i+1}_token>"


these new special tokens need to be added to the tokenizer jsons -- https://github.com/swiss-ai/ml-4m/blob/main/fourm/utils/tokenizer/trained/text_tokenizer_4m_wordpiece_30k.json (we can manually add these tokens to the json file)

kdu4108 · 2024-07-03T13:56:07Z

fourm/data/modality_transforms.py

+            bboxes_str = self.convert_bboxes_to_string(bboxes_per_frame)
+            output_str += bboxes_str
+            output_str += f"<frame_{i+1}_token>"
+        output_str += "<eos_token>"  # TODO: Do we need to explicitly add EOS token here? If yes, we need to do this so it adds the actual eos_token str of the model not just this hardcoded string.


let's reuse the eos token from the text tokenizer

https://github.com/swiss-ai/ml-4m/blob/4c2c9a56a9e2cd3e94316e766028c71bb6e248d8/fourm/utils/tokenizer/trained/text_tokenizer_4m_wordpiece_30k.json#L35

kdu4108 · 2024-07-03T13:57:56Z

fourm/data/modality_transforms.py

+        print(
+            "WARNING: no augmentations implemented for transcripts yet. Decide whether to augment/what these should be and then remove this warning."
+        )
+        return val


No augmentations for now. val is actually the caption.

kdu4108 · 2024-07-03T14:00:54Z

fourm/data/modality_transforms.py

+    def preprocess(self, sample):
+        return sample
+
+    def image_augment(


Precondition: assume val does not have any augmentations (thus list[dict])
postcondition: output is type list[dict]

…ionTransform, VideoTranscriptTransform, VideoDetTransform. Added a stub for videoRGBTransform. also sneak in a precommit

… keys and their mappings to transforms

This was referenced Jun 21, 2024

[PARENT ISSUE] Implement the temporal changes in 4M to account for video #2

Open

[PARENT ISSUE] Data preprocessing and pseudolabeling #3

Open

kdu4108 commented Jul 3, 2024

View reviewed changes

kdu4108 force-pushed the kdu/temporal branch from 74b9e43 to 03baf5f Compare August 1, 2024 20:55

kdu4108 added 9 commits August 5, 2024 13:26

Add notes from Ali

5e3396b

Add notes

a482910

Add modality info and transforms for VideoTokTransform, VideoDescript…

1d18ca2

…ionTransform, VideoTranscriptTransform, VideoDetTransform. Added a stub for videoRGBTransform. also sneak in a precommit

update the MODALITY_TRANSFORMS dict to include the new video modality…

bebdd6d

… keys and their mappings to transforms

remove precommit because it makes diffs hard

71769d1

Add initial configs for running training onvideo

d87939f

Add data config with video_det for now

ab2a10c

Fix more configs

87a2aa9

Update configs to work with a filtered raw + add debug script with pdb

e686e66

kdu4108 force-pushed the kdu/temporal branch from a23098e to e686e66 Compare August 5, 2024 11:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add modality info and transforms for raw RGB, tokens, video descriptions, video transcripts, and video bounding boxes. #1

Add modality info and transforms for raw RGB, tokens, video descriptions, video transcripts, and video bounding boxes. #1

kdu4108 commented Jun 21, 2024

kdu4108 commented Jun 21, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

kdu4108 Jul 3, 2024

Add modality info and transforms for raw RGB, tokens, video descriptions, video transcripts, and video bounding boxes. #1

Are you sure you want to change the base?

Add modality info and transforms for raw RGB, tokens, video descriptions, video transcripts, and video bounding boxes. #1

Conversation

kdu4108 commented Jun 21, 2024

kdu4108 commented Jun 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment