Object class to color mapping for semantic segmentation

[zerolatnc]

I am trying to do semantic segmentation over data collected from AI2THOR scenes.

The event attribute object_id_to_color does have a mapping from object ids and object types in a particular scene to their respective RGB tuples. But I am collecting data from multiple scenes and so I was wondering if you have data that maps all the 125 object types mentioned in the documentation to RGB tuples.

[mattdeitke]

Hey @sashi2295!

I'll give a contrived example with smaller dimensions, since it's easier to visualize.

Consider that the (semantic) segmentation masks will appear in the following form, where 0, 1, and 2 are each correspond to separate object type segmentation:

import numpy as np

segmentation = np.random.randint(low=0, high=3, size=(3, 3))
array([[2, 2, 0],
       [1, 2, 1],
       [0, 0, 1]])

If the object_id_to_color does something like assign 2 to the Cup object type, we can then condition the segmentation where each entry is equal to 2, as in:

condition = (segmentation == 2)
array([[False,  True, False],
       [ True, False, False],
       [ True,  True,  True]])

Now, the RGB frame may look something like the following:

rgb_frame = np.random.randint(low=0, high=256, size=(3, 3))
array([[231,  31, 222],
       [101, 245, 223],
       [ 15, 101, 106]])

where multiplying it by the condition matrix gives us the masks

mask = rgb_frame * condition
array([[  0,   31,   0],
       [101,   0,    0],
       [ 15, 101, 106]])

If we then want to consistently store the image mask for each object, in something like a NumPy array with 125 separate entries, we can then create an initial array:

masks_array = np.zeros((125, 3, 3))
array([[[0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]],

       [[0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]],

       [[0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]],
.....
       [[0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]]])

where each of the 125 (3, 3) image matrices can be changed and indexed as follows:

masks_array[0] = mask

# masks_array now looks like:
array([[[  0.,   31.,   0.],
        [101.,   0.,    0.],
        [ 15., 101., 106.]],

       [[  0.,   0.,   0.],
        [  0.,   0.,   0.],
        [  0.,   0.,   0.]],

       [[  0.,   0.,   0.],
        [  0.,   0.,   0.],
        [  0.,   0.,   0.]],
.....
       [[  0.,   0.,   0.],
        [  0.,   0.,   0.],
        [  0.,   0.,   0.]]])

Here, masks_array[0] would be assigned to one object type, masks_array[1] would be assigned to another object type, and so on.

[mattdeitke]

I see what you're saying. With renderClassImage, you want to know the object color of each type across all rooms. For instance, ceilings are always this blueish color, faucets are always black, sinks are always green, etc.

I was able to generate the dict:

{'ScrubBrush': (222, 148, 80),
 'CoffeeMachine': (147, 71, 238),
 'RemoteControl': (187, 19, 208),
 'TissueBox': (98, 43, 249),
 'DishSponge': (166, 58, 136),
 'SideTable': (202, 45, 114),
 'CounterTop': (103, 209, 30),
 'Spoon': (235, 57, 90),
 'CoffeeTable': (18, 14, 75),
 'Bed': (209, 156, 101),
 'VacuumCleaner': (230, 13, 166),
 'Window': (200, 150, 134),
 'Watch': (242, 6, 88),
 'GarbageBag': (250, 186, 207),
 'PaperTowelRoll': (144, 173, 28),
 'TowelHolder': (232, 28, 225),
 'Television': (27, 245, 217),
 'Sofa': (82, 143, 39),
 'SoapBottle': (168, 222, 137),
 'Boots': (121, 126, 101),
 'SaltShaker': (36, 222, 26),
 'ArmChair': (96, 52, 68),
 'Desk': (14, 120, 179),
 'Pot': (132, 237, 87),
 'Tomato': (119, 189, 121),
 'LightSwitch': (11, 51, 121),
 'Curtains': (6, 62, 102),
 'DeskLamp': (99, 164, 25),
 'Lettuce': (203, 156, 88),
 'Pan': (246, 212, 161),
 'RoomDecor': (216, 96, 246),
 'CellPhone': (227, 98, 136),
 'Floor': (243, 246, 208),
 'BaseballBat': (171, 20, 38),
 'Dumbbell': (45, 57, 144),
 'Cup': (35, 71, 130),
 'ToiletPaper': (162, 204, 152),
 'SinkBasin': (80, 192, 81),
 'ShowerGlass': (80, 68, 237),
 'Fork': (54, 200, 25),
 'SprayBottle': (89, 126, 121),
 'TeddyBear': (229, 73, 134),
 'StoveBurner': (156, 249, 101),
 'FloorLamp': (253, 73, 35),
 'Stool': (13, 54, 156),
 'Potato': (187, 142, 9),
 'Toaster': (55, 33, 114),
 'Ottoman': (160, 135, 174),
 'HandTowel': (182, 187, 236),
 'Bottle': (64, 80, 115),
 'HandTowelHolder': (58, 218, 247),
 'Laptop': (20, 107, 222),
 'Kettle': (7, 83, 48),
 'Pillow': (217, 193, 130),
 'Candle': (233, 102, 178),
 'ShelvingUnit': (125, 226, 119),
 'Shelf': (39, 54, 158),
 'DogBed': (106, 193, 45),
 'Ladle': (174, 98, 216),
 'Faucet': (21, 38, 98),
 'ButterKnife': (135, 147, 55),
 'Knife': (211, 157, 122),
 'Dresser': (51, 128, 146),
 'AluminumFoil': (181, 163, 89),
 'Poster': (145, 87, 153),
 'Pen': (239, 130, 152),
 'TennisRacket': (138, 71, 107),
 'Towel': (170, 186, 210),
 'Newspaper': (19, 196, 2),
 'ShowerHead': (248, 167, 29),
 'Bowl': (209, 182, 193),
 'Pencil': (177, 226, 23),
 'Blinds': (214, 223, 197),
 'Footstool': (74, 187, 51),
 'ShowerCurtain': (60, 12, 39),
 'TVStand': (94, 234, 136),
 'Plate': (188, 154, 128),
 'Drawer': (155, 30, 210),
 'GarbageCan': (225, 40, 55),
 'Mirror': (36, 3, 222),
 'Book': (43, 31, 148),
 'BathtubBasin': (109, 206, 121),
 'Plunger': (74, 209, 56),
 'AlarmClock': (184, 20, 170),
 'Vase': (83, 152, 69),
 'SoapBar': (43, 97, 155),
 'Chair': (166, 13, 176),
 'BasketBall': (97, 58, 36),
 'LaundryHamper': (35, 109, 26),
 'Toilet': (21, 27, 163),
 'Sink': (30, 181, 88),
 'KeyChain': (27, 54, 18),
 'Microwave': (54, 96, 202),
 'Bathtub': (59, 170, 176),
 'ToiletPaperHanger': (124, 32, 10),
 'Box': (60, 252, 230),
 'Egg': (240, 75, 163),
 'Cloth': (110, 184, 56),
 'TableTopDecor': (126, 204, 158),
 'Cabinet': (210, 149, 89),
 'DiningTable': (83, 33, 33),
 'Fridge': (91, 156, 207),
 'Apple': (159, 98, 144),
 'WateringCan': (147, 67, 249),
 'CD': (65, 112, 172),
 'Mug': (8, 94, 186),
 'StoveKnob': (106, 252, 95),
 'CreditCard': (56, 235, 12),
 'Desktop': (35, 16, 64),
 'Safe': (198, 238, 160),
 'PepperShaker': (5, 204, 214),
 'Spatula': (30, 98, 242),
 'ShowerDoor': (36, 253, 61),
 'HousePlant': (73, 144, 213),
 'WineBottle': (53, 130, 252),
 'Statue': (243, 75, 41),
 'Bread': (18, 150, 252),
 'Painting': (40, 117, 236)}

This includes the 118 object types seen in iTHOR. Note that the website lists 125 object types, but it also includes objects that may be spawned in or change, such as AppleSliced, BreadSliced, EggCracked, LettuceSliced, PaperTowel, PotatoSliced, TargetCircle, TomatoSlice.

The code to reproduce is given as follows:

from ai2thor.controller import Controller
import ai2thor
assert ai2thor.__version__ == '2.4.22'

c = Controller(
    renderObjectImage=True,
    renderClassImage=True,
    scene='FloorPlan1')
scenes = [f'FloorPlan{i+j}' for i in {0, 200, 300, 400} for j in range(1, 31)]
huge_class_segmentation_map = dict()
valid_obj_types = set()
for scene in scenes:
    c.reset(scene)

    # add object types
    objs = c.last_event.metadata['objects']
    for obj in objs:
        valid_obj_types.add(obj['objectType'])

    # add mapping
    c.step('ToggleMapView')
    for k, v in c.last_event.object_id_to_color.items():
        if '|' not in k and '_' not in k:
            huge_class_segmentation_map[k] = v

obj_type_to_color = dict()
for obj_type in valid_obj_types:
    obj_type_to_color[obj_type] = huge_class_segmentation_map[obj_type]

Then to access the segmentations at runtime, simply run:

from ai2thor.controller import Controller
import ai2thor
assert ai2thor.__version__ == '2.4.22'

c = Controller(renderObjectImage=True)
seg = c.last_event.class_segmentation_frame  # (300, 300, 3) colored class segmentation frame (pictured above)

Note that I wouldn't count on the colors always being the same across all versions of AI2-THOR. For a version other than 2.4.22, rerunning the script provided should suffice.

[zerolatnc]

Thank you so much for this! This should do the trick for me :)

[zerolatnc]

On a related note, there are a lot of objects in huge_class_segmentation_map that are not present in valid_obj_types. I get that there are many objects like FP201:Cube.802, FP203:Cube.045 etc. in huge_class_segmentation_map which are not in valid_obj_types and this makes sense to me. There are also classes like Wall and Ceiling which are also not in valid_obj_types, but it can be argued that these are not really objects in the sense that you can't interact with them.

But there are also classes like AirConditioner, Bucket, WashingMachine etc. which are not present in valid_obj_types. Is there a criteria these objects must fulfill to be considered as valid object types?

[winthos]

Valid object types are only objects within THOR that are considered "sim objects." A sim object is any interactable object that has full object metadata, and can have actions performed on them. Not every object in a scene is a sim object, and many of the ones you are describing (AirConditioner, Bucket, Wall, etc.) are objects in the scene that do have physical collision but are not fully set up to have the functionality of a sim object. These non sim objects do still show up as having mappings because they are individual objects that have a mesh renderer, which is why you are still seeing them in the segmentation map.

Basically you are correct in that if an "object" is unable to be interacted with, it is not a full, valid object (sim object).

Additionally, the reason you are seeing names like FP203:Cube.045 and also AirConditioner grouped together as not valid objects is because the name of the object comes directly from the name of the game object the mesh is attached to in the Unity editor. Due to the way we add objects into scenes, the names of them can range from things like random cubes to seemingly more typified names like Bucket, but the name itself has no bearing on if it is a fully functional sim object or not.

[zerolatnc]

Got it, thanks! 👍