-
Notifications
You must be signed in to change notification settings - Fork 1
Global Workspace Module
Benjamin Devillers edited this page Mar 30, 2023
·
6 revisions
Let's again remember this diagram:
We will now focus on the Global Workspace module bim_gw.modules.gw.GlobalWorkspace
.
A previous chapter showed how to use the global workspace.
The GW module can encode multimodal representations into a common latent space.
It uses encoders and decoders (ev, el, dv, dl in the figure) to translate the unimodal representations to the GW state and back.
The encoder and decoder class used for the translation is defined by each module's module.domain_specs.workspace_encoder_cls
and module.domain_specs.workspace_decoder_cls
attributes.
By default, the encoders and decoders from this file are used.
These modules can be configured using the configuration values
-
global_workspace.hidden_size
to set the hidden dimensions of the encoder or decoder -
global_workspace.n_layers
to set the depth of the encoder and decoders -
global_workspace.z_size
to set the dimension of the GW representation.
When logging during training, the module calls:
-
unimodal_module.log_domain
for the "original" at the beginning of training (in theon_train_start
PytorchLightning hook). -
unimodal_module.log_domain_from_latent
for the translations, cycles, and demi-cycles.