Layers that are never used in ByteLatentTransformer class

[isayoften]

Hello! Please check this line :

blt/bytelatent/model/blt.py

Line 834 in 6ffeb66

self.layers = nn.ModuleList(

It seems that these layers are here by mistake. All transformerblocks are defined inside the local and global parts. There is no self.layers in the forward at all also

[isayoften]

The same question about:

blt/bytelatent/model/blt.py

Line 831 in 6ffeb66

self.tok_embeddings = torch.nn.Embedding(args.vocab_size, args.dim)

and:

blt/bytelatent/model/blt.py

Line 854 in 6ffeb66

self.output = nn.Linear(args.dim, args.vocab_size, bias=False)

because we already have tok_embeds and unembeds in encoder and decoder:

blt/bytelatent/model/local_models.py

Line 81 in 6ffeb66

self.tok_embeddings = nn.Embedding(self.vocab_size, args.dim)

blt/bytelatent/model/local_models.py

Line 330 in 6ffeb66

self.output = nn.Linear(

[isayoften]

Also, we initialise tok_embeds twice because of super() call in the decoder:

blt/bytelatent/model/local_models.py

Line 308 in 6ffeb66

super().__init__(args)

[Vectorrent]

There is a lot of orphaned code in this project. For example:

blt/bytelatent/model/local_models.py

Line 83 in a809259

self.pos_embeddings = nn.Embedding(args.max_length, args.dim)

Here, positional embeddings are created when args.use_rope == False - yet, these embeddings are never actually used anywhere.

[EntilZha]

Thanks for reports on this, we'll go through and do some cleanup. Feel free to comment on any other spots.

[Vectorrent]

Weight-tying will not work here, because self.embeddings are not valid weights:

blt/bytelatent/transformer.py

Line 72 in 7622d28

if args.weight_tying:

[Vectorrent]

Here at line 824, n-gram embeddings are initialized, only to be invalidated just after, at line 838:

blt/bytelatent/model/blt.py

Line 824 in 7044771

self.encoder_ngram_embedding = init_embeddings(

Honestly, lines 838-849 can probably be removed, since it seems like the init_embeddings at line 824 already implements that logic gracefully.

At line 959, h_cross branches are redundant, since they always evaluate to None:

h_cross = None
(h_encoder, h_cross), cache_encoder = self.local_encoder(
    tokens=local_encoder_tokens,
    embeds=local_encoder_embeds,
    patch_embeds=h_cross if self.cross_attn_encoder else None,
    cross_mask=cross_attn_mask_enc,
    num_patches=patch_lengths.shape[1],
    patch_ids=patch_ids,
)

[Vectorrent]

Another one here:

blt/bytelatent/model/local_models.py

Line 92 in 7044771

self.token_embedding_projection = (

self.token_embedding_projection can be created here, but is never used anywhere.