Custom dataset input format for DPRNNTasNet model?

[denmonz]

I've been training on a custom dataset (~13 hours) over several epochs (~100) and noticed that my val_loss doesn't change much. I have my model outputting the results of the model to wandb, and even my gradients don't seem right.

I was thinking this may be because I don't have my input in the correct format?

In my torch.utils.data.Dataset class, under the get_item() method, I return two objects:

1. the mixture wave file, converted into float numpy arrays, of size (1, duration), and
2. each of the source wave files that are from the mixture, as a stacked numpy array of float arrays, of size (num_sources, duration)

Is this input format correct?

[mpariente]

It does sound correct to me. When you listen to the output, how does it sound?

[denmonz]

Thank you for confirming! My custom dataset doesn't use musical instruments, so it's a bit unique and can't be listened to for verification. I'll have to generate some waveforms/spectrograms and compare the target against the estimated targets to verify.

[denmonz]

While training, these are the metrics that are obtained at every 10 steps:

My concern is that the loss fluctuates between [25, 40], but the average loss doesn't change.

Even looking through the gradients, it looks like the weights oddly stay at 0:

My model was configured as such:

# Configure the model parameters
model = DPRNNTasNet(n_src=NUMBER_OF_SOURCES, num_layers=5, dropout=0.1, fb_name='stft', norm_type='cLN',
                    sample_rate=SAMPLE_RATE)
model.to(device, non_blocking=True)
loss = PITLossWrapper(pairwise_neg_sisdr, pit_from="pw_mtx") # (PITLossWrapper works with any loss function)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
system = System(model, optimizer, loss, train_dl, val_dl)

and trained using:

# Train the model
trainer = Trainer(max_epochs=100, gpus=1, log_every_n_steps=10, logger=wandb_logger, precision=16)
trainer.fit(system)

[mpariente]

Have you tried with float32 precision? I have no experience with 16-bit trainings.

[denmonz]

Have you tried with float32 precision? I have no experience with 16-bit trainings.

I ran my model with float32 precision, and I'm getting the same issue. I'm wondering if it's an issue inherent in my data or I just haven't trained for long enough. Will keep this thread open and get back to the discussion as I figure that out.