Jupyter Notebook: Unable to release GPU memory after training without restarting the kernel

[animeshkumarpaul]

After running the training cell in Jupter notebook, the code gives gpu outofmemory error.

How I can release memory before starting training without restarting the kernel?

net = NeuralNetClassifier(
    ClassifierModule,
    module__layer_sizes=[n1, n2, n3, 2],
    module__input_size=X_train.shape[1],
    max_epochs=200,
    batch_size=50000,
    criterion=nn.CrossEntropyLoss, 
    criterion__reduction='mean',
    optimizer=torch.optim.Adam,
    lr=0.01,
    optimizer__weight_decay=1e-4,
    iterator_train__shuffle=False,
    iterator_train__num_workers=1,
    iterator_train__sampler=sampler_train,
    
    device='cuda:0',
    callbacks=[early_stopping,lr_scheduler,initializer_cb, checkpoint_cb,EpochScoring_cb_train,EpochScoring_cb_valid],
    train_split=predefined_split(valid_ds),
)

net.fit(X_train,y_train)

torch.cuda.empty_cache() does not work here

[BenjaminBossan]

batch_size=50000,

Oh I see :D

I'm not sure if that situation is recoverable or not. After torch.cuda.empty_cache(), you could try import gc; gc.collect() and see if that works. You may also have to delete all references first (del net in your case). LMK if any of this works.

Do you have an idea of the same thing would not happen with pure PyTorch? It would be good to know in case there is some form of memory leak.

[animeshkumarpaul]

I have tried import gc; gc.collect() and del net also, but it did not work.
I am not 100% sure, but a long time ago I used the pure Pytorch, I believe there was no issue like this.

[BenjaminBossan]

In this case I have no further ideas. Maybe @ottonemo knows a trick.