Question - Loss computation

[t-sanchez]

Hello Junshen,

I had a question on the computation of the losses. I saw in your code that you detach the bias and c computation for the computation of the variance.

NeSVoR/nesvor/nesvor/models.py

Lines 310 to 312 in 671c023

	var = (bias_detach * psf * var).mean(-1)
	var = c.detach() * var
	var = var**2

Is there a particular reason for it? Did you observe something like a more stable training?

Thanks for your help!

Best,
Thomas

[daviddmc]

Hi Thomas,

Yes. Recently, we found that detaching the variables can stabilize the training in some cases. But for most of the data we evaluated, this doesn't change the performance much.