Effectiveness of the defense

[zimmerrol]

Hi,
thanks for your interesting paper and for releasing the source code for the experiments, too. When I saw your paper for the first time, I was really interested in it as the results looked very promising.

During the past few weeks, I have worked with this code and tried to reproduce your results. Doing this, I noticed some things in your code that I would like to talk about.

My first observation is about your experiments with the VGG network. I have noticed that you calculate the KL-divergence for all layers in the network but do not use this sum of the divergences but actually just the divergence of the last layer (cf. L39 of vgg_vi.py). Therefore, your proposed regularization is only applied to the last layer during training and not the entire network. This is different from the description you give in the ICLR paper. Did you do this on purpose? I tried to train the network using the divergence on the whole network and failed using the hyperparameters reported in the paper. Is there something I am overlooking?

My second observation is about your STL10 experiments, on which I focused after my initial observations of the issues with the VGG network. I noticed that one can improve the effectiveness of the PGD attack on the Bayesian network by averaging the gradients over multiple forward-backward passes before actually performing the PGD step on the input. Using this method I was able to decrease the accuracy of your model (based on the checkpoint you've uploaded). The following table compares the results you've listed in the ICLR print with the ones I obtained with my modified attack:

Attack	0.015	0.035	0.055	0.07
Your PGD Attack	51.8	37.6	27.2	21.1
My PGD Attack	47.0	30.3	16.0	8.6

Here, one sees that for every value of the perturbation strength the modified attack is stronger, i.e. decreases the model's accuracy more strongly. This is especially true for large perturbations (e.g. 21.1% vs. 8.6%).
Next, I played a little bit around with the training of the normal VGG network (w/o Bayesian reasoning). By tweaking the hyperparameters I was able to get these accuracies on a PGD attack with the same attack parameters as you used for the results in your paper:

Training method	0.015	0.035	0.055	0.07
Your hyperparameters	46.7	27.4	12.8	7.0
My hyperparameters	43.5	30.7	18.9	12.8

Now, one can compare this set of results with the results I got with my modified PGD attack on your Bayesian model. It looks like the Bayesian model/training is not really improving the robustness of the network.

I am really looking forward to your answers and explanations, hoping that all of this can easily be solved.

Thanks,
Roland

[xuanqing94]

@flashtek
To your first question, Line 33 adds all KL terms.

I will try your new PGD-attack later. I noticed the potential problem in the attack a month ago, but haven't fully investigated it.

Thanks for sharing your ideas!

[wielandbrendel]

@xuanqing94 In line 39 you only pass on kl and not kl_sum. I think @flashtek is right on this point.

[xuanqing94]

Oh sorry, that's a bug, I will fix it soon. It is supposed to be something in aaron_vi.py

Overall, there are two major problems in this repo. First I will add kl_sum as regularization, then I will test multiple forward-backward.

[zimmerrol]

@xuanqing94 Did you run any additional experiments or do you have any explanations for my observations yet? Right now, the results of my experiments look like they contradict the main claim of your paper, that an adversarially trained Bayesian network is way more robust than a deterministic network. Provided my evaluation method was not faulty, I feel that we should shed some light on the question of whether using a Bayesian network increases the robustness because currently, it does not look like it.

[xuanqing94]

@flashtek I have some initial experiments after fixing the bugs, the original hyperparameters are no longer suitable. So I am still finding better ones (specifically the original alpha value is too big).

It is still too early to draw a conclusion, the reason you can easily attack BayesianDefense is that, due to the bug you found, only the last layer has large noise, so you can imagine the final output is just Gaussian distribution, as O = W * x, and W is Gaussian. Multiple backpropagation can cancel the noise out. However, things become different when each layer has independent noise, you can refer to the twin paper "https://arxiv.org/abs/1712.00673".

I am currently having a internship this summer, so pls expect a slow experiment progress.

[zimmerrol]

@xuanqing94 Thanks for taking a look into it. I think your argument is only valid for the networks on CIFAR10/ImageNet-143 as for those networks there was this bug in the code for your initial experiments. But this does not hold true for the network on the STL10 dataset as you did not do the same typo again - this network is properly trained, just like you describe it in your paper.

I think that here we have two different problems:

1. the missing kl terms in the implementation of the network for CIFAR10/ImageNet-143
2. the general question whether this defense is really increasing the robustness if one evaluates it in a fair way.

I wrote done a more detailed description of the problems I faced testing the robustness of this defense and my conclusions about the defenses robustness in a comment here. Right now, I do not see a clear path of how to overcome this issue, even though the defense looked very promising at the beginning. If you'd like, we can further discuss this (personally) once you are done with your internship/have some free time again:)

[sungyoon-lee]

@xuanqing94 Any update on this topic?