LIG on contextual BERT embedding may make more sense

[neoyipeng2018]

Digging into the code, noticed that the LIG was on the first embedding input into BERT - this may not adequately capture the subsequent attention and contextual representation of the input text.

I did a quick switch of the code to take in the last layer output and the results make more sense to me, e.g. the model doesn't focus on [SEP] token in the later layers (as discussed in "What Does BERT Look At? An Analysis of BERT's Attention, Clark 2019")

Wondering if this makes sense, or if my understanding is wrong in any way?

[cdpierse]

Hi @neoyipeng2018,

This is a really interesting question and it's something I thought about too when I started working with Captum for explanations. From my understanding the main reason why the embedding layer is chosen is because each embedding in this layer can be directly related back to a specific token and thus a direct relationship (positive or negative) can be inferred as we integrate the baseline embedding values into the actual values. Once you go past this initial embedding layer it becomes more difficult to say that a particular embedding is related to a particular token's influence.

That being said this was just my interpretation of how it was working from what I read but there might well be a lot of merit in looking at the lower layers, how did you go about swapping out the input embeddings for the output one? I'd love to see a comparison of results :-)

Thanks for raising such an interesting question btw !