optimize_weight_bits inflatting weight values?

[Michaeljurado42]

I initially reported this as a bug but am thinking that there is actually a reason behind this behavior.

Basically I have been observing some of the synapse weight values of an exported netx model. Surprisingly, I am seeing weight values scaled much higher than i was expecting. I traced the issue back to this optimize_weight_bits which produces results like this:

weights = np.array([-1, 1])
optimized_weights, num_weight_bits, weight_exponent, sign_mode = optimize_weight_bits(weights)
print(f"  Original Weights:  {weights}")
print(f"  Optimized Weights: {optimized_weights}")
print(f"  Number of Weight Bits: {num_weight_bits}")
print(f"  Weight Exponent: {weight_exponent}")
print(f"  Sign Mode: {sign_mode}\n")
print("--------------------------------------------")

Original Weights: [-1 1]
Optimized Weights: [-128 128]
Number of Weight Bits: 2
Weight Exponent: -7
Sign Mode: 1

It seems like a better way to encode this would be leave the optimized weight bits alone and set weight_exponent = 0. So my question is, what is the reason behind this behavior?

I'll update the discussion if I figure out what the answer is.

[bamsumit]

Hi @Michaeljurado42 I think this is a bug. From a cursory look, it should have been optimized_weights = [-1, 1], num_wgt_bits = 2 and wgt_exp = -7. The weight exponent is to compensate for MSB alignment in Loihi. Weight optimization is a lava issue though and not related to lava-dl.
I suspect that this bug has to do with sparse weights. Can you try a simple network Sparse -> LIF and Dense -> LIF with the same weights you are using in this particular case and see if the same behavior shows up in both the cases. In any case, please raise an issue in Lava repo.

[Michaeljurado42]

Hey @bamsumit, thanks for the response.

I've made a small test confirming that the error is independent of Sparse vs Dense.

I was working on a fix for it, which I can resuscitate now that I know its a bug.

While I was digging through the code I actually noticed another potential error which could affect DelaySynapses
https://github.com/lava-nc/lava-dl/blob/main/src/lava/lib/dl/netx/blocks/process.py#L177

The weight exponent is not being passed into the DelayDense and DelaySparseSynapses. See this test.

I opened up an issue and PR

[Michaeljurado42]

Okay, I made an issue. Looks like a call to lava's optimize_weight_bits was missing here.

Although, we could potentially solve the problem with a change in lava-dl where we pre-optimize the weights properly.