Document (or revisit) the reason that site is a mutable field

[henrybear327]

What would you like to be added?

Discuss and document why we make site a mutable field but not tenant.

Why is this needed?

During the implementation of #90, we discovered that site being a mutable field will have an impact on allocation and restoration flow.

For example, if we have the following prefixes that will match the ParentPrefixSelector conditions:

10.0.1.0/24, tenant1, site1
10.0.2.0/24, tenant1, site2
10.0.3.0/24, tenant2, site1

Consider the following operation sequence (allocation):

• we apply a CR with tenant value tenant1, site value site1, and the ParentPrefixSelector that will return the aforementioned set of prefixes -> 10.0.1.0/24 is picked as the parent prefix after the reconcile loop completes
• we modify the site value to site2 -> what should we do?

Consider the following operation sequence (restoration):

• we apply a CR with tenant value tenant1, site value site1, enable preserveInNetBox, and the ParentPrefixSelector that will return the aforementioned set of prefixes -> 10.0.1.0/24 is picked as the parent prefix after the reconcile loop completes, and the restoration hash is computed against the immutable fields only
• we delete the CR
• we modify the site value to site2
• we apply the CR -> since site isn't used to calculate the hash, we can recover a prefix that doesn't satisfy the current CR requirement. Should we just abandon the restoration flow and allocate a new prefix from scratch? Or do we need to do something about the existing prefix first before we move on with further processing flow

[henrybear327]

@jstudler what's your recollection of the decision to make site mutable :)