Component upgrade state information

[ejbrever]

I'd like to propose adding upgrade state information for components. This will allow details regarding upgrades to be exposed to users as an upgrade process can take many minutes to hours. This would be read-only information.

This could support components such as transceivers (i.e. ZR transceivers) or the device operating system.

These could be placed under a new grouping such as:
openconfig/platform/component/state/upgrade/

Proposed leaves would be:
new-firmware-version: string, the new firmware package version.
new-firmware-version-service-impacting: bool, true if the firmware would require a service impacting upgrade to take effect.
status: identityref (sw loading, in-progress, fail, complete)
step: string, describing the step the device is attempting to do.
step-percent-complete: oc-types:percentage, the percent of the step that is completed.
total-percent-complete: oc-types:percentage, the percent of the total upgrade that is completed.
start-time: timeticks64, the time when the upgrade was initiated.
duration: yang:counter64, the time elapsed since the upgrade was initiated.
stop-time: timeticks64, the time when the upgrade was initiated.
last-known-failure: string, describing the last known failure encountered.

@robshakir @ahsaanyousaf

[proberts2022]

My current focus is around the use of this for transceivers...I may come back with more comments on the generality of this to other component types.
A few comments and questions for better understanding:

1. The new-version description indicates its available on the component. To me this means the firmware image has been already successfully transferred into the component but is not yet the active image. Is that the intent?
2. The duration field proposal above is how long the upgrade has been in progress but in PR#1242 the leaf description is "the duration of the upgrade" which I think can be interpreted in different ways. Also is it your intention that once an upgrade operation is complete, that this leaf would report the total time the last upgrade required or it would not be reported since there is no upgrade in progress. Perhaps the description of the leaf could be "The time elapsed since the upgrade was initiated or the duration of the last upgrade"
3. Why is the duration leaf tracking microseconds of elapsed time. Seems overkill for an operation that might take seconds. Do you see value on this granularity for debugging purposes?
4. Do you want a separate OPENCONFIG_UPGRADE_STATUS of UPGRADE_ABORT or is UPGRADE_FAIL sufficient?
5. For my education, does the openconfig model include 'action' definitions? Basically, how does one of these transceiver upgrades get triggered via openconfig?

[ejbrever]

1. Correct, that could only be specified once an image was transferred.
2. Thanks for the suggestion on the description, updated to that.
3. Agree. I had seen ms used elsewhere in OC, but there really is no value I can think of for that here. I changed it to specify seconds.
4. Seems reasonable to specify abort if you think platforms would support this. I added this.
5. That definitely needs to be tackled for transceivers and something I want to do :) gNOI is the mechanism for all actions like this. For operating systems, gNOI.OS or gNOI.System.SetPackage have been used for upgrade type needs. For transceivers, these don't have all the details we need (i.e. to be able to specify an optic for upgrade), but I think there could be several approaches to consider.

[proberts2022]

Thanks for accommodating my suggestions. Is there a reason you didn't add a
units "seconds";
statement for the duration and instead added this info into the description?
I am just trying to learn best practices...

[dplore]

It would be good to understand what the expected API is for initiating an image install. (In OC, we model config and state together. Agreed that gnoi would be the RPC path to invoke the install action)

The gnoi.OS service https://github.com/openconfig/gnoi/blob/main/os/os.proto#L40-L45 currently requires that any hardware component firmware installs are embedded within a device OS image. If we intend to use gnoi to invoke a component image install, we should propose an update this text and a suitable RPC.

I'd also recommend this be renamed to "install" instead of "upgrade". Upgrade implies an incremental version, but it's possible that the version could be decremented. Changing the meaning from upgrade to "install" makes this state data generic to whether it's an increment or decrement of firmware version.

[ejbrever]

@proberts2022 I think adding the units should be the best practice. I'm not sure it's done consistently, but I think we should. I've added that now.

[ejbrever]

@dplore I agree, 'install' is better and the PR is now updated with that change.

Updating that gNOI text makes sense and I can do that. I was thinking I'd tackle the gNOI work in an upcoming PR as we might want more discussion on that and we can then decouple some PRs, but please let me know.

[ggrammel]

A few questions for better understanding some corner cases:
@proberts2022 wrote in #1241 (comment)

1. The new-version description indicates its available on the component. To me this means the firmware image has been already successfully transferred into the component but is not yet the active image. Is that the intent?
   and @ejbrever responded
   Correct, that could only be specified once an image was transferred.

1. how to deal with components that do not support pre-loading. IOW, the new firmware may be downloaded to the host already, but firmware install is performed by downloading and re-writing the code into component. It probably doesn't make a difference in the model, but in the description we should not assume the pre-loading is in the component. Alternatively, a component-pre-loaded flag could indicate where the image is located.
2. in general, a service affecting installation process is triggered by a component reset. However, an unintended reset caused by external events after pre-loading can also trigger a pre-mature install operation. This can screw up a lot of things, in particular if a sequence of component installs must be kept to be successful. A ready-to-install flag would be useful but needs to be rw.
3. Assuming an external reset had triggered a successful install process which now needs to be reversed. Then it would be great to know what happened with the old image after installation. e.g.
   1. Is it safe to assume that in case of a component pre-load, the old image is still available in the component itself or does the old-image need to be again pre-loaded?
   2. In case the new image is held by the host, is it safe to assume that the host still holds a copy of the old image or does it need to be downloaded again to the host?

Thoughts?

[ejbrever]

1. I think if the image is on the host, but not yet on the optic, then it would not report anything here yet. An action to initiate the transfer would first need to occur. My understanding with ZR optics is this would be its own lengthy transfer in the tens of minutes. If the optic doesn't support that in two or more distinct steps (i.e. load, activate), then nothing would be reported until the action to initiate the install begins. There is another parallel discussion also occurring in OIF around the host managing a range of images that could be applied to optics when desired. The concept there is that the host has a default mapping of part-numbers to what their default firmware version should be. That would enable the host to automatically install the desired firmware as soon as an optic in inserted (allowing a similar plug-and-play approach that people are used to with grey optics). I think this default firmware approach in the future would tackle the component-pre-load flag in a different way. Essentially, if you want to know details of what is in on the optic itself, this proposed modeling would satisfy that. And if you wanted to know what images are pre-loaded on the host, that would be found within this other future modeling.
2. This is a good point and something I think we should make sure we discuss for how to do the 'action' of install. I am planning on pushing on that area next. In general, my initial thought is we should push to avoid what to me sound like indeterminate states. Basically, if the optic supports pre-loading in the sense of transferring an image to an ignored image slot, then that seems helpful. However, if the pre-loading puts the optic in a state where a random reseat or reboot at some point in the future would perform an install that doesn't seem like a good thing. In that case, pre-loading seems more trouble then its worth. In my opinion, if the optic only supports the pre-loading into an indeterminate state, then my suggestion would be to not support that and that optic should just support load+activate in a single action. Luckily, I am not sure this is much of a concern, because the real focus is on the newer ZR optics where firmware is expected to need upgrading more regularly (and are supposed to support two firmware image slots), unlike grey optics where my experience is that is a very rare event.
3. This is probably a hardware/implementation question. My understanding in some cases though is that the optics hold A/B firmware image slots and do a swap on activate. So the reversion could be quicker. However, if the hardware only has a single slot or the part doesn't implement reversion, then I suppose a fresh install would be required.

[ggrammel]

I was under the impression that the proposed YANG extensions would apply during a SW upgrade to provide better visibility into the installation progress across the system. Perhaps I have misinterpreted the intention.

The OIF proposal describes such case where a specific Host-SW-release comes with a bouquet of FW-images. A specific FW-image out of the bouquet is installed into the component once a matching HW is found. This would be a host pre-load of FW images and leaves it up to the implementation whether dual memory banks are used or not. Such procedure appears quite pragmatic to me and additional visibility into where the FW upgrade currently stands can be quite useful in case things 'hang'.

In contrast, it would be a little touchy to modify a FW package underneath a SW release without properly updating the SW release information. In case of surprises, debugging and RCA can become quite challenging. Also operationally it is hard to manage not to use a specific FW image if a combination of other factors (Host HW, SW, specific RI,...) applies - in particular if this FW image is already used in other combinations. Is it really worth going down this primrose path?

[ejbrever]

Yep, the intention for this specific PR is to provide visibility during the installation process.

I agree there is complexity in decoupling these FW images with host SW images :)

However, I know Google (and from my understanding also some other hyperscalers) feels pretty strongly that this is needed...and is already starting to happen. There are at least a couple reasons for this.

1. We have always operated under the model of router SW and optical transponder SW are entirely decoupled which was true because these functions lived in two distinct chassis'. That has been extremely valuable for technology evolution to occur quickly without dependencies on each other. With ZR optics we have blurred the lines and do not want to completely rely on router SW to enable innovation in optics. This won't always work, but in many cases (i.e. introducing new longer reach capabilities) it would.

2. In some cases, we are already receiving custom FW images from transceiver vendors that are specific for us and not shared with other companies. For that reason, there won't really be an option to bake these into host SW. You can imagine Google, Meta, MSFT etc all having custom optic FW and having to then get router vendors (Cisco, Juniper, etc.) to test/integrate every one into their own host SW.

[ggrammel]

It seems there is a need to be addressed since a while. Is there something to learn something from the present mode of operation? How is the FW update performed today while the transponder function is still in another chassis? IOW, how does the FW installation work when you are receiving custom FW images from transceiver vendors that are specific for Google, Meta, MSFT?
Does it work (give an take) like to OIF way described in #1241 (comment) or completely different?

... figured out that oif2024.339.06 provides some insight into the state of the art and is a good starting point going forward.

[LimeHat]

That definitely needs to be tackled for transceivers and something I want to do :) gNOI is the mechanism for all actions like this. For operating systems, gNOI.OS or gNOI.System.SetPackage have been used for upgrade type needs. For transceivers, these don't have all the details we need (i.e. to be able to specify an optic for upgrade), but I think there could be several approaches to consider.

If you plan to introduce a new RPC/API interface for this type of upgrades, do you still need all of the proposed yang exposed via gnmi?

For example, if you are going to create a gRPC service, the below leafs seem to belong to the new service (explicitly or implicitly):

step: string, describing the step the device is attempting to do.
step-percent-complete: oc-types:percentage, the percent of the step that is completed.
total-percent-complete: oc-types:percentage, the percent of the total upgrade that is completed.
start-time: timeticks64, the time when the upgrade was initiated.
duration: yang:counter64, the time elapsed since the upgrade was initiated.
stop-time: timeticks64, the time when the upgrade was initiated.

where the upgrade status can be streamed to the client which initiated the action.

[ejbrever]

It seems there is a need to be addressed since a while. Is there something to learn something from the present mode of operation? How is the FW update performed today while the transponder function is still in another chassis? IOW, how does the FW installation work when you are receiving custom FW images from transceiver vendors that are specific for Google, Meta, MSFT? Does it work (give an take) like to OIF way described in #1241 (comment) or completely different?

... figured out that oif2024.339.06 provides some insight into the state of the art and is a good starting point going forward.

In the case of the transponder function being in a separate chassis, when it comes to transceivers, we haven't actually faced this problem. This is because in that scenario we are only using gray optics in which optic firmware upgrades aren't much of an issue. This is because the piece that is iterating more quickly (and requiring upgrades) is the DWDM technology. When the transponder function is in a separate chassis, the DWDM technology lies within the cards of that chassis. This allows for a simpler scenario because the vendor of the OS and the firmware of the card is guaranteed to be same and everything is nicely tested/bundled together.

[ejbrever]

That definitely needs to be tackled for transceivers and something I want to do :) gNOI is the mechanism for all actions like this. For operating systems, gNOI.OS or gNOI.System.SetPackage have been used for upgrade type needs. For transceivers, these don't have all the details we need (i.e. to be able to specify an optic for upgrade), but I think there could be several approaches to consider.

If you plan to introduce a new RPC/API interface for this type of upgrades, do you still need all of the proposed yang exposed via gnmi?

For example, if you are going to create a gRPC service, the below leafs seem to belong to the new service (explicitly or implicitly):

step: string, describing the step the device is attempting to do.
step-percent-complete: oc-types:percentage, the percent of the step that is completed.
total-percent-complete: oc-types:percentage, the percent of the total upgrade that is completed.
start-time: timeticks64, the time when the upgrade was initiated.
duration: yang:counter64, the time elapsed since the upgrade was initiated.
stop-time: timeticks64, the time when the upgrade was initiated.

where the upgrade status can be streamed to the client which initiated the action.

I view these at two separate things because one is an 'action' to initiate something on the device (i.e. start the install). Where the other is a 'read' operation. Installs can take a really long time and involve restarts. Because of that it isn't practical to maintain a long running RPC that may be many 10s of minutes (maybe multiple hours for an OS install), especially with restarts where the RPC will have to terminate. We could have a gNOI that returns these details and could be called ad-hoc, but in that case, it feels like a read-only RPC which is replicating gNMI read-only functionality.

[LimeHat]

Does a transceiver FW upgrade take hours?

I think it is somewhat idiomatic to couple actions and related state in the same service, esp. when state is directly associated with the ongoing action (such as progress reporting). Doesn't have to be a single RPC.
We already have examples of using this approach; for instance, gnoi os service (since you mentioned an OS install..)

[ejbrever]

Hours is probably unlikely, but I've been told that due to the sometimes very limited bus speed on some transceivers the transfer can take up to an hour in some cases. But sounds like many are in ~10 min range.

[ejbrever]

I agree there are some slightly blurry lines here regarding what belongs in a gNOI RPC vs a gNMI read (via modeled data). Generally (may not be perfect) I think we should aspire towards keeping gNOI as temporal actions that are performed on a device and gNMI as the place to read state data.

@robshakir @dplore would be curious on your thoughts as well.

[LimeHat]

I think we should aspire towards keeping gNOI as temporal actions that are performed on a device and gNMI as the place to read state data.

I'd argue that OC yang/gNMI is a place for globally significant state data (unless you are using yang actions instead of gNOI; but that's a yang1.1 feature), and perhaps not all data in this scenario has this significance.
And, at the same time, I don't agree with the idea that gNOI should be strictly limited to initiating the actions (and not receiving status updates about those actions). IMO it is fine to design self-contained services in such cases.

If there's a system that is responsible for performing FW upgrades on transceivers, then it is probably interested in monitoring the progress and status of this operation. Do other systems care about the % of completeness of the upgrade?

I also don't think that this model extends well to other component types (hence I left a suggestion to limit this to transceivers in the PR), which is also a contributing factor.