[FC] process FC after apply view #3326
Conversation
|
/azpw run |
|
/AzurePipelines run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
stepanblyschak
force-pushed
the
fc-after-apply-view
branch
from
da85acb to
ffbd073
Compare
stepanblyschak
force-pushed
the
fc-after-apply-view
branch
from
ffbd073 to
1536dfb
Compare
stepanblyschak
force-pushed
the
fc-after-apply-view
branch
3 times, most recently
from
97e081f to
95b1e63
Compare
stepanblyschak
force-pushed
the
fc-after-apply-view
branch
from
6e1f7d5 to
8dca8ef
Compare
stepanblyschak
force-pushed
the
fc-after-apply-view
branch
2 times, most recently
from
7f0a73e to
e580645
Compare
|
Putting back delay for 60 sec as we found some cases where oper state update handling is delayed due to FC configuration after APPLY_VIEW |
| m_bufferQueueConfigTable(db, CFG_BUFFER_QUEUE_TABLE_NAME), | ||
| m_bufferPgConfigTable(db, CFG_BUFFER_PG_TABLE_NAME), | ||
| m_deviceMetadataConfigTable(db, CFG_DEVICE_METADATA_TABLE_NAME) | ||
| { | ||
| SWSS_LOG_ENTER(); | ||
| m_delayTimer = new SelectableTimer(timespec{.tv_sec = FLEX_COUNTER_DELAY_SEC, .tv_nsec = 0}); | ||
| if (WarmStart::isWarmStart()) |
There was a problem hiding this comment.
Just to confirm this will also handle fast-reboot.
| SWSS_LOG_ENTER(); | ||
|
|
||
| SWSS_LOG_NOTICE("Processing counters"); | ||
| m_delayTimer->stop(); |
There was a problem hiding this comment.
How about following?
if (!m_delayTimerExpired)
{
m_delayTimer->stop();
m_delayTimerExpired = true;
}
orchagent/flexcounterorch.cpp
Outdated
| @@ -254,6 +258,15 @@ void FlexCounterOrch::doTask(Consumer &consumer) | |||
| } | |||
| } | |||
|
|
|||
| void FlexCounterOrch::doTask(SelectableTimer &timer) | |||
There was a problem hiding this comment.
should we delete the timer here?
Signed-off-by: Stepan Blyschak <[email protected]>
Signed-off-by: Stepan Blyschak <[email protected]>
Signed-off-by: Stepan Blyschak <[email protected]>
Signed-off-by: Stepan Blyschak <[email protected]>
Signed-off-by: Stepan Blyschak <[email protected]>
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
orchagent/flexcounterorch.cpp
Outdated
| m_delayTimer = new SelectableTimer(timespec{.tv_sec = FLEX_COUNTER_DELAY_SEC, .tv_nsec = 0}); | ||
| if (WarmStart::isWarmStart()) | ||
| { | ||
| auto executor = new ExecutableTimer(m_delayTimer, this, "FLEX_COUNTER_DELAY"); |
There was a problem hiding this comment.
I am not convinced with this approach. I can use some to understand the approach.
Previously we had hardcoded delay in FC.
The hardcoded delay became a problem and we needed more deterministic delayed approach. So we added fast-reboot new infra which used warm-reboot kind of reconciliation and used fast-reboot done status to enable FC.
But now we again go back to hardcoded delay. I am afraid that to solve the new problem we (in a way) drift towards the day1 problem. Some scenarios that I can image that will now become problematic can be when 60s not being enough and overlaps w/ apply_view. Or similar race conditions that led to more deterministic delay.
Also, I think the concerns that Jingwen raised about new entries showing up in the table will still be present, right?
The config will still look different than what init config would want.
@wen587 , what do you think about this? Does this change really solve the concern that you have raised?
There was a problem hiding this comment.
@vaibhavhd This is the problem that is being solved sonic-net/sonic-buildimage#20302
The hardcoded delay is not a reported problem and this PR does not attempt to address that.
Not sure about race conditions, could you provide an example? The timer can't overlap with apply view as it is triggered in a different flow.
There was a problem hiding this comment.
Hi @vaibhavhd , i think so. We want the config just stay as it is in init_cfg.json. We do not want any derived changes after service restart. Because all config db are static in Golden Config. We need Golden Config to be ground of truth.
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Signed-off-by: Stepan Blyschak <[email protected]>
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Related to sonic-net/sonic-swss#3326. Why I did it Simplify approach to delaying counters on warm boot and fast boot. Removed FLEX_COUNTER_DELAY_STATUS_FIELD and instead postpone all FC processing to happen after apply view to not delay data plane configuration. The CONFIG_DB should not be updated in runtime anymore for counters to be delayed. To address sonic-net#20302. Work item tracking Microsoft ADO (number only): How I did it Removed FLEX_COUNTER_DELAY_STATUS_FIELD set in enable_counters.py. How to verify it Run warm-boot - make sure FC orch runs only after APPLY_VIEW.
|
Infra issue with tests: failure to install dotnet package: Restarting |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@qiluo-msft @bingwang-ms @vaibhavhd Can be merged? |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
What I did
Simplify approach to delaying counters on warm boot and fast boot. Removed FLEX_COUNTER_DELAY_STATUS_FIELD and instead postpone all FC processing to happen after apply view to not delay data plane configuration.
The CONFIG_DB should not be updated in runtime anymore for counters to be delayed.
Why I did it
To address sonic-net/sonic-buildimage#20302.
How I verified it
Run warm-boot - make sure FC orch runs only after APPLY_VIEW.
Details if related