Distributed deployment orchestrator before config changes #1886

nshoes · 2025-02-05T19:59:10Z

No description provided.

Device connections might be flakey, but the update might still be happening in the background.

If the inflight update firmware uuid matches the device metadata then the update was likely a success.

Otherwise the delay in starting can cause it to miss some broadcasts, and make testing a bit harder.

This adds a new event that is sent to the orchestrator when the device is fully 'online' and has gone through the `after_boot` and device registration steps.

If the devices firmware matches the deployments, there is no need to trigger an orchestrator run

Since we now tell the deployment that a device assigned to it is online, we can place the "device finished updating" broadcast in the right place

And allow it to shutdown after its run

This pattern was recommended by `Horde` but isn't needed with `ProcessHub`

…application.ex`

…device to update

…n, he likes to be inclusive. Co-authored-by: Nate Shoemaker <[email protected]>

The update has failed, so clear the inflight update for the device and update the registry.

…ve or inactive

nshoes · 2025-02-05T19:59:44Z

Creating a PR for this branch just so CI can build an image.

joshk added 30 commits February 4, 2025 08:11

Run one deployment orchestrator per deployment per cluster

c4ea461

Support devices telling the orchestrator that they might of updated

d626d08

Minor cleanup

4949b88

Don't forcefully clear inflight updates

dff7764

Device connections might be flakey, but the update might still be happening in the background.

Remove inflight updates where the firmware matches the device

0f44a8e

If the inflight update firmware uuid matches the device metadata then the update was likely a success.

Only message an orchestrator if the device has a deployment

975105f

Some cleanup, some docs, some encapsulation

19953da

Use singular where the event is for one deployment

104fd3b

Prefer Phoenix.Socket.Broadcast

1047693

Use init over handle_continue(:boot, _)

795675e

Otherwise the delay in starting can cause it to miss some broadcasts, and make testing a bit harder.

Tests and fixes from the tests

b5476c5

Test scheduling updating as devices come online

f0a5441

This adds a new event that is sent to the orchestrator when the device is fully 'online' and has gone through the `after_boot` and device registration steps.

Remove a commented out test which has been covered

5936c1f

Bump the Orchestrator timer interval to 90 secs

466a127

Fix the need to have a second start_link function on the orchestrator

c38c188

Remove an unneeded subscribe in a test

37a36ef

Allow the orchestrator to ignore the device-online

07b69e8

If the devices firmware matches the deployments, there is no need to trigger an orchestrator run

Only send the "device finished updating" broadcast if it did update

fa3edd0

Since we now tell the deployment that a device assigned to it is online, we can place the "device finished updating" broadcast in the right place

The original Monitor shouldn't have been changed

10579fd

Make it easier to start a cluster in dev

82da381

Only start the clustered orchestrator on the web nodes

cce4ce8

Switch from Horde to ProcessHub

ee871d2

Tweak the log message when a device updates.

938f213

If the payload is nil, use an empty map

b260089

Simplify the Monitor to a one off OrchestratorRegistration

6f0bb27

Fix the new orchestrator tests

b2c9ac8

Add a child_spec to the OrchestratorRegistration module

70b6cf3

And allow it to shutdown after its run

Remove the Distributed.Supervisor, the extra layer isn't needed

d142aa7

Clean up Distributed.Orchestrator.start_link/1

a07baa2

This pattern was recommended by `Horde` but isn't needed with `ProcessHub`

Add some doc comments around the deployments orchestrator config in `…

850d4e3

…application.ex`

joshk and others added 10 commits February 4, 2025 08:11

Have the Orchestrator undertake threshold checks before scheduling a …

b7baf43

…device to update

Nate doesn't like counting IDs, he feels it excludes other informatio…

cd0b2e4

…n, he likes to be inclusive. Co-authored-by: Nate Shoemaker <[email protected]>

Fix a missing alias after merging in main

795d734

Fix a warning from dialyzer that this code can never be reached

672b178

Remove a superfluous broadcast, and move an audit to a better place

e43298e

Handle fwup error messages

4d63d1f

The update has failed, so clear the inflight update for the device and update the registry.

Clear a devices inflight update during connection if it isn't updating

cac0f87

Fix skip the queue (use phx-click not phx-submit)

ee80a66

Test deployments starting/stopping orchestrator when updating to acti…

b40062d

…ve or inactive

[publish]

8cd974f

nshoes closed this Feb 5, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Distributed deployment orchestrator before config changes #1886

Distributed deployment orchestrator before config changes #1886

nshoes commented Feb 5, 2025

nshoes commented Feb 5, 2025 •

edited

Loading

Distributed deployment orchestrator before config changes #1886

Distributed deployment orchestrator before config changes #1886

Conversation

nshoes commented Feb 5, 2025

nshoes commented Feb 5, 2025 • edited Loading

nshoes commented Feb 5, 2025 •

edited

Loading