From 9aafccbb3071ec6f00e6ce157d4a560880bf59f8 Mon Sep 17 00:00:00 2001 From: Kristian Amlie Date: Tue, 22 Oct 2024 16:06:24 +0200 Subject: [PATCH 1/8] Import READMEs verbatim from the orchestrator repo. This commit only exists to make the review of the next commit easier, so one can easily see what has been changed from the original when basing the docs on these READMEs. A few portions that dealt with compiling the source code has been taken out, since the repository is closed source. Signed-off-by: Kristian Amlie --- import/README-examples.md | 274 ++++++++++++++++++++++++++++++++++++++ import/README.md | 250 ++++++++++++++++++++++++++++++++++ 2 files changed, 524 insertions(+) create mode 100644 import/README-examples.md create mode 100644 import/README.md diff --git a/import/README-examples.md b/import/README-examples.md new file mode 100644 index 000000000..3b6d491ff --- /dev/null +++ b/import/README-examples.md @@ -0,0 +1,274 @@ +## Preparing the QEMU + +The starting point for successfully completing the examples in this README is a clean `orch-install` directory ready. + +REQUIREMENTS: +* you can run mender-artifact on your host +* you can run a Docker image on your host +* you have a Hosted Mender tenant and can get a token + +### Manual installation on QEMU + +The content of the orch-install directory is ready to be copied to the QEMU device. + +Run the command below in a separate terminal on the host. +The command will run a virtual device and consume the terminal. +You have to set the tenant token from your tenant. + +```bash +TENANT_TOKEN='' + +docker run -it -p 85:85 -e SERVER_URL='https://hosted.mender.io' -e TENANT_TOKEN=$TENANT_TOKEN --pull=always mendersoftware/mender-client-qemu:resized-for-demo +``` + +Once the logging prompt shows, you can log in. +The username is root. + + +Once the device boots it will be visible in Hosted Mender as pending. +Accept it. + +With the previous terminal consumed by the virtual device, execute these commands on a different terminal on the host: + +``` bash +CONTAINER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aqf "ancestor=mendersoftware/mender-client-qemu:resized-for-demo")) +# If you get multiple IPs use the alternative +# where you need to manually get the container id first +# docker ps +# CONTAINER_ID= +# CONTAINER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $CONTAINER_ID) + +scp -r -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 8822 orch-install root@$CONTAINER_IP: +``` + +On the device: + + +```bash +# Manually install the Orchestrator and the environment +orch-install/manual_install.sh +# Export the variable the readme suggests +``` + +``` bash +# Test if the cli is executing correctly +mender-update-orchestrator -h +``` + +``` bash +# Do a mock install +mender-update-orchestrator install $MOCK_DIR/system-core-v1/manifest.yaml + +# Check the state of the system after the install +mender-update-orchestrator show-provides +``` + + +## Examples with mock interfaces and devices + +These examples are all executed with mocked devices and versions. +The fact that they are running on a device with an installed Mender client is just a convenience to have a common environment. +They could have easily enough been natively run on an x86_64 machine. + +They are all executed on the terminal of the QEMU device. + + +### Example: Standard system update + + +Point the Orchestrator to the bundle manifest to update the system. + +```bash +mender-update-orchestrator install $MOCK_DIR/system-core-v1/manifest.yaml +``` + +Once that succeeds, you can see the system components versions with the command below: + +```bash +mender-update-orchestrator show-provides +``` + +The Orchestrator keeps track of the last successfully applied Manifest and prints that with the `show-provides` command. + + +#### Don't apply version if already present + + +Attempting to apply the v1 again skips the steps. + +```bash +mender-update-orchestrator install $MOCK_DIR/system-core-v1/manifest.yaml +mender-update-orchestrator show-provides +``` + +#### System rollback + +In this case, one of the devices in a system will fail. +As a result, the system rolls back as a whole. + +Set up a failure for one of the mock interfaces. + +```bash +touch $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL +``` + +Assuming the device is on v.1 commence an update to v.2: + +```bash +mender-update-orchestrator install $MOCK_DIR/system-core-v2/manifest.yaml +``` + +Because of the failure in one component the whole system stays on v1. + +``` bash +mender-update-orchestrator show-provides +``` + +Clean up the mocked failure and reapply the update: + +```bash +rm $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL +mender-update-orchestrator install $MOCK_DIR/system-core-v2/manifest.yaml +mender-update-orchestrator show-provides +``` + +As there was no failure this time, the system successfully updated to v2. + + +### Example: Multipart Manifest + +Sometimes, the full system topology isn't available at the time the bundle is created. +You know what components the system can, in theory, consist of, but the final combination is known only once it's installed in the field. +This is often the case with systems that get assembled on the customer site from modular components. + +To accommodate this use case, the bundle comes with a manifest that isn't fully defined and needs to get information available only on the end device. +This is where the multipart Manifest is used. + +The top part of the Manifest which comes with the bundle is already present in the environment. + +Create the bottom manifest that didn't come with the bundle. + +```bash +cat > $MOCK_DIR/system-core-v1.bottom.manifest.yaml << EOF +api_version: mender/v1 +kind: update_manifest +components: + - component_type: gateway + id: "G13" + args: + - arg1 + - component_type: rtos + id: "R456" + args: + - arg1 + - component_type: rtos + id: "R998" + args: + - arg1 +EOF +``` + +We're updating the device back to v1 again using the split manifest approach. +When invoking the mender comment, pass the bottom Manifest to the cli as well. + + +```bash +mender-update-orchestrator --manifest-components $MOCK_DIR/system-core-v1.bottom.manifest.yaml install $MOCK_DIR/system-core-v1-top-only/manifest.yaml +mender-update-orchestrator --manifest-components $MOCK_DIR/system-core-v1.bottom.manifest.yaml show-provides +``` + +The Orchestrator will merges the two manifests and updates the system as a whole. + + +## Examples with real gateway updates + + +The most complicated orchestrator case is the one where the device that runs the Orchestrator gets updated. +The cases below still use mock devices for the peripheral ROTS devices, but the gateway is going through a full rootfs update. + + +### Prepare the real rootfs artifacts + +The real rootfs update artifacts will be created from runtime using the mender-artifact tool. + +On the host. + +``` bash +USER="root" +DEVICE_TYPE="qemux86-64" +TYPE="rootfs-image" + +mender-artifact write rootfs-image -f ssh://${USER}@${CONTAINER_IP} \ + -n gateway-v3 \ + -o gateway-v3.mender \ + -t $DEVICE_TYPE \ + --ssh-args '-p 8822' \ + --ssh-args '-o UserKnownHostsFile=/dev/null' \ + --ssh-args '-o StrictHostKeyChecking=no' +``` + +``` bash +USER="root" +DEVICE_TYPE="qemux86-64" +TYPE="rootfs-image" +mender-artifact write rootfs-image -f ssh://${USER}@${CONTAINER_IP} \ + -n gateway-v4 \ + -o gateway-v4.mender \ + -t $DEVICE_TYPE \ + --ssh-args '-p 8822' \ + --ssh-args '-o UserKnownHostsFile=/dev/null' \ + --ssh-args '-o StrictHostKeyChecking=no' +``` + +```bash +# This is the path on the device where we deliver the created artifact +MOCK_DIR=/data/mender-update-orchestrator/mock_env +scp -P 8822 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no gateway-v3.mender ${USER}@${CONTAINER_IP}:${MOCK_DIR}/system-core-v3/gateway-v3.mender +``` + +Upload the `gateway-v4.mender` to Hosted Mender to the tenant the device is accepted on. + + +### Example: Standard system update - real gateway - offline + + +On the device. + +``` bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +mender-update-orchestrator --log-level debug install $MOCK_DIR/system-core-v3/manifest.yaml +``` + +The Orchestrator installs the rootfs for the gateway and reboots. + +Log in into the new system to resume the system update: + +``` bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +mender-update-orchestrator resume +mender-update-orchestrator show-provides +``` + +The orchestrator continues updating the rest of the system components and concludes the system update. + + +### Example: Standard system update - real gateway - Hosted Mender download + +In this case the artifact of the gateway is not available on the device but is present on Hosted Mender. +The RTOS artifacts are present on the device. + +The orchestrator will pull the gateway artifact form Hosted Mender using the authentication from the Mender client. + + +```bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +mender-update-orchestrator --log-level debug install $MOCK_DIR/system-core-v4/manifest.yaml +# Orchestrator installs the rootfs and reboots +``` +Log in into the new system + +``` bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +mender-update-orchestrator resume +mender-update-orchestrator show-provides +``` diff --git a/import/README.md b/import/README.md new file mode 100644 index 000000000..34d84f23e --- /dev/null +++ b/import/README.md @@ -0,0 +1,250 @@ +# Update Orchestrator + +The Update Orchestrator is the software responsible for updating the devices in a system. + +It is responsible for inspecting the system’s current state, such as the software versions running on the devices. It can also read the system's desired state from a configuration file, the Update Manifest, and apply the changes to the different Devices to reach the state described in the Manifest. If this is not possible, the Orchestrator may roll back to the previously known working state, in other words, the previous version of the Update Manifest. + + +Skip concepts, give me [working examples](#installing-from-source). + +## Concepts + +### A system + +A combination of coupled connected devices belonging to the same physical or logical product. + + +For the purpose of introducing the Orchestrator, the example system will consist of the following devices: + +``` ++--------+ +-------+ +-------+ +|Linux | | RTOS | |RTOS | +|Gateway | | | | | +|ID: G13 | |ID:R998| |ID:R456| ++--------+ +-------+ +-------+ +``` + +Linux Gateway G13: +* A device running Linux and having direct internet connectivity +* Can deliver updates to all other devices +* Has means to provide backward compatibility + * Communicate with different RTOS versions +* Can fix (update) the RTOS +* Can fetch the update content outside system boundaries + + +Two specialized Real-Time Operating System (RTOS) devices with no direct internet connectivity. +Besides running their specific function and having a rudimentary update system, there is not much they can do. + +The IDs are arbitrary unique device names used to distinguish one from the other. + +### Orchestrator runner + +The Orchestrator is software that needs to be executed on some device within the system. +In the example case, the Linux Gateway is the obvious choice as it can update other devices within the system. + + +#### Components vs devices + +A device can contain more than one software version or contain multiple logical components within a single hardware. +That is why, in the context of a system, we talk about components that have a software version regardless of how they map to devices. + +For simplicity, the example system has a simple 1-1 mapping of components to devices. + +i.e. +A Linux gateway that has OS and application versioned and updated separately would represent a device with two components. + +### Declarative approach to updates + + +Systems consist of multiple components, all of which need to be updated by respecting certain update constraints. +Two types of constraints exist: +* end state constraints - the acceptable combinations of versions the system components can be once the update completes +* transition state constraints - the acceptable combination of versions the system components can be in while transitioning from one end state to another + +In an imperative approach used by the Mender client, you would define a set of individual actions that need to take place for the system to update. +i.e. + +``` +Action 1 - Update the first system component from v1 to version v2 +Action 2 - Update the second system component from v1 to v2 +... +... +``` + +In the declarative approach, you specify the end state and the constraints, and the Orchestrator is in charge of defining and executing the steps. + + +### Constraints + +#### End state constraints + +An acceptable combination of versions representing a working version of a system. + + +| Gateway | Rtos | Acceptable | System version | +|--------------------|------------------|------------|----------------| +| gateway-v1.mender | rtos-v1.mender | YES | v.1 | +| gateway-v1.mender | rtos-v2.mender | NO | n/a | +| gateway-v2.mender | rtos-v1.mender | NO | n/a | +| gateway-v2.mender | rtos-v2.mender | YES | v.2 | + + +#### Transition state constraints + +Because the Gateway is the more capable of the devices with the capacity to rollback on its own, it is the component that needs to update first. + +An acceptable combination of versions to go through while transitioning to a valid end state of the system. + + +| Gateway | Rtos | Acceptable | +|-------------------|-----------------|------------| +| gateway-v1.mender | rtos-v1.mender | n/a | +| gateway-v1.mender | rtos-v2.mender | NO | +| gateway-v2.mender | rtos-v1.mender | YES | +| gateway-v2.mender | rtos-v2.mender | n/a | + + + +The Manifest is an instruction to the Orchestrator on how to update a system while respecting the update constraints. + +Below is an example Manifest for the given system. +The explanation of what field means is best explored through the examples below. + + +``` +api_version: mender/v1 +kind: update_manifest +version: "system-core-v1" +component_types: + gateway: + artifact_path: gateway-v1.mender + update_strategy: + order: 10 + rtos: + artifact_path: rtos-v1.mender + update_strategy: + order: 20 + +components: + - component_type: gateway + id: "G13" + args: + - arg1 + - component_type: rtos + id: "R456" + args: + - arg1 + - component_type: rtos + id: "R998" + args: + - arg1 +``` + + +### Update interface + +An update interface is a command line application that serves as a translator between the updatable component and the Orchestrator + +The update interface protocol finds inspiration in the update [modules protocol](https://github.com/mendersoftware/mender/blob/master/Documentation/update-modules-v3-file-api.md) in such that the Orchestrator acts as a state machine runner that invokes the update interface. + +The update interface has an expanded set of arguments compared to an update module. + +For all states, the arguments look like this: + + +``` +./update-interface [] +``` + +`State` - the current state in the state machine as defined by the Orchestrator + +`Work Dir` - the working directory for the interface. It follows the same structure and guarantees defined for the [Update modules protocol](https://github.com/mendersoftware/mender/blob/master/Documentation/update-modules-v3-file-api.md#file-api). +When using packed Artifacts and "files tree", the Artifact will be at the path `files/artifact.mender`. + +`Component Type` - the type of the component instance + +`Component Id` - a unique ID of the component instance + +``` +components: + - component_type: gateway # This is the component type + id: G13 # This is the component ID +``` + +`Custom arguments` - a list of customizable list of arguments as defined in the updatable component instance. + +``` +components: + - component_type: gateway + id: G13 + args: + - arg1 # These are the custom arguments + - arg2 + - arg3 +``` + +### The Orchestrator bundle + +For the Orchestrator to be able to update the system, the Manifest and the artifacts for the system components need to be accessible. +Those two things (Manifest + artifact) are called the Orchestrator bundle. + +The Orchestrator is agnostic about the way the bundle is delivered to it. +For the default case, the Mender client (in daemon mode, communicating with the Mender server) is used to deploy the bundle to the device so that the Orchestrator can act on it. + +It is also possible to just pass a Manifest file without the artifacts. +In that case, nothing is 'bundled', and the Manifest is the sole input. +The Orchestrator pulls the needed artifacts from the server on its own. + +## Known limitations + +When provided through a bundle, the artifacts need to be available to the Orchestrator in complete content as a .mender file. +In the worst-case scenario, this means the device needs to have enough storage to download its own compressed rootfs update first. +For comparison, in the Mender client implementation, the content is streamed into the inactive partition. + + +## Persistent data store + +The persistent data store is the location where the Orchestrator stores the current active Manifest and artifacts it might have additionally downloaded. + +The recommended setup is to have the data store of `mender-update-orchestrator` in a persistent separate partition, `/data` in the commands below, and use a symlink from `/var/lib`. + + +Create the directory and the symlink: + +``` +mkdir -p /data/mender-update-orchestrator +ln -s /data/mender-update-orchestrator /var/lib/mender-update-orchestrator +``` + + +# Examples + +All the commands expect you to be in the root directory of the Orchestrator folder (`mender-update-orchestrator*`). + +This section will explain how the Orchestrator works through copy-pastable working examples. +The code can be executed on a machine that can run Python. + + +## Prepare the environment + +Copy the pregenerated demo env: + +``` bash +cp -r demo/pregenerated_env/* orch-install +``` + +Copy the real interface + +```bash +git clone https://github.com/mendersoftware/mender-orchestrator-update-interfaces +cp -r mender-orchestrator-update-interfaces/interfaces/v1/rootfs-image \ + $ORCH_EVAL_DIR/share/mender-update-orchestrator/update-interfaces/v1 +``` + + +## Examples + +For examples to run please look the README-examples.md +That's the document that is also shared with externals during the beta stage. + From 846039256ec38e42e15400f70cf716782ca84400 Mon Sep 17 00:00:00 2001 From: Kristian Amlie Date: Mon, 21 Oct 2024 14:15:23 +0200 Subject: [PATCH 2/8] Rename and adapt orchestrator demo docs to new conditions. Ticket: MEN-7016 Signed-off-by: Kristian Amlie --- .../05.Orchestrator/01.Installation/docs.md | 10 ++ .../05.Orchestrator/02.Examples/docs.md | 96 ++++++++++++++----- .../05.Orchestrator/docs.md | 66 ++----------- 3 files changed, 92 insertions(+), 80 deletions(-) create mode 100644 09.Add-ons/05.Orchestrator/01.Installation/docs.md rename import/README-examples.md => 09.Add-ons/05.Orchestrator/02.Examples/docs.md (76%) rename import/README.md => 09.Add-ons/05.Orchestrator/docs.md (80%) diff --git a/09.Add-ons/05.Orchestrator/01.Installation/docs.md b/09.Add-ons/05.Orchestrator/01.Installation/docs.md new file mode 100644 index 000000000..e1410ab2e --- /dev/null +++ b/09.Add-ons/05.Orchestrator/01.Installation/docs.md @@ -0,0 +1,10 @@ +--- +title: Installation +taxonomy: + category: docs +--- + +TODO + + diff --git a/import/README-examples.md b/09.Add-ons/05.Orchestrator/02.Examples/docs.md similarity index 76% rename from import/README-examples.md rename to 09.Add-ons/05.Orchestrator/02.Examples/docs.md index 3b6d491ff..815732b7d 100644 --- a/import/README-examples.md +++ b/09.Add-ons/05.Orchestrator/02.Examples/docs.md @@ -1,12 +1,46 @@ -## Preparing the QEMU +--- +title: Examples +taxonomy: + category: docs +--- -The starting point for successfully completing the examples in this README is a clean `orch-install` directory ready. +# Examples + +This section will explain how the Orchestrator works through copy-pastable working examples. REQUIREMENTS: -* you can run mender-artifact on your host +* you can run Python on your host +* you can run [mender-artifact](../../../10.Downloads/docs.md#mender-artifact) on your host * you can run a Docker image on your host * you have a Hosted Mender tenant and can get a token + +## Prepare the environment + +```bash +mkdir orch-install +ORCH_EVAL_DIR=$(realpath orch-install) +``` + +Clone the repository which contains the support files and the demo: + +```bash +git clone https://github.com/mendersoftware/mender-orchestrator-update-interfaces +``` + +Copy the pregenerated demo env and the interface: + +``` bash +cp -r mender-orchestrator-update-interfaces/demo/pregenerated_env/* orch-install +cp -r mender-orchestrator-update-interfaces/interfaces/v1/rootfs-image \ + $ORCH_EVAL_DIR/share/mender-update-orchestrator/update-interfaces/v1 +``` + + +## Preparing the QEMU host + +The starting point for successfully completing the examples in this README is a clean `orch-install` directory ready. + ### Manual installation on QEMU The content of the orch-install directory is ready to be copied to the QEMU device. @@ -15,10 +49,14 @@ Run the command below in a separate terminal on the host. The command will run a virtual device and consume the terminal. You have to set the tenant token from your tenant. + ```bash TENANT_TOKEN='' -docker run -it -p 85:85 -e SERVER_URL='https://hosted.mender.io' -e TENANT_TOKEN=$TENANT_TOKEN --pull=always mendersoftware/mender-client-qemu:resized-for-demo +# Change this to eu.hosted.mender.io if you're in that zone. +SERVER_URL=https://hosted.mender.io + +docker run -it -p 85:85 -e SERVER_URL=$SERVER_URL -e TENANT_TOKEN=$TENANT_TOKEN mendersoftware/mender-client-qemu:mender-master ``` Once the logging prompt shows, you can log in. @@ -28,10 +66,11 @@ The username is root. Once the device boots it will be visible in Hosted Mender as pending. Accept it. -With the previous terminal consumed by the virtual device, execute these commands on a different terminal on the host: +With the previous terminal consumed by the virtual device, execute these commands on a different terminal on the **host**: + ``` bash -CONTAINER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aqf "ancestor=mendersoftware/mender-client-qemu:resized-for-demo")) +CONTAINER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aqf "ancestor=mendersoftware/mender-client-qemu:mender-master")) # If you get multiple IPs use the alternative # where you need to manually get the container id first # docker ps @@ -41,7 +80,7 @@ CONTAINER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddres scp -r -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 8822 orch-install root@$CONTAINER_IP: ``` -On the device: +On the **device**: ```bash @@ -55,15 +94,6 @@ orch-install/manual_install.sh mender-update-orchestrator -h ``` -``` bash -# Do a mock install -mender-update-orchestrator install $MOCK_DIR/system-core-v1/manifest.yaml - -# Check the state of the system after the install -mender-update-orchestrator show-provides -``` - - ## Examples with mock interfaces and devices These examples are all executed with mocked devices and versions. @@ -169,15 +199,33 @@ EOF ``` We're updating the device back to v1 again using the split manifest approach. -When invoking the mender comment, pass the bottom Manifest to the cli as well. +Configure the orchestrator to use the split manifest: + +```bash +mkdir -p /etc/mender-update-orchestrator +cat > /etc/mender-update-orchestrator/mender-update-orchestrator.conf < The update interface protocol finds inspiration in the update [modules protocol](https://github.com/mendersoftware/mender/blob/master/Documentation/update-modules-v3-file-api.md) in such that the Orchestrator acts as a state machine runner that invokes the update interface. The update interface has an expanded set of arguments compared to an update module. @@ -159,6 +165,7 @@ For all states, the arguments look like this: `State` - the current state in the state machine as defined by the Orchestrator + `Work Dir` - the working directory for the interface. It follows the same structure and guarantees defined for the [Update modules protocol](https://github.com/mendersoftware/mender/blob/master/Documentation/update-modules-v3-file-api.md#file-api). When using packed Artifacts and "files tree", the Artifact will be at the path `files/artifact.mender`. @@ -195,56 +202,3 @@ For the default case, the Mender client (in daemon mode, communicating with the It is also possible to just pass a Manifest file without the artifacts. In that case, nothing is 'bundled', and the Manifest is the sole input. The Orchestrator pulls the needed artifacts from the server on its own. - -## Known limitations - -When provided through a bundle, the artifacts need to be available to the Orchestrator in complete content as a .mender file. -In the worst-case scenario, this means the device needs to have enough storage to download its own compressed rootfs update first. -For comparison, in the Mender client implementation, the content is streamed into the inactive partition. - - -## Persistent data store - -The persistent data store is the location where the Orchestrator stores the current active Manifest and artifacts it might have additionally downloaded. - -The recommended setup is to have the data store of `mender-update-orchestrator` in a persistent separate partition, `/data` in the commands below, and use a symlink from `/var/lib`. - - -Create the directory and the symlink: - -``` -mkdir -p /data/mender-update-orchestrator -ln -s /data/mender-update-orchestrator /var/lib/mender-update-orchestrator -``` - - -# Examples - -All the commands expect you to be in the root directory of the Orchestrator folder (`mender-update-orchestrator*`). - -This section will explain how the Orchestrator works through copy-pastable working examples. -The code can be executed on a machine that can run Python. - - -## Prepare the environment - -Copy the pregenerated demo env: - -``` bash -cp -r demo/pregenerated_env/* orch-install -``` - -Copy the real interface - -```bash -git clone https://github.com/mendersoftware/mender-orchestrator-update-interfaces -cp -r mender-orchestrator-update-interfaces/interfaces/v1/rootfs-image \ - $ORCH_EVAL_DIR/share/mender-update-orchestrator/update-interfaces/v1 -``` - - -## Examples - -For examples to run please look the README-examples.md -That's the document that is also shared with externals during the beta stage. - From 4f8b5bb2e325104dcac560175cf4d2c1391390c1 Mon Sep 17 00:00:00 2001 From: Kristian Amlie Date: Wed, 23 Oct 2024 10:13:57 +0200 Subject: [PATCH 3/8] Add an orchestrator example using a deployment to distribute manifest. Ticket: MEN-7016 Signed-off-by: Kristian Amlie --- .../05.Orchestrator/02.Examples/docs.md | 104 +++++++++++++++++- 1 file changed, 103 insertions(+), 1 deletion(-) diff --git a/09.Add-ons/05.Orchestrator/02.Examples/docs.md b/09.Add-ons/05.Orchestrator/02.Examples/docs.md index 815732b7d..0935f1a54 100644 --- a/09.Add-ons/05.Orchestrator/02.Examples/docs.md +++ b/09.Add-ons/05.Orchestrator/02.Examples/docs.md @@ -20,6 +20,8 @@ REQUIREMENTS: ```bash mkdir orch-install ORCH_EVAL_DIR=$(realpath orch-install) +mkdir -p $ORCH_EVAL_DIR/bin +PATH=$PATH:$ORCH_EVAL_DIR/bin ``` Clone the repository which contains the support files and the demo: @@ -36,6 +38,24 @@ cp -r mender-orchestrator-update-interfaces/interfaces/v1/rootfs-image \ $ORCH_EVAL_DIR/share/mender-update-orchestrator/update-interfaces/v1 ``` +Copy the update module and the inventory script: + +```bash +mkdir -p $ORCH_EVAL_DIR/share/mender/modules/v3 +cp mender-orchestrator-update-interfaces/modules/mender-orchestrator-manifest/module/v3/mender-orchestrator-manifest \ + $ORCH_EVAL_DIR/share/mender/modules/v3 +mkdir -p $ORCH_EVAL_DIR/share/mender/inventory +cp mender-orchestrator-update-interfaces/inventory/mender-inventory-mender-orchestrator \ + $ORCH_EVAL_DIR/share/mender/inventory +``` + +Install the manifest artifact generator for the host: + +```bash +cp mender-orchestrator-update-interfaces/modules/mender-orchestrator-manifest/module-artifact-gen/mender-orchestrator-manifest-gen \ + $ORCH_EVAL_DIR/bin +``` + ## Preparing the QEMU host @@ -268,13 +288,27 @@ mender-artifact write rootfs-image -f ssh://${USER}@${CONTAINER_IP} \ --ssh-args '-o StrictHostKeyChecking=no' ``` +``` bash +USER="root" +DEVICE_TYPE="qemux86-64" +TYPE="rootfs-image" +mender-artifact write rootfs-image -f ssh://${USER}@${CONTAINER_IP} \ + -n gateway-v5 \ + -o gateway-v5.mender \ + -t $DEVICE_TYPE \ + --ssh-args '-p 8822' \ + --ssh-args '-o UserKnownHostsFile=/dev/null' \ + --ssh-args '-o StrictHostKeyChecking=no' +``` + ```bash # This is the path on the device where we deliver the created artifact MOCK_DIR=/data/mender-update-orchestrator/mock_env scp -P 8822 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no gateway-v3.mender ${USER}@${CONTAINER_IP}:${MOCK_DIR}/system-core-v3/gateway-v3.mender ``` -Upload the `gateway-v4.mender` to Hosted Mender to the tenant the device is accepted on. +Upload the `gateway-v4.mender` and `gateway-v5.mender` artifacts to Hosted Mender to the tenant the +device is accepted on. ### Example: Standard system update - real gateway - offline @@ -320,3 +354,71 @@ MOCK_DIR=/data/mender-update-orchestrator/mock_env mender-update-orchestrator resume mender-update-orchestrator show-provides ``` + + +### Example: Standard system update - real gateway - managed by Hosted Mender + +In this example we will perform the same update as in the previous example, but we will do so using +a Hosted Mender deployment instead of installing the manifest using the command line. This makes use +of the `mender-orchestrator-manifest` update module that we installed earlier. + +This works just like a regular Mender deployment, including rolling back automatically if a failure +is detected. + +#### Prepare the deployment + +Let's introduce a failure to see this in action. On the **device**, execute this: + +```bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +touch $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL +``` + +We need to create a special artifact containing the new manifest we want to install. For this we use +the `mender-orchestrator-manifest-gen` tool. On the **host**, execute this: + +```bash +DEVICE_TYPE="qemux86-64" +mender-orchestrator-manifest-gen \ + -n manifest-v5 \ + -o manifest-v5.mender \ + -t $DEVICE_TYPE \ + $ORCH_EVAL_DIR/mock_env/system-core-v5/manifest.yaml +``` + +Upload the resulting `manifest-v5.mender` artifact to Hosted Mender. + +We also need to upload the `rtos-v5.mender` artifact for the mock device, in addition to the +`gateway-v5.mender` artifact which we already uploaded above when creating the rootfs artifacts. Go +ahead and upload the artifact which you can find at this path: +`orch-install/mock_env/system-core-v5/rtos-v5.mender`. + +#### Launch the deployment + +While logged into Hosted Mender in your browser, click on the "Releases" tab and select the +`manifest-v5` release from the list. Then select "Create a deployment from this release" from the +"Release actions" menu, select "All devices" from the device group list, and click "Create +deployment" to start the deployment. + +!!! Make sure the `gateway-v5` release is also present in the list, which we uploaded when preparing +!!! the rootfs artifacts above. + +After a little while, the deployment should finish with the failed status, because of the failure we +introduced above. View it under the "Finished" tab. Go ahead and take a look at the failure log by +clicking "View details", and then "View log" while inside the failed deployment popout. Note in +particular that it was not the rootfs update of the gateway that failed, but the rtos component +which was updated after it, yet it caused both to be rolled back. + +Let's remove the failure and retry the deployment. On the **device**, execute this: + +```bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +rm -f $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL +``` + +Then repeat the steps to start the deployment, and wait for it to finish. It should finish with +success this time. + +Take a look at the updated inventory for the device by clicking the "Devices" tab in the UI, then +click on the device. Then click on "Software" and "Inventory" tabs to view the installed software +and the rest of the inventory, respectively. From e1b98d2ec297b24fd82e80685303524ad1bf3f7a Mon Sep 17 00:00:00 2001 From: Kristian Amlie Date: Wed, 23 Oct 2024 12:11:03 +0200 Subject: [PATCH 4/8] Add stop-gap installation instructions before binary distribution is ready. Signed-off-by: Kristian Amlie --- .../05.Orchestrator/01.Installation/docs.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/09.Add-ons/05.Orchestrator/01.Installation/docs.md b/09.Add-ons/05.Orchestrator/01.Installation/docs.md index e1410ab2e..9505a189b 100644 --- a/09.Add-ons/05.Orchestrator/01.Installation/docs.md +++ b/09.Add-ons/05.Orchestrator/01.Installation/docs.md @@ -6,5 +6,19 @@ taxonomy: TODO - + From 16936b3b9269600d164f435a57b296981d246c9a Mon Sep 17 00:00:00 2001 From: Kristian Amlie Date: Mon, 28 Oct 2024 08:02:46 +0100 Subject: [PATCH 5/8] Remove a few example cases after pull request review. Signed-off-by: Kristian Amlie --- .../05.Orchestrator/02.Examples/docs.md | 96 ------------------- 1 file changed, 96 deletions(-) diff --git a/09.Add-ons/05.Orchestrator/02.Examples/docs.md b/09.Add-ons/05.Orchestrator/02.Examples/docs.md index 0935f1a54..d70e48e15 100644 --- a/09.Add-ons/05.Orchestrator/02.Examples/docs.md +++ b/09.Add-ons/05.Orchestrator/02.Examples/docs.md @@ -151,102 +151,6 @@ mender-update-orchestrator install $MOCK_DIR/system-core-v1/manifest.yaml mender-update-orchestrator show-provides ``` -#### System rollback - -In this case, one of the devices in a system will fail. -As a result, the system rolls back as a whole. - -Set up a failure for one of the mock interfaces. - -```bash -touch $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL -``` - -Assuming the device is on v.1 commence an update to v.2: - -```bash -mender-update-orchestrator install $MOCK_DIR/system-core-v2/manifest.yaml -``` - -Because of the failure in one component the whole system stays on v1. - -``` bash -mender-update-orchestrator show-provides -``` - -Clean up the mocked failure and reapply the update: - -```bash -rm $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL -mender-update-orchestrator install $MOCK_DIR/system-core-v2/manifest.yaml -mender-update-orchestrator show-provides -``` - -As there was no failure this time, the system successfully updated to v2. - - -### Example: Multipart Manifest - -Sometimes, the full system topology isn't available at the time the bundle is created. -You know what components the system can, in theory, consist of, but the final combination is known only once it's installed in the field. -This is often the case with systems that get assembled on the customer site from modular components. - -To accommodate this use case, the bundle comes with a manifest that isn't fully defined and needs to get information available only on the end device. -This is where the multipart Manifest is used. - -The top part of the Manifest which comes with the bundle is already present in the environment. - -Create the bottom manifest that didn't come with the bundle. - -```bash -cat > $MOCK_DIR/system-core-v1.bottom.manifest.yaml << EOF -api_version: mender/v1 -kind: update_manifest -components: - - component_type: gateway - id: "G13" - args: - - arg1 - - component_type: rtos - id: "R456" - args: - - arg1 - - component_type: rtos - id: "R998" - args: - - arg1 -EOF -``` - -We're updating the device back to v1 again using the split manifest approach. -Configure the orchestrator to use the split manifest: - -```bash -mkdir -p /etc/mender-update-orchestrator -cat > /etc/mender-update-orchestrator/mender-update-orchestrator.conf < Date: Mon, 28 Oct 2024 08:31:55 +0100 Subject: [PATCH 6/8] Add failure example to the offline installation case too. Signed-off-by: Kristian Amlie --- .../05.Orchestrator/02.Examples/docs.md | 48 ++++++++++++++++++- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/09.Add-ons/05.Orchestrator/02.Examples/docs.md b/09.Add-ons/05.Orchestrator/02.Examples/docs.md index d70e48e15..774a299a1 100644 --- a/09.Add-ons/05.Orchestrator/02.Examples/docs.md +++ b/09.Add-ons/05.Orchestrator/02.Examples/docs.md @@ -217,15 +217,25 @@ device is accepted on. ### Example: Standard system update - real gateway - offline +When updating the gateway, any failure in applying the manifest, whether it happens before or after +updating the gateway, triggers a full rollback of all components mentioned in the manifest. Let's +try this functionality in practice by simulating a failure in the last component we update. Execute +this on the **device**: -On the **device**. +```bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +touch $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL +``` + +Then execute this to start the installation. ``` bash MOCK_DIR=/data/mender-update-orchestrator/mock_env mender-update-orchestrator install $MOCK_DIR/system-core-v3/manifest.yaml ``` -The Orchestrator installs the rootfs for the gateway and reboots. +The Orchestrator installs the rootfs for the gateway and reboots. So far there are no failures, +since the failure simulation has not kicked in yet. Log in into the new system to resume the system update: @@ -235,6 +245,40 @@ mender-update-orchestrator resume mender-update-orchestrator show-provides ``` +The orchestrator tries to continue, but the simulated failure happens, which makes it roll back +instead, and reboot once more. + +``` bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +mender-update-orchestrator resume +mender-update-orchestrator show-provides +``` + +This completes the rollback, and from the listed Provides data, we can see that the version we +wanted to install has not been installed. + +Now execute this to remove the failure simulation: + +```bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +rm -f $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL +``` + +Then execute this to apply the manifest a second time: + +``` bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +mender-update-orchestrator install $MOCK_DIR/system-core-v3/manifest.yaml +``` + +The system should reboot. When it is back online, log in and execute this: + +``` bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +mender-update-orchestrator resume +mender-update-orchestrator show-provides +``` + The orchestrator continues updating the rest of the system components and concludes the system update. From a3f2719d2a6b408e238bd762d62dc97894000dec Mon Sep 17 00:00:00 2001 From: Kristian Amlie Date: Mon, 4 Nov 2024 14:57:01 +0100 Subject: [PATCH 7/8] WIP --- .../05.Orchestrator/02.Examples/docs.md | 83 +++++++----------- .../02.Examples/system-overview.md | 10 +++ .../02.Examples/system-overview.png | Bin 0 -> 36069 bytes 301.Troubleshoot/03.Mender-Client/docs.md | 4 +- 4 files changed, 43 insertions(+), 54 deletions(-) create mode 100644 09.Add-ons/05.Orchestrator/02.Examples/system-overview.md create mode 100644 09.Add-ons/05.Orchestrator/02.Examples/system-overview.png diff --git a/09.Add-ons/05.Orchestrator/02.Examples/docs.md b/09.Add-ons/05.Orchestrator/02.Examples/docs.md index 774a299a1..11b3d7013 100644 --- a/09.Add-ons/05.Orchestrator/02.Examples/docs.md +++ b/09.Add-ons/05.Orchestrator/02.Examples/docs.md @@ -9,14 +9,17 @@ taxonomy: This section will explain how the Orchestrator works through copy-pastable working examples. REQUIREMENTS: -* you can run Python on your host * you can run [mender-artifact](../../../10.Downloads/docs.md#mender-artifact) on your host * you can run a Docker image on your host * you have a Hosted Mender tenant and can get a token + +* you have the preview binary of the Mender Orchestrator software (please [contact us](https://mender.io/contact?target=_blank) to get this) ## Prepare the environment +Run this on the **build host**: + ```bash mkdir orch-install ORCH_EVAL_DIR=$(realpath orch-install) @@ -24,6 +27,9 @@ mkdir -p $ORCH_EVAL_DIR/bin PATH=$PATH:$ORCH_EVAL_DIR/bin ``` + +Put the `mender-update-orchestrator` binary in the `orch-install/bin` folder. + Clone the repository which contains the support files and the demo: ```bash @@ -90,20 +96,22 @@ With the previous terminal consumed by the virtual device, execute these command ``` bash -CONTAINER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aqf "ancestor=mendersoftware/mender-client-qemu:mender-master")) +IP_ADDRESS=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aqf "ancestor=mendersoftware/mender-client-qemu:mender-master")) # If you get multiple IPs use the alternative # where you need to manually get the container id first # docker ps # CONTAINER_ID= -# CONTAINER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $CONTAINER_ID) +# IP_ADDRESS=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $CONTAINER_ID) -scp -r -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 8822 orch-install root@$CONTAINER_IP: +scp -r -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 8822 orch-install root@$IP_ADDRESS: ``` On the **device**: ```bash +# Merge this script + # Manually install the Orchestrator and the environment orch-install/manual_install.sh # Export the variable the readme suggests @@ -114,51 +122,6 @@ orch-install/manual_install.sh mender-update-orchestrator -h ``` -## Examples with mock interfaces and devices - -These examples are all executed with mocked devices and versions. -The fact that they are running on a device with an installed Mender client is just a convenience to have a common environment. -They could have easily enough been natively run on an x86_64 machine. - -They are all executed on the terminal of the QEMU device. - - -### Example: Standard system update - - -Point the Orchestrator to the bundle manifest to update the system. - -```bash -mender-update-orchestrator install $MOCK_DIR/system-core-v1/manifest.yaml -``` - -Once that succeeds, you can see the system components versions with the command below: - -```bash -mender-update-orchestrator show-provides -``` - -The Orchestrator keeps track of the last successfully applied Manifest and prints that with the `show-provides` command. - - -#### Don't apply version if already present - - -Attempting to apply the v1 again skips the steps. - -```bash -mender-update-orchestrator install $MOCK_DIR/system-core-v1/manifest.yaml -mender-update-orchestrator show-provides -``` - - -## Examples with real gateway updates - - -The most complicated orchestrator case is the one where the device that runs the Orchestrator gets updated. -The cases below still use mock devices for the peripheral ROTS devices, but the gateway is going through a full rootfs update. - - ### Prepare the real rootfs artifacts The real rootfs update artifacts will be created from runtime using the mender-artifact tool. @@ -170,7 +133,7 @@ USER="root" DEVICE_TYPE="qemux86-64" TYPE="rootfs-image" -mender-artifact write rootfs-image -f ssh://${USER}@${CONTAINER_IP} \ +mender-artifact write rootfs-image -f ssh://${USER}@${IP_ADDRESS} \ -n gateway-v3 \ -o gateway-v3.mender \ -t $DEVICE_TYPE \ @@ -179,11 +142,15 @@ mender-artifact write rootfs-image -f ssh://${USER}@${CONTAINER_IP} \ --ssh-args '-o StrictHostKeyChecking=no' ``` +!!! If you hit an error looking like `runFsck error: fsck error: exit status 12`, please use the workaround described [here](../../../301.Troubleshoot/03.Mender-Client/docs.md#fsck-error-when-creating-a-mender-artifact-using-the-snapshot-fe) instead. **Remember to replace `system-v1` with `gateway-v3` before executing the snippet!** The same workaround applies to the commands below as well. + +Make two more artifacts that we will use in the subsequent examples: + ``` bash USER="root" DEVICE_TYPE="qemux86-64" TYPE="rootfs-image" -mender-artifact write rootfs-image -f ssh://${USER}@${CONTAINER_IP} \ +mender-artifact write rootfs-image -f ssh://${USER}@${IP_ADDRESS} \ -n gateway-v4 \ -o gateway-v4.mender \ -t $DEVICE_TYPE \ @@ -196,7 +163,7 @@ mender-artifact write rootfs-image -f ssh://${USER}@${CONTAINER_IP} \ USER="root" DEVICE_TYPE="qemux86-64" TYPE="rootfs-image" -mender-artifact write rootfs-image -f ssh://${USER}@${CONTAINER_IP} \ +mender-artifact write rootfs-image -f ssh://${USER}@${IP_ADDRESS} \ -n gateway-v5 \ -o gateway-v5.mender \ -t $DEVICE_TYPE \ @@ -205,16 +172,26 @@ mender-artifact write rootfs-image -f ssh://${USER}@${CONTAINER_IP} \ --ssh-args '-o StrictHostKeyChecking=no' ``` +Put the `gateway-v3.mender` artifact on the device: + ```bash # This is the path on the device where we deliver the created artifact MOCK_DIR=/data/mender-update-orchestrator/mock_env -scp -P 8822 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no gateway-v3.mender ${USER}@${CONTAINER_IP}:${MOCK_DIR}/system-core-v3/gateway-v3.mender +scp -P 8822 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no gateway-v3.mender ${USER}@${IP_ADDRESS}:${MOCK_DIR}/system-core-v3/gateway-v3.mender ``` Upload the `gateway-v4.mender` and `gateway-v5.mender` artifacts to Hosted Mender to the tenant the device is accepted on. +## Examples + +In these examples we will update the root filesystem of the gateway where the Orchestrator runs, as +well as update two peripheral mock RTOS devices. The two RTOS devices are configured in the manifest +so that one device installs the update before the gateway, and the other one after the gateway. + +![System overview](system-overview.png) + ### Example: Standard system update - real gateway - offline When updating the gateway, any failure in applying the manifest, whether it happens before or after diff --git a/09.Add-ons/05.Orchestrator/02.Examples/system-overview.md b/09.Add-ons/05.Orchestrator/02.Examples/system-overview.md new file mode 100644 index 000000000..6f9eeee39 --- /dev/null +++ b/09.Add-ons/05.Orchestrator/02.Examples/system-overview.md @@ -0,0 +1,10 @@ +flowchart TD + subgraph System[System             ] + Gateway + R456["RTOS device 1 (R456)"] + R998["RTOS device 2 (R998)"] + end + + Server[(Hosted Mender)] --> Gateway + Gateway --> R456 + Gateway --> R998 diff --git a/09.Add-ons/05.Orchestrator/02.Examples/system-overview.png b/09.Add-ons/05.Orchestrator/02.Examples/system-overview.png new file mode 100644 index 0000000000000000000000000000000000000000..b9c34535d418bd877d97b3d7074c7dbf71e95bb3 GIT binary patch literal 36069 zcmeFZby$?&+CMsgf+C^PB}hp)bf<#EP*MZZIdpd;prS|)CEZ=p-O|!A(jeVPH|H6? zd%t_{>+Iio<2t|pUM?7hXV$vcT6f={dtH1{RFJ~PB*g@QK-duJH_9Lonmh=E`t$)h z@Z@nJZ5;^oJ11oWS1LOPTN86D7?qQ|9gGU*W^MukxlLuon>#Fhhkv+T!gsuv z`o7J9yzq%*0RN4bMDGFpspV;8TY5Y-%2-U?{i^BP)8?*gTvlui?Ga_izR5!wF!zR6 z-i^=k7U!9hw{A&=msjGo)7yw_}sNnwT-Kg3fTjgoKd)a+!nr=|Fp)2mjg5^)bvo`M(IDp-1(Y5t)YnkLuTl6$^gK{2 z3meA8I#pG^9X{M|PacNw6B>>hzIeyTc&?&M7buVwtIBOToG}1S?K{&L2=auPJ-%``PM-FcHQ)DpsL_-lGD+$ih}UYk?GhogX@zU ziLo{s3kQi_mY0LM_Ek~U&;47!X#RXa{PO8%x6;SqLx${!+?lk5{9og2#h|(0r7OP` z=|7?IH^&&pn=@Qn4$YO=Z4OFc-cXrY)}vW_>C_xNb6V3FG{x~aS^dW@6IafK^l?&I z1yS%%>qU2LNc-imx9}e!1-FLXRL2SE8pQ9uB#z}NyHj}uhZMGRRD;V?^`*GO?Q8^r zwQB#T0`C6ptwQ_cb_;x|3i`8z5?vFGPUuPnSbQr}Tj!5I*~CoVPXc-cNxBbSJP}8a zYzYW?21I_=WNgZLjunY{6n#UJl{Sji)qcB^Scc$JxiI&O!YuuLQRhmig-z?NpmX-w zc>Ao@rpy>mQ(LBVt95_rB)Ol(HyCPEcrQ4&%kC**4BFb%Sz!YQ{m|%#morBA{>q`9 zqpAf)amElvMK>HdL21r1e$ymxWkgPn1)@*8=ECjgVaBC*C!d7)bZ59$80S|GmY?n> zzb&4RYWfDRAeK03W-h)KP5BR0PyE31pS?FgF}{@^iJK5`9JsJ$G* zdidx*M~!D7mRZF;5Ed&Y6<>seUEXAaq*j67wj&Xw{tZ}6U!}VUuMrN4;iLV1`cksf z;5{yzlcC}05t7Q4SDlhw zMdcbs1_X6xv_ zh}}yS-1tI;imw@(Q=f(LSW?oJk|sRB7>(t9eoghB+JH0NNSdkaYY&37HwJv<9NqYQ zY|fjR^}~!e8h(%WP1r=L{UvH~slK~^H>_NJo1c!b711`=lP1(R3mG1yuIic$uKish zHdo9PiTVjMzW<%QiRdzV_-Av(1GG3a$m6x7Zf6>{m#nQ|@j4St(cJBZXCF(#cL?xq zDZ_}!at|XKS{=xvAuW%I;=D;pQyX~Xwn|AAof^|z-{rYHCf=37AjMAbay+3?RU7b! zZ=cL8X=8`Ahpz>D&(5j0nSvLXf>>N}dWw^UuFwUZv=i`Oco3A#Ue%~z z)-GM0i(Ja`&JQD|PGZk(a_R#SzAlGd=GS2ggS(XpzKJ(djq_o2=8)U z)QgA2uggNlT*=9FEmgW6tK2suq&4Q;w?NZIc@aFzhW*(ReOuf9%P&7U@aN%Vn?z|J z-N!T5Da*&DOSBjS24a1W6~Ccrj6OGrvDEg{B;3d6E-z~#lm6qvK}}L6CD`}vJt+m9 z>L_{l7^ox4DA!ajF&4JmutB%bjs6PPSX1gz%i>}MF=iUv>5?;TfqUSykOHX`o*jP( z%cr=!@hMj+JFb+emM-s{pW)8L%o&f?9|~fXD{=Xo^(m0E>264`$yXFrY<_J~n0qFjU(@^GNWDra23UIs72 zQ|%h9y$@Jxyy*P&Vs($71z1^x%y2|*Ry7z%M+rVH9`?_;=%%J55@2y5Vf+vu)`2$L z<;<++0e)pXB9(9SL&>bxiUw{nv?ivvYQ2*iF=Ngm^Ck|3zUeKK6dBAUgg8Wfh+-+o zu&BavI!Y2F`+@j@=g1aaCT_Gb$B$U2^GCy9Yz1ED>&vp=e{-`yUDw0^eav7uJ4T=G zwd_;&x3*J+X;}A?)6p#L%^JBellw~!JFXpnd-af=;(iGbq78Oq@9a^m;*ENEI{-_E znr;2mRs30{D}S2|U2;!GnEysxl? z->U1xCc?D*wWQxw2#0&NQX1C#LyxZCKbv~s9qKqd<6tdXI9C`61t&ZU=6Pa|`_z_v zrpg0;ypXNgZH$SU9uic3AHh3a8d0FVx+*3{w%n!5PYPoGN_&|{ZUMu^z%+ysQ3Z9j4yvCs?!Tb}7lDu{vJ<}(TTBJVg z(tdDTdqxI!4Y3ir6%ip&4&fIqp3>=4NQvz!tgv*6@%(zLADFf6p1EDm$dhy_GH)Gr za*vg_IUqBtsBIl2C@xh%HX(-jRy-3$zU>-xys{ZX6OUlb8d~;!>C1-GBj>Y4)!KLv*lvg7}8M zIMZ$$`G2|9>SNYQ!He9fm!PfOY3esnI{fY|QdPpKLG(NTZ9RtGdIEJ+(AC-V$0}GU z=H!*0e{l;1_~F#Xz0J3up6Ke1V!YDpns|7)6JPXE*_6VeAe^JGJY8X*XrhbUF%9jt z*c1+x21QDw@3-kSyNzrfC9A4fU1tKja4IR4PddsqQwnK{kEcJ=w#B0ToFM!%XN~b} z@>4si_aNOb^#P0N0R3L`XSzR_t|(b=w>*g}Or}14>dkE`qB_|&G`K%S@6XUWya6hU zUI~!GW5uSS#xIz8;)Rk;C_u>&Chh+mD_Iq_VC^}_14c6+IZ@VMt$ioZ zJ$>KeozG4TPHKiRLE^k%ZQEJTy8u!J6zi-i6Z2)IRCmsvVp_Lp_2FbgTn#+ZFv75k zjW0nApKpsRf0e(r6zShrCuwQ3b{!$0yRiIaq`f7jNX&onFky5-;_?vX`Mh%jl!W_2=*D zqF84lQ=bYie>p^jr7QSZz5<(Hf&S<{o~sgz-;ElY6wac34xJJsE%5hgcf{^`tt<%{ zwM*TW{gU|J$WQinDd@dq$k3O>k0M!n^Rq6G<0Ht;4eI5(0|%FW(J=Y$7~h|69{0=% zA3Jb4Z*<}7d`_j13uHH{K*vHE~{~M9Or3CAL6iWpJNii*lXL`TMTx*P0lM<*;)_? z<%7AnxFSSc{O@=OfPy4?M+i%|zasC@8x2Ya=!UumFc;piZC$(|-Zub9q2y`q!wl=FB1LiGU@ahWgVQPmBl09HWIT z_fj{i;$(hw$hknbeIzJRA3XoW9;eZnNJ}?t?={=rg{Ws3-1OQVe4^|~h~`E8=Ow3H zf?EJzl{s1B>6i9GhJqh!KbGiq!p|bMg1T%IAF(_V;z=Ez~houRr)R$uP7_MyVy?KVqB;bKS?S z_4%TWSu3Nz{O;9#Qey<}4MuZ)<$+41t}+{rQzPF%*Mp(PM_d-6hu+(KA$TpfSlSz4 z-z=P;UT|GY^9Y($U$UAS8}r_0cC~0&A|z#u_3)JJ@Ikaa(<6Be{RP5xokI`H-K8TG zX%+PzvW11m#9+k}k3iSciqn(5I1^vdLRE(*mR8Rj0JkDF2k@zP@^S)3w$>~L#|4Q*#?(>b?4AYASPMVQO`5c{X`FahRF8w1)#s#X~{W z$ivcz-B}{GRbxjg^ktPn)RP&I66tV07U+_p?^EWQ5AssSe0Rpw$2Vl zFbNl!jT6m3lQ1^=kNI}a4pw)^F*ag_S;4GmDGB^1Lv;Jjo$WQK$^UoUrrvC@se;NJvv)>H{M#;+yysIWyoy+He};B1REI}8S-;+@v?F8vHcS%h>fF@fsGLi zNeUp&Vh+&Z=VCMF;p5-|8*=g)fw>HfOu&YQM%-XF19o;^6K*~}9uAIwBJtM29MDPw ztAE}Vl9Vw(%7n|r#F&p02IlAHGzN2-aGHSmOxU==90o?bMtocx>_#SrcchGs1SD-8 ztPKEhnp+!~!dUHWOz%EGGF;%bB1D*)gXOP{-#t;ZGH@~hCJ0l@ncFzK{bQ)Axiw71 z$pFbsc3w_4UJgEP4qk3vZgx(-e+*KCIXD8Eh@{ER#=`j*U8Gn9fWrWcH9+bqK;Ui$ za25e^2bh7At%Itqt(7n}asyPznRn_n{_qGQazX zit0{n1q_V-TEx-71!jD=5HRnrkBrO=Y)oN*eg93N{{Gzj-w2DHotvM_#K0KLZD7O( z=3?jK0P`8~n1K1Yc-i@kfrA5V{#WXbwkA%l1`e>-roc&o(*O!|cN!{&J61CO>u6Uq z7?MwH99&>_b}$>4Dm#||H@5%>FEbmv02>=M>)#H{inQv#pIC_X|HTuby8-{?27q~g zJpzVJE(em%64)^G%Y z9^oN>P(bmC#K1!gCy2ZR#`>ds*t}#jeYXA}5ETgW=C!KZ)b^~an=15o=O5P#IgSkq zu@Y2FCVvJx224y;rrsciV#DdmLYTU%`tWfMv!#}+GXY|50v$8x^N}t-e)uyBREYFm zoJg}MJ;~>X)F=@L*Fu9roQ`|7oTtgPae}r3wQHmey0(gNEc5`M?XFQ$W=Fn56ktOO zZ;UdL*AqbRhwdSNeV|9aQgIVMC#eFIIEckgB{N=2MgkvTqX)>*_vmwVn_0Qpa*!EUlEIvbVdV+Nj>O`F?W4hE z0&-!9}@Sh^*=3?FfXuuX2vG>u?5rXrw)vj?xCD?ZTZOI01hx{|zn z>{l2c(&KxFDpCxKuqjTYRrnUjiOqb--P#7R;lua>CdBfe_JuOoxxkMfe9T1)K==&? zw5Ix3zJ5uZd#f!Pzvp~^*&g^l$0Z%``T49?RDY62-k~#|fWT7CxKLx*-bAAF8v%uF z#AL5MF2F8;hkOP#`KW1Bs=F%CTD@_C%2SvKbX=bKv}an{j*XZup&j4C*`62{_CUCb zU#?7o7*^nI6UNqKfh%7wASdvo`9zo-#a$T1eKswjYXujBIpqbFYlxn%>njo0SIJZ7 z(R3oU1!U;b<>R^QZsm!km$pi!CK4SQ76H+5G0cGYp2UL|zTK|7-dC+b6t$6u1v$(x z{XEX)L(G~HbHrggyorN_arG>6_3d&oQun@;iqgQzfw<%VuNqYk6zqkg-s|P(?dR3a zXpzZ@qg2vg{bF`?PHlCFU^k)TDQ9}#pZ$TW;A#9MKETo)lLvPS6DqA#EcKD!hG5Md zxvGDo4K#Mo_79s@NRWIL6U#WgXd#)kr@)Pl)V77x1DD#U>b|TiVH<*7?E6`05-;B2M_Q?P<;$}})HKb5A;YSaKRtRh522?Tck4kld+uQ_*Pr?;87#J3LSSY z%!dT<2K`=w>a~K}7WVNfrpB}k0MWY@PTWyx`Jp8+LhM}4Z z<72TWy;~2NmU#Q|=YwBU*!`ZYY!d6s*;N6=q~jvXsbLjdg@JUKd@_C;k!$4yPpl{2 z5BD#BHdN(EtO+`)tJrTRm*~)PTG81kd&lpC*7A07UKV8c0L534bFMp@R>#g-!0+LU zh7YSQ*bP;+n2s}=d~_+8C`h#@kHco}g(j{kr98TJiB8#>)qrt9%IAKn zX`rn6F8rwCEtff*n=hzQO|W_*sTqH_aj5|k)8VY@il}j$v0(MG3K)zr+8-};5O1tn zkHBnLs@7tHMLEFM>UrekD_MXYsA#W2ErZ5i**1QYX{}%Kf4s19NpmR#Blu(Knb~I* zRu*ciJ;gsZB|sT~IHW5cIllXI{hHI<9QBTy@}@X4s$T~bzzKD4%jc&uVsiM$(@cZm zTKu`>fsp?hdUdW>}}wZey9ZW7%7|bIH-~ z?OKxl8@xHhDQT?`1|K&_hwm5UF`QUjo~}%lR%!2AG%HWP$=!?21JtejwFcVX=>!d1 zDF!C9BPVv$+E`x}%WV3^c7OC#X}0#fS6qp0QxshDSM&^%7zDfx z_`KN3USc6W2Vh4Bu83T!=`8R1d}#b$M30qXL9x;%Kc5l=?5pdl^f#$=X#+f}T;GnW zr%%<{>WwaM)J%wpHEV|tH6Izl3z6Ftnih>{&OgwZ>l##|Nstpq89;Zq8s7#S~g3q=f(h zGf2KP?|PCw=xsS83mtxLNOne|gW2D7>tVoascX?cCI!#ELP!0Ok(Sx6fJ^j9yw0vf zb~1r^EBXs$iZVMLPki<=9q*-=Vpl1=L6Hi%Od%D1c)pDHaWCJ}OXL#JYK9JaErJ9u zta_Ufz!H(p<*8Ib*2bE&nTh4lZ138`4`QlmQplJ|R09py$taY+Y>xuYll~J3tOQSB zF*H*zO$(86{K;@y!mRmf=55ALz|!kxh?>S#$3%*M-OfN)N{AGwjer%)@}LL?z+MRw zjG<1yS%*BF-U;%j(A{a+gR7@t`M_fdYi=g?+2S8Z7e>u2f?l5tJ5jOeNURkDl5X z-e4eeh+t0x0lrVoxI*IzyNLu}KgV=%*{|TTDe==BtH%fEEA|-6KnCS}xnh4`6}k#_ zuJ!e0%YQKIIqMhDq+lGZVQ*fgSfFoRpkxJyN*>yO(|$O5XLpei9TRTf4m<<$^NxC; z8@RLT=a#<9mn-=PdD$a5{)MEq1=oRI*L4Zw2JjAx@D3aJxJ}fgMO425;8oxqCcqo` zuvPf5arlPRsdHF=OvY{B%o=v@-Xo?3+}XA;%f0KHR3R2$YY=9)4qkZIO6(w>$F4KR zvi(OJ-DseNX_v-`uK^Vq676Ox9K%#RD zCWncc`IpaKSWIm=bI!ZC+k(A z(7bMG(}45waj0g!OEeQx@A-QakQ4zGG^?y$XTX}y zP4c`%Qe3uwF_j?t0~ zxC`Zd#^+skkR7>Jm6e~!?UfPC#0EaSqikqcHXu?TCK=ZBx~C`axK~Y0&8#=-w8HI= zHUa)JBNLM(g+OBV%{4}zUh`{?gnp0n6;m$f&A>%iX9(FdAFqup_4VEQTjf1&Sy|bm zKV7z#=Lh$D2L~hjP1Fn2thx1DI_i!_n2jqn$#nK6%X2I*j(VmlEfcm~0yQfwat;Uj z%#SR1@(T-d5m`fPhZ_mDF`rUwXX}(G1YOMc1p>>#0Mod9t{AY%cx0Bk z=Cf|E_ng5R9O~!i?m@v$o7?4=>)8prs)mLKO4B!hJU24J=0@tA3slwAm~uE}k_4Q` z>0-5uMZD3+nCpG6NoQT>-5?`BhwD$UP z{_SQTYmPLWnkFgetE{4~ZfN$~l*02v)6LtPJ-yAT%6i(Kt>ev`OWTQoq-k0RPM~;J zRtF$4zlmaVPjBx#yY>;jKB9>i_06X3(P!&6=CItELTU{(a?{}Q z1bmD-QU-hE^rs{CV@{Dyy$lo@eLQj56<-F1bD0h}Spymu7nd-WCF|PJ*=adl)gRBh z$80@QqZUc;%@9cQ`*XGL>Q8fz#g8xA!4^1DA7o@@D;!py7HQYM5&slCM0v9xj)g8B z0Mqk1C-J^H?-IB^;v{(TB=7TQY1F?Giyyy_J(gYS$&%JiFG-VbI*3x%0hV=}aczbQ zJ{t;uP6OkJQ&m+>%}I_=<+f}ZUkVdVMKZcuI;ZKnPh1Db~L^wbtaV)Dnf2V0@9paw zF4WL7`|){Ls^WUyyIT470)dB#9#D6&Xb=4~ zmiqjN7H;V6Sr@JIo{x)L5@78_Ye22WYISv{1Z?n1()O+BwKUE1zJk+vU#N0!-^@^& zdap4D1pf3X{Afu0R`Jc7%%HE2xIIstAegumHf+RW&9@S8+VaUlz(O}x6ZS;5oo^Nb z5Kv|1Le2p`tuT>`Lg0AdfL5yd*LE(`O?Vk=mHDw2hgyK>%A)v5jgVU?zmXZE6->IY>jM8a<^N z7lxv&`G@pw1xj_9wG()D#!Kcu!ZQIOW52cSJ;4CjchKvIc@LG8bCrJ9X*3C7Pb95` zgv8dAW!KhbK^Fura(#3yxwU@K*YyD{65m1s?JkqZtg>IKv*Rs`pz$5m_fb`?7~o$M zf{bjA_t8Oel3YIhG%5FQO`Y@D5?!>>m4EW!UWI-uqEfWGBx3m2zM}nvMq|65eqRhW zJX51wM4jrfqNe7dq}ChKEln2&&r_4U>}*N;NC?s`)UKTX?sIe`>yYka-7Ep&Dzk87 zw1VJ{Q&vv3GS%auD(i$WGc6qBa&D~WV6CbMT3|?E4ma$w+IBu)r@pFBpAFE4c`i*1 zXB}oohZ8P=%!}05;^!GC@AZgh4?zn=CnwV>yi@lTFm+~Je*b8mNQjB9q&d28%~Rlh z7!$&RNS9cbx=a?Xr-SszXIWmI)pMJVl*bF53xjn8^k}4XGh|t6wRsYDzEAM-y4s#~ zozLI6+&=HS^h$f)#B23Vi%e|5TKScQ+$`1Ojg2C}*K*q}JO;mGll@BAKLJ=@g_pb8 zSOW^0SP|ejoK@8(6s!TyI$L*ac2GqgG`ky)PukR|pi(8u=@>~n?$S@@)P0~^lG9pI zQQ>sceV2*n#B6RdL6qg>v@^goX_Px@6S%Pj^g=raeI-ieCg7t^GFCH0@y=a%fc1LT z?KMe_Wpwe2US2aZ^|pmw9+5o8d4Mrmtn&h*Y;T`N%gDlVlRGXkdoE`^?GjFVyzQP0 z__|7y{%RlqV+Upj2e($P2w2=)m9(#Mb31F{WU=5=@Kv~~R5jaNZ@bh3VdSi*Ofo~s z-u@vr?y?Ea%3b!SA_tfVZO@L#*~4pR(V?y0iH+I%5|3j6y}nr1v4RqjlEg$|kM*?r zqY~f7B9|S3o5E93ws+c|JN=tOX@`TF#k&m!@$vB)&*G;lHh?W(Uto^}lcZQ6V)X=!MS~mbR}zc8R|ef%JP|44X(tLnAFcy-l)j zdyZnWEjD?su2l#?2!V(<^oX^gATtx+y-`;_b;;4UGCSY#9v1mzXJ$KbUsJk~d zwhG4=Nm-A5AP?sPgb5j0?%9)pqhr2Dwps5p&{5GlY<8s1Mq}wP4AFFq5-gj!A*DFaJCCOsZ83?CcSzG&< zL&^93wasoadWmiMBNinMrn+_mD<8K>ugP(w#cD+8bN+-nB7f`oyu$X)7n zG9&@{d#oivkonBnOkAU9`SEh3Zv<^t7B}KPkv27>(=cM%wplX8=c0Lkf2rQZ8W;D= zY@x>2{Vge}u3U}sp#>C5ul-L!U%%2q?%%&`>(lkUwgzHnZ_i`~2IW-}kF74HZeV&k zYUAMmZR6De=kbK+P->ctv^0XGwfr?Oysb#-ftTs4^#5=#fiyk6J)L^5H|s z;s$!abMhzwm+hx>F#^s-^3DM-xnOrEvMqggG(A7}oRMq!T;;7ikOU~W-GoxadHnu@ zQK{eh@`eqxp#Fm&$lb{6HuJYCt*1?ZXlTUj?t^OLjvqfZ(*lz*aUYa= z96RjHHHw!s93<*`AG`CDKgrL}#|jK(E+N<2J1%Ly;gL@~3+J)Ec#Y8Jd>0X+0EI#~ zfhY(~5kVWvhiYA~u;L*ZHxUCSb2^>n?Y61zGv)}?L)cVXZvWE4Cm;YbI{A#J++Jlq z^EqRQj>6yi^5R+4q*V+c__DH*02~W@wh-t30!~1;Z@FBBaBZ*_9A7l z;OHo(Zo91DXhinbl5P3C#ZUaG1?@Hmc+3$e2Lx7&**cv@4j|}{_p$os%Ca_!0h#Ke zrKt1zkVb~od@3YT#NsgrQYEj~L~lm{*GoW|lB&5i_g}djulnU6?9SF z^7`U~_L(`{8s}2S&LmEKgmx4WNqm}!70~;o9*noJp52Pv+jTA`v=>Qvw^Ml&ITD*l zU=Rz$_Wb{E zs6hS8%rYqhyQ^j5S+cTppjjf(L>_qnP_9~;a zsBj{!5VwyvZ^0`@u(gb}IAFX;Y!M}JG_w_vl?&jWu$0xFH9G#qA8I7SywZF;i~oj zR97&m6i6n>^2>;Xl{(DaVg6TQYJ-f!2drNKyl?a9=Anh7WYkZ!HoQ>LVwvVD0L1x3 zM^`3A%DumdUnxif54v0E>Okh8QIkJIdU`^w*O({Q`qZpalv%Ts(P_H9gw@rv(xn)T z*bRvWpovW{g(wd_IF;wU#b2E^X99sz4k!N6sX4_%2*6wKTn~t}0|gtU?J6#_r)q8bdu^ z#S7XD60(qL5`xa;rjjI4H;{9ODjpwzpcl zV6xU$D&8&{Uo_}l8nu!!vr(5}4g^R6>`+oVo9unm$TXvsaW%_Aurl_6wZt3X^NBEYL{Ee|bEf0AO)YCx2)uqgn_3Bz}^ak`uMT<{MS` znwq(v6by-ZxvR#^?;nQBmoYHTW>@f}rA9>!d=9>gN@GjT7yw|drj#tz;r@(Z%|kV6 zO${0+s@-V@H+)(YRBRm*1i`9Q;9q+Ad+Q8Au#W-NU508jY0)7Gf6@$Rl5JG}zMIO} zLHiCr#RBS;4}F-Y){%qkaYYROz<7WJL|#R@X`>LVOrYmAD26i)9Ol+}`F`UdrVAXU z4IF)0Kl;+sc)hgU94_43aBH8bA*qli&~Q=mup7FfBA=yK%#@-X?K7@=u~K}HSJ(DJ zJ-XQ?xmnjPB$OBNi6-Iqs0EsGezZXg3KAsYQ}dcjWeW1nXFAPG|51?7(aa?44X`P{ z>}O7|a*8(854{g)1xdX5=pPM$UL7Xz4jUx&1>j!eCQ*|n@NwPC8C_4u#2WKH6})f+ z@J$hpSc&ZdUZIyW6yf!qdV;J}VS-d)eXQzPdgCiil6tI61BKnh<%s}}*ZFBTiQReZN_1@f)R0;T`(^)9>^~pW&(o z^`P!r)DkGdpOIh&@dPTJNJml|$)zdc8>y)#N<<9PBUP@|ED#VlHBfHl0n1vz>=7R{ zwP(iD_7{u)Am%WD&P2D4VE@{~kJ0GWQ;(NU(ZenT;(s2aO@^f;SOz4G8$|Uti9Q&Y zFPHk(1~vm$;i->!^cW542;S?(BChdTdx^$A1ir|H^%2(XoM;(y?cs{im%q-Qnqd4I zvV;v<_{FYrAN1a+$MAAvNJYC`o>B5ET#Mr;572tn|DSc!H*(G||9%(XA0g)7#|vZ( z`me!n|L30le-^y{C2aj?ga6HD{^vXY&u$z1XPfyS?;L5~z}LUoxt LX%ne6dLFX z)gNO91mc^*>Z0cY>v)7YIL0c@gWhKpmnruS3{1R8`I*iE0!is0imP*FW3_zFhiD-# zo`q9Yjp3RYAabbM76M2jRbp8hbFn}zN_beL73QPM^kC07X?d22dBsbB7I}OOBYieO z7cnpZN8XPs)txlS6>FL<`&!LADu}y2=NevVg3v$EJFt6{D-bcD|ezvW2i6Ro4Sg(8V4J z(r+oLs^-U&#u^$XdNJJHA6Yni3>5lyi%gJ_8haWpt9O!ay7Nt@AJOpdWYEIcvv%0G zbu0tdtdPF!DVwe~H9uOvG-e|{>L9vRvanDN;7H)HMmQsck;!2k=WW5+88OUsZL<=Q zZFang&BVy4E+gYVAW-o26V^rnP?AI7#kH$GFq7RF+LaZ8eAB^#wvzK7c&?WF3r)v! zrqfj~8;>VUfvh<|MwS`KyYq^5rne6ISjXO@tc@`Xx*s6o`TlTUAEw|kadRu(o)6w0 z&AKKmpI%1C#AN!Rd2W0nleE~I^J+X_HPvkN28&|VRr#@zLBrA!>W~P{glW?n(ti(*wWFV0Mo~hfYpzh@r zTvq1bJnM`LR2Q1?WhY4{LBY2g8nwNHtT=($E<2jQb4h;wa{$-1m~VoeaT^(?Z#H&S zm>(W3N0#)i9YWyL)Z_qDM&Wb?mBu|5A;k86&pg(7NU%P6^a)>XYcaiisbpnU;c>KH zxHDZ|X^GIiJ4l0DWm);5arHC+Its2VFI%4Ok~i+RT{1B+3}2t##w6Xp(*c$7?g%2Y zk<2hIqfTl%+8-8IXUqg7B*SKti|wpKogvfQ-WUFo;-fRQX+x^6&&G062GpR?%JhsR z0B3l(IhGf~#9Y7)>wO8#{S@nY^`{<5NdDWt|5H4p<2lb;v!iSEV(pw4$!-#@`ie`P zA!ENv+kt{L9HCoYVrGr=Xj}+B@{6KN(8;-ukZ64)@c* zH-UDweJp)L2t8H+gq3xk_vY=ks2uy0y#iOY{d^$Hx>NG; zQS*3%JM(c1+F-s^*eI9F_IG-nK=aXT3b+_pApm_Az_y-+oqpcRWRa#ICP%qTR2hp+ z2W>odDs7Mw1#}CS+j2rVgoIOdw$2#<$ClG<_CHH}RWlz-6-OODjOE<=QOOgfHE+6ih#S!tPVFvGLP& zUoqf=+&(s)$m zF=}L(c(D**`fKv;Iws!}`EPdhBInDNK!<~#06z-o#V~-ukVemxG|Li-?A3l1QXLk7 zsk~7I%L3qEJ&o5nAMSTp;&OQ$(UraX3BYq9C%dyRdQw9vM!5i=MJjypG)^wtV};Uf zfxe?t(tN7I9S~m>V8cKI3aYEin8WF`lapZEBqQwUGJ3i977@>#0M*ba%FUH_-kNxQ zeR(>zzev*TmNdD;v$9IaK#av>rgM0CiowUo^~OVonG6K#if`RIMyKQE)92H*izyTR4dK?a5x88vqFY<{_69=;n%2$;euhuWCa(%9Vi0&O6J5a6y9$# zevAlv;0nG6T6njFM14E+3oM<6C6?L;1)VK2FrL+HL=}a3?$+w^&g{pgymZ{DZ2x8d zmw2GWSm^voPXK7EmZdf`MKZKO5x~!Y;dK+)fH=ss!+NcF04T2yNGWry0h??F*AW*WT`x(-`~5pr$_B{S2)k@j-Pg31wi{(^!D5r=*}qwxJyWU)NJxfi(%EvwO_J9 zd}HUO_;cyK-%5sr&+GQvWFG=ItdU{l3|9_7fOQ33jAh_;PW<9TdP3UBo=U$70?1#H zJy4)&^$JL|1)28ZVyo5tUcK+dxYLZTX5Z_rFLwHM=c}S-W4WcrtJOBM3f`BxsSOR} zS=l@pZGUCec}80kA)4RyV`w*kc8~evZ#YihTPlEv>HVkU1(5pI0uXUTkFm0*W}V0B zZcagQovwnZYK;{P_OZCZLI3Q0dvX-$Ssv{l;)X2}q$8np%VoXsZ7?)Uai0d6zr!v~50VyT03Co)AW}C>RhX9mFtTwM+WSbXj5U-N$aX=AnCJloPGDzfzm^lRl z@5!6u`Y&(QiGb|w>M(38LTY#1b-oC={E;O`A9-+i$OW`=XWiGFfiAJ;x;~7{qUAl% z*!W5`X#G8zjHhrMxKIF;E-75!s$qTTE6dvv?TZg^o&Fq+;vKCf+ztcqzp9D~O$nj^ zkeK)N@mk$dXz5g)vyt2RBbFMZQv_O_mhGP^Lnc&aYYovba13(L?zHAjnad7PMQNBO zX_ZK)`kpul6f+^sKJFDS|G-3T5LJ{2m=Qz5dPs-W|@+!>6HurCGf%-me>H3!xc zniyln&;rf+ktXlj;|a&k;J0)T$5pR9pzG#wNci$h6YjKIiv}c2bdcur{t$t~pSm8~ zrEA9%TBAU(+L7D-x2(l*dL_VrAX3)uA1EmeFYd%~ntB8X0QXKtSNpfNifHb94BzXq zk6(cvAGd%gF;?%6OCRvP^ap7Ebnl%am^m%Roa@$-1IH^ZT|2s7Fo}pL5$SNfoRP3u z>Fo$?)CIar$onj^aG-<2h)u#F2HfjV&P`Ydz!gbDHaPZ9f%7T4Ru3fA=V_EXK}O(o zhixTLC8gqyui{`~;9gHWudQU`j#iCb6|MO~%e7y&Y%C`iu>DP-B{?z@&kS^(Op_;N zPiGFl)0$sjaOdYT;Knwe{a%D;etcvK>=rQ{Q>T^HXBrgC~kpdTd zAOdF0DkeZz-F@x&I9td2m=D=s$L_V6YtWtdDt}X3>q&rxZp`-KW-lh8{`5P-`p@tG zr?~fwifZk)gv*G6fJ#ynFrY|Qf=EUXkeop>DmhC|1sK313P_P8mgG?6C|N`#=OiE? zO3tYWMe4PA@7Le$(S7^&?LX}p=f{CvyY_yDic!Vre~X3bHgbu^}%3a*^`R?@_rP&2(p;|>|I4k$x{IV z0sVt!TNXU(90o#Ki2tZXBWYq{KY@vpQyL}&lU&##qV?`w6AN4Gz}h4Utbo;H!{P%% zM@)(0mSJ%WOeO}knk~_|2=(4DGi3YA@yaCiylF~a!M9$VJRs$v8 zd9GI@PSCYyWjqPLQ0UXMF^f%ioVL{|Ft&YQO*b=Wc9`nEUX1|fcY#4ckRF!@9RD zrte?FqBlj6NB8o2W#BeG;!)?v(8L?J;392v#xj?P!D}xsqOI{#7UGG61*22r9mr?_ zZkKWQ+^S7Z@R(_tVMnA*Oj1Bo?VU}-QW#oiwn`D}guu-1eY;U{ za2cH86otL!)ONyG?IhyJmdWxt?7TaYl5z{M@FCsi+L%v`Dh6QYKc@$@2>fLI&bc{w z^Z9;K($nz|!oaqm|JQqaTo7@4<>$W`Uo3)PHGyeCW)~FBn|ef2N%^(+ofn6y42%f; zbEn^Y680H0w z1ZS)UtrHTe938iK1ZwQYNN)MHM`l(;^mc?t^y|Sag^n#p4$SwZL>gRwUw<)P)cWTg zK*cMae?^y~K}sY(mG=;|Xz;gHjJo}mlu^ZsTpkaB{w`qLzMlO7rCEp1LB&yP|WV$Qe4M9eIqH2C=84QU1?>QC2 z7*8EP23$g~ZjM@74kj5si;O+|aA*HurA)(SsAxcvhBWj@{3gsKm42gX!h?aZ%END| zY+7n^a+0T9-+5$Q{m%(Y#g(Yip`hR8&wSIu?0uxger+)z$${aLMgjrCyEqYs86UKHY9Z zXb`3E2U8wL%~-Bx_Rw5Pr1;3m<;wLAaE5g8jj{CsBuEEj9t?&N4SjtodX{}KcJ$c! z?CzePAB3$8yg^pGsL$t%6ropI8AQB_sM(`?X6j8lgiC$4ebgX0i)u4YEPL;^K)g%>e#(ybM6#msPUM z5J3~Hetj8sp3mOuC%Jy%LdaYXlf>@kBGgYo03gCK-FgUGLCjbXGiDK=WPh$<5-yj% z^zipHuGzhHPUVLW*YbI~yO%>GK<7A@}_L5lNE36|GoV?+Xg4*Pu`RW7$~NwCFdD_= zZcKcd$9>J*dC}v2xl`F>Uc}^;izwf^Jey%^0D-Lh{2KZdE@|SZ-;`g$kb$}?4(2a_ z;b5M9H|lPNVVn(M&9NzpJbe@qzA{fkY_$CL#`9YT5!uy5u;~T-OBzOL(RFM6gQ0ASC{)+7j&upt8b)gRuMSOTL*lKG76$`)-nx$D{9#pE5E2 z1-?uVeIPhhm{Im9ZVePl)!REHRUsy=Llm0}Y*4XzNNU9rE$$_Ng(8}ChS*N?kCyje(?sx~&%N6=f&g#G2*E?}iL2F(zlha*gjj`g3S+35f00!=!A zP>`viI>Gx|tA8Z#&?C=3ar@rAe8{Vj$B*A< zeXw2B+Sb;WrLA99QL4F5;A}LyYg~?XJ}~aHH2v(`SbvU7$JkpbipzyoplF)G47+JD zSU?&r;QZ1HO_QOLqyqv#cx>z(5Im|;{zvR{hpJ?Qu03<|?}2uFFLKb_Z4S|i*v+Gi2%!53hR{#hEqH~-sZvXNhG5m7B@ED z0Vt)xs%hQpNAlDs%U8>AX{p`vXCAd#xG&RZy|QkPtq~EZL$ttHP0y}NoxHQIWp%LE z2j0!-Un6mo3%``7ro0x96bYuO-?e{rO4uHQlU2jZt8!scZU#8>h?p2|xZB}BDJDM( zm2g^i$o6Mtz-ohOFiJ>xI31=r;MCFc_|ME-LY}83;?(TkjxsR1{QQ06Sr=C*Wj!}7 z>iqSgs!y6~(@g=IGQ(D1x~xJF2en-+`Z#Q4--he39wOLDRFykz>+46*%CQyq$t?ErUU^YQlttZ~gcXmHf?*BttGTVr|9zrswWON)lx672M=l zito=E$uVhp<+L{84m2>Y^%FUI`s~RGb!k&mQ_8cAPn*M8voG8<=+U)vXT4)y3g6tF zrF{@BEGQO8Ta^x4Rwk^lz1`Euv%Nigwv0`*##yJ(wBp&bXBtK6*&bUip=oATgM%*^ zX&Xtiva@^hpOZK*{1i89*9*gr2>{xasBqc8Em-jfo-L_8PS@e~<*#aW=NTBVY|K4n zD4!~J{Bng0shYKQx=!f_N5ic+xBBzgJ?90(tfC>M!j+(g8)pZY$ zOiTpF$mjKG2@_k}94LX)tp-XwR_mvk&-?nY@$<{bMY^cr=k_(Sbq=i;Tlg-Z4x_H{ z-6(RJJ-xGQr|PzCLv_dKCMPG2yL(wWEDy>vjS|+@YdGCfm%0AzII^n`GH=QCTdhx_ zfhvq-QMdn?W_G2lTp&a2*p+}&zu(>DIp+4qdmrz*vvbOFcXMA>R#r8fWs=w4h_Bdc zn$gONmH6bu3+Fj?tvG=`#3w)5^18@tM+z)`q1F2?@k6C{sY@gJ{CUS=!Cc#V7Z}(x z*Ed|`_TqCpmYi3=qrRsw@G2dv+BMW=N_I)N_*S|6^I8R|?ImwE_&~7L^{i%Pc zQ)>Gb#Gl*J4b5|1Ny0e0<8Id5jK@}j^();ZNvN~>;c=<1TuI|HS=d;rK*Pdw9Q-+v zCWboDfOw?*`ZTW-2)gY0w2Tl6K?K9dhTUO-Nr(N`(qD4T57}-j6N`h6kKtxr@xpHz zWjL$U)i`SwGJgabwmLO(qV{4I7G%v=e$&s5)!>ewaJ8&`)rSOL)7B@ME{m}n&8&3V z&g8JyzSb-s?7M;Vqd-EF(^uvgX>4P23qqO4WO&fI+7m14c|}FaX^QN<8EPbpV+>h- zBm*qhj0*Sm`7#WuXnLzEa1g-l?TL_E{dJz#Oy1i&UJT21q#7-KOOCqdckL)_pEedZ zRJ@M!yg(*|4L^T_aIE2H&75C}<8@4pwY3JLqlzF z+!kP-q0dFjsv}+kq%L4xs$s*c6e; zCb7j~KDCbcxQX9im7gOii9FwN`WRr(i*%yin*IHnuyVU_xh|M0^x1$D*9Pvwy+W)p=@>;5g1b3W)u>lDXS8Ulwz4-HmSWr+nvVo#`and1>cYG@swpR-Y zg!w@Mx0Nr=Vfj5XUHcQ`UH;!tjgH9wrmLWNjl{IrpX5o&MOb$VF3YR?5tGgJG$_~% z@nE{llaIk=j?h0ss8d z%iR&C4g&)7F9#u9`P#L&Pp_XrX=B2|lJWDd7X55Kal)P|q;T}_hs%G}ahe(dCs005 zpNAS&c~)e2Zf!!3%loO;vz^Wy z_e>`fwlX7AgH`z|%Izd8s~i-@nvfAwPbi6qaPwM4)DSjaghE0aKMT;av5A^PFG;&_ zkBZ(~S0TDLBDkX>Q$2&t@0hPOw0Jl;+aOCF{3Ztb+kY>jcP2Y5zNxWlKRF+d4CU45 z)9ojsc`R}u8{tgbGNbQ4{Q|;m{5RBBQP3AAAhl-eR3(#Maj_t5q<~h+)D}hYxpZ_* zGv#2+Dz+~1+8G#1ysgfJ$Z_W)5(Ny)omo{yL<)<%_Z0_+BRhe&Pavat2{G?NAxq4XIaXKh6P(g&0elKd&$EV=NjT_8l$hX?bC@R8e9L#k+ zPX@}5odJ7+uv9^=p7=-wR*zD~G&VQgef{=KoU4ZwWT~~4ONnuUew-VFkZ zLJg#3!?u_K@OKy%ve2U@!Z`IzO@WH9qUJo%Ns)`}?o9X#^PzV3>{&`8ST1f?0kcI! z4%kCmo+j3d>&f2+a;<~Cojd1H6L&W^Z}C|5Pa`|U+k1Xv4wpnp>kHxOgD*$K?oOn^{M|wV9H=OBA zpg8He(D1PWxgMCMxTZC^ghtR5u^*H9m&18MuT>I10yJ(tnhyIX&m&%o4wY#ta; ziin^l?5$s?Jb_xf3y=e(7-ju0=Tqz7pE$banoLND3Ni5zJ9%(Fi|Y4?S12)htt2Tb zWaWXf8T1U96%2xjZr!@oRF}kCk3v1h2np%T&-Gmys@S?)57tJp(TK1xMu2>lzqX-i zHT*M_x-zQkQ+Ooyvg?<>MDB9UuEPGhJVIs+6|=)#K7q6erOcE(1w5+#WHYIZ>f%t} zoztjmaYMyeExljDl2TGC=ir4W^q)Q*Kro!wP8Ku#o=$i873M(b>uKb7GFrL*ICB-v z))OdW^)L3dq8{}M0msfq`&Pmc16K-)*~PA-P*HDE#ZwIs=j((vJV8NzlM?*&@)I4M z#^z?#@$nGK6R>FAwHzD@FrlH$pwzxZ-?(=#L>zg0`1c>z-v3e|e_wB9KDJ`acCCj5#!jdOoa+dxsjy#n5n{+s+{p3kH zpjXOssI^bmB@Cc&*^?-}cj;WFURhSA2JUe!EZL8{6CGHkDi7y~l*!k@H*}zqP{MK) zl$0bIp*}XKXQ&>8n8kog!k-Ks{_Ge`>(<)l(lu^wZoqwJUpV>swV`By#GRlFt5SHu zIuzLZ6xj~IB2(PROzfE=O%S!FdSFYXx$3_EHg3l z-;-$*4y62h;aL)?Kslcbb-yVW?arDnQJO4=ipez>G-VO7w-yKnZzj)n5&wd zZSaYXFNlv_Q+iOzugA1Gns|P&^D|n8U&zo`9Mwp1zpaXmIY|5;g!ua39l!N|w)|lN zklXxc7eg%%|NcMS{?XC-U;kGhJ^Hr)z9-#(f~&2s#72LIKv4d`G;^@OE8sl$BnW+w zpPvufTDg~(h^Dx>c%QwMAr##`3YbKL?{*EEj?c6*=~K# z%9Uq9hv@J1^|Mu5r{@TG9LOAY8u@=5mGJBOCDzqs^uN-8-rQt39ds#>em@ns@ucZ^ zoN0L#>_}n~5`|KmYOfvS0b%ndl-LN)?@Svu_IJpV@+L{4P0MYzma+*Gp{@+&W814^ z^Yt|IW*%dHB-0>Q)=^PYvshVKm3?svNqU}rZa9wLQGCp6J>$0G-osPGloKb;Y@8)L z%9*2x5X;;UD|kWgyouJW%17M)q8Kdz_(WoB*c@fjcp2xw*qG z!ewzN3#V0Q3~}BJoBbJHr>U<`ANCyV5DJIP@82IuN{)0jg-ScGO{nbba`pnsgAJ>2 zu=>0smR|!}Smj-H)qClXt7dzcCYEcI?W%X;he{U1Y-3hd5I;n9xx=JG`EagbO(q=K z(A-~QpJVP1FUW%)+@6e#Q*t)f`Lbfx7QeXrRVswe&}Tm*$FL^RW${-*?*mS~2WccU z?G_^fKJK1d8XfT`3O5|5+G27(f9|vF`Zk&0cb#YBl~mBx&%>p58Ai3@K(H=rlsWXt zmpZ%Hg5A)k`^{Mf^Iu<@yngzMJj6EgqW|<}Jg4Qg{0IXOZ7cWq3yy;6mGhrRdEo2J zJxh_w?BgAimZ+4aU!fPubjUekL&F(A03scrl+!!g@=7%ANr_9)QG6iabMRrdq_J@r zkW$*_;&4X1Fj4bxgkq%g8YwHiydk$HUpt8wDK7HQZS~h#2%I&^3jbSt#^D8JQ zP;{*c4-aQ-f{Dwx(2pBU6{Mq^~N@;IvJQiu%chiR7*?9)XLrI z#8OZe={QawtaRECQx!m|XT|*sdf|<2<5zEjXr+kwd_kx+28|NgW-njLAf)5OWS^W` z>W6^{_LsrDmB(NkeD`6#dsnQh%N3?m13$<6SR+$i%V|SD+mPWi%xgak>*O?_9GXzM zgWY;37by<*k4G|?r049USGRM3Pe@Tq3u+qJl7k1Fi?RDL3s#c$jqU1c5oG4zD|w8* z+Vk4}5{Iv#x-^puvbsfktOaapgK~ojxN|}R4YDo9>l&bCQYs*zJPGR)p9qFQ^OguP zpwW0oM2*b@RSEO&To}aWh7V1ndE&qqO*5ks5a03&s1rlvFS!P$K{E$NX8S0(TSxSN}U14awfIPHjdwYkXBtWUzbo_=p) z!x>4hY$m`5ye32fX^T5u#}qna`Fnb^v}4XmaH9w((Knbj-#vUSm9Eb0>AbX8!2FYv z#TYkQMKp_?LSVmgeV;I&WBoSOCrK$wyHFmAlY)Ha&q|_k#Knv=~M>c$f`HiH-M$N<9^3=O_4GAuepUkLasnXFnIKx?rBE9+fufGlwt040Sw z#SY<&gG1E)jUj(~i>>5FYJ3bEf*G1RYi20k+@27lV%5C2ztbMqShN42C8A9Ys+#~R z)`V*GTjO$t+E;!&Ufa?h>k4*A(E1T*)eB-SG(>}*1anwkRQKG|20~`v-cAp`g|mW^ zjxOcmt%crn703GTk3{XU+uI14BoEoEKShq&@Ho&=ID@Jm4z638U5WMbZ8dg=N;2Kk zSS9G;C*>CEhi}-JjAY1AzOLJbtx=Tl-{>xLOjFqE_>`|sMmE{KNI{3zI(E!^sKT{E zT!Irasb@hUYc>HNlMb%5RR55<%|%_EO1F~Dg~6i;+T5&T(xRzcXlgStHy5A1ItlQL zR`lQmu-Wb&)rom|%Fw=yXV@nrqu-O9?!c<-tc4q!ZONBxva2SiBO{TQ*ZH=ZPeQzX zu187w&b@mo9YSii>waz#0le}sPbzm?tvr5wk5LBBBWPOd`kE;Deio8kXZ~~PBD*m% zUhAJ9$Z0#7W+?cpl|IG905>}tHEL$2rBhq`)@j*bljgxTAK^P(&Dca)-jk`EP|zI4 zjDsz@(ZVUHk)tOWC*-a!BO`-Jo@yI5-sV?`DS^XSbc=irmz=6@;y=t8lK;@w)rCan zyEGR3vDjIJ@Q7Y)q3wiXypSbfy|5$VP&YUvHWpkEd~(oDV(&f^vL_p!neH$-eM<(y zW&|UYh0(V!9TMi38rI5#=mZmAo-oujGV;A~!?qc`nNw|6$cTFH+8CzpK9U;k7hK*5 z&&>^1#EK-md2{mC?b|Gj6Il^#QAvEGZrW)|43Uid55m;iqMdKaJbZ}bLRaGF!EmFe zSKip&t?EZir8)jhl72f@2`iH0y)VS;S&|FSn)&99TU3nhOn1t82+c|i-VeTSBygmB z;Dnz3mAaEw{;qV=tAK8{mFe4R8(9G@Sa(k+5V}Dak<|6{-%V7Sbad? zguhz`yu-Uv@VI@rr{rimsYkzh-!hP?Kvcvpi^}s__PWnJ|IsiuFpUjXjpR_6oEs8@ znxeZy)F@qFvS03}H* z$Vq!Q)u~#XYW~#A$zm)Hx*=|KpUi(o{!4^|gJUGN{I-IQEC#&VMwu_e3fwxgbuV!7 z@o9~e2OjM~p4C;KpocZitm3*QC!>FSP9v+`7UVFjHYaqVdn+kqQZ;PbWBH$!x7DY- zX9O!qakL0}O(J+rSoYC(1@uPOT}kONs7*9BKn`WSW|S7eIhK0irbv_1k}L!`k=+Gh zWDRwQ=o(NkDXNKmRXlZap?9LGP#uy55VLYvueF?|kqRbe6SsF^T-1V#-=8o~X0?Uf zBr-NNO(mmK4dPM!^c%61B~(DwULpF?)2@R85}lst2t2w-RAP#pk3EUsJh z4zA0qjnmMGj|0w@<^FvsK!D{Qyz<`Og**-uU!5Fz{`~n<IY47iQkl!TS-T86o z-N@dWn#jF9)2JX1vsCLrVodUDqAin8yT^|9Y_XE@K}D4@aX%R8-B}2a=AKc)i*cc| z1YK?AFt%esACsNxWb8&O556w;EZvmJ$ytKL9s(GH*K2zoiBg~e$**UC{=fgV&}`$~ zFTZj3vetB!*T~RLOc+B=8e^NQS7zzor(bY$@oo3(h<9xZ|Fl8YmoIg9$9G3@Gl9oM z6V6Z3dinBT?pIlifUCE5Nr_(}-pjO&VBos+2vAKCpyFC5%TaTl=E1Tm0IHa$KL>;8 zVw@%#k5O^xJ^I_QoB1UFPwFn@s@^Xz#xm^;$w$7tP=MD^#cAe%#uybq!w9kSp{km<~RR`&lb&g)^^b5w?_kQlF21!#d4t zzOn30PYM=2Xzpdq&dvrI*ounn$>SvHnhcnbOrJw5Z9y(XUppt`nYc!x9B0HgXNBkg`ZU=L$AebK|cxXtCUBsgPB5V!^7 zZFd)!6ov8m)+mZh&Fs?*VxN^dnH;>ZjeG72e8NObX_~N~)73LFVZ&vB)Bg_FnoId^ zj($ZVI9cT|OJhoBOD*1h29{D0T3a5BbSRR#SZv8G#xVHo75@mL%RE7}IhPddkAr*< z-VI3ugDPMTPtb3B%utk(m_<7zBqt|31+HY5N?CL#YeVYZ9`kN3qOp+McZd$>%!5Fm z%9%<{O--1&$pD8gxoKo=BZJiqGtgUTM${I=Cp=>RT3R>~xK0^NW!0{U2c3WGRDSYO zPo?{?Q{Y{VkQ5$^oPRP?4d?b{=dZj4`mTGr?xth@4LfSaNn%JBRB9@+f zrAtxO@k7N|KII&KbTMxWbsGORcuso1z@hh^M$COW{CWo~moUrz{+1!M)x|kDWu-u1 z39mJBoB4}CYM`=J4i6+u%(gzPO}3-~FvIUmk45=^OIAD{VQiV6LF&FSa|q_Uz-UmN z_4L&u>VzF8d?8na&UVP{dZ^K0P%{*;EBcwp7)EmyVpKI5=^m_%SaX`M*x}zng(WAe z9lZwwI=e8a1|68t)FsVh`|C{4ssM z*KEGpmP~eGa7TlHy;1H}IxCAm-z`|QI82IuRHFr*HWxC8s>z_f{uY(0H-*bJZS2aA zK!zH)DrT;=5?Fe+wA3rEDZ<%tgTFr3$d3H@^-HMkWG}KFi!ut42j+K7Wqz>zajJd) zVI)V~z)V*#y|_Kd8$EBI#tD!X?$N5|iY+bIJU%#hPhV~`G|tMwp%14!okI#yfG_Ft zQ3Zs>Qgr!?ztPPR*+#S-2X_IVq&~gES*g4mGHh7pFv!z7sX}8Np8V#`_Tj|7%X~jp z;u1ZtD9a+AQ!3b7t!BSOsUi3mRoat!@S zBxIiO2r0A<2x>B_g{IjWxdusTsY;;hNQ{o6d40TPM~F!0RvWAD8aM8OJIi^b(p?{_ zhBZ9CDX*%_T)y;tQ@}Y2M0b=wfA+w8NQ`;!`hjCCV{=isTOTc0(H-ASPFl6{mDTZZXWT~yqn`ET9g(dX z+zE~EJIDTHCDgQI6T0@UOicCv4UrQ`(`Tu89FnN*O`2hd!65+s{L7+Hf%jCbIX?n* z&4){_ojR4Ri50Q4+$V(HR0T(5OMi#rj;g^#$Z(pXeR#4zX^e0|S2CT;3S-KCw~<8@ z-lUr7+@&18(U}}TXHdL9j*rj%v@~4f6YNc}2w((pS{7LN9k~-^kzo#Rmhl6^8$(vF z`@Xz*|MsA=cXbN(Hu%$kRNUzn9`nG{n*B^OX>k$o*vLcTOM1E>%ac6i^4%UqheJV} z%u!j_Gs@(>w|YgtiuShd@Hbi>^9i*crf&bNwMo{NF`onr+t}9 zmP19wLPC{=c!HqQOs387D^TTh$|^iTp0z>hp9psS!Xlrc_hKBhljl*WbN?WJBfxa> zjP=me^y0{J0*&rIXNrxljLbXjQOKPEmSdw5I8F90s6zS@{oY&Lk$>D6#bfC84Ll3+ zv(Eug0BjVf`##?~7Rr3ccj;W}?Fd53cd#d&dE~Qy{L+;xp-*Fq>ZA`mLX?h}k6HAy z!vkSBm(Y)KHjr-=Jrm%gRIZdfInm2fCxnA~HJa?xKAz1GY8Wv4dV;xIxW^_7?1NR$5<^cYKM_2JB=s>bZpGErRfiX|v6E8Pc7{KS&PZr_S687g_gu)XS@*(* z6+K@ntl&}xibx~R=sJ{YR#UB_xj&a?cQX5P#}2<9>glQe$S*Y?PsGn};S^$mg7>t6 zh1&jG_1um&7%LEVgP(uUX_Fzu&?(>mD-you_tPGH9nqBo}Xo`7rZdk91Hg&l=#)z;Y<2ktu?fNDH z@3w}DSyOxFw4>hW!GQGmor>hdVkf7u1U&>7Eq!~X9WF5H14|8@0H9o1MQjz9l-wts ze3Jj{Y5CcnsQiKg)qR43eC{fitzy?sR?zE~fB?&I;6}I>K@J0{bbfo)XLN&Gkd=xK zJy0ZIFk~g9k+WAhi!MU%{y4XrqL>HZUA6qhtdDobmrx73BOV}a>)mqAAt*U~#?RHn z>}rGs8aCbmkrj1ZxjvvHLtn5|`T*gdxY(yw2ClnhJQTiu{dx~cZ5jvs!8VJi*FK zjy~uWT=ojH&6on=+~TWm7|9Bfh1%l;vq`ULS^!Rj^ctR$k`fyyD75B&>8L)ow3Me{ z3eo^kCOM3s75u%u@D34AMQI{A9PIC;!NvkKp9-hA8b{x`N(>3WftEwO7t6-O9rx!AEZ5V_%bByS>Q8)uxK0@XyA#ImgWCu3bY$kd4aUN zkC_xt#*~mlsR0=lrW7y3Vpxr_xFoL`bYQx9>ZUVhsW*e7ppr`y&IsVo7%*_`NtEOU zdjM5EaMeI1Hj@k5mRqWiHSKbsVIsm;ql$YIgfhL0{e>d>= zGRj!~`eKqwLfysWKr_WNd2DktWpZhWU9?7<_{>V^dpVk(^KTYFxWZMqE}O$iv_S4Z z`TN5p_ENNHO`z0=?MGk=k#pSz!QW~1k2_czlI4T*0xzvvL|pk0qf1UF?C}K7oa;(e zu!3qyXYePbvZ^Wu>XPn2GbK;;r5+`w+5IgdRyHP ziNHY-1umt^Blr$={dEbCmZ2dXsid;m$9Vuq4BS(EQtKMx-+1k;;ednyKH0t`I(_y; z<1f7vMAb?6o}ZrIn57^-ckV9xRveglZ4j;(<~cKpJI(Y9G&5Y!)Z5u{Dusjro>Mpp zZ8)PTlsl&p3H{J>hXG`Xt^(4>=;tW}Tk3T&-MDF&R&jpm)hSVn+E-XEYvB68GFCL6 zF?%$Vt)FVaID{yKqznfk&EdH1&IVFq9h-77o4(8-kNNBmX1&DJY{|z?jCM^lVdo*h z!O09yQ;CRApSyGfTS{j}_jwUO0R=f|Jm!1b%ZHoKNiW^igC=Gyagnm5}a2t z<40qObDj?(ufm`~@^N1I%{i<2wkLNP zdM9-`44{o0hGUXPWZLprHeAF-+lKJVeL4G>p<>6~j^k=9P%t5?)5vkML6x+2Im^zk zG7u#?v$WM9SI*g0r5aR?evBL89;$SYLqZMkO&Q7w-!cuVQp=rYIiwqOqVE5Nvt0pv zWGgFoTK4QhpHNFnU!U-ZB+DTSbc509oII+;%amH<7h`PSt;BXR!7?>({Ad6IQ#dRr zfAT{(ov+w?<}YY2TyZZwSL+GTN%U^kGCBf>i$Ul#Xb+f~Xfgu8f@Bbohrn)3?cg}S z`qkYEs*n1Ab8+@L9C+x?jkzw;rvCh?Tx`2tUNlSZ!Gj~*5fa3Cn3pp~OKagjFj<%jLgL_<@8}AexIa_`S zDK5Qsl(th+RP63f8BwqD@X^5ai|X!jrJemMs#;`z4}6-b6|PB?SH12kRZhXSWV!Jt z6#~%oPpLDgvJ@pZAUP8xL>t#S+Yn|FwtZ85%6bj;C#Xls!7d`&UpOsB#jc|Vq^kQL z{OOb1P5mW#2A+%T-aAKB7l@(1yND_8q$4WJ{;umhoSjidC_YWZ&>ci~h1X-_@L;o^ z{ulzAzw3%4o zKcke&b2UDcag-9}`++ShxuB!xCYTHUfTN(KSbV=Uu-RuR6!~!{Nlx>`Gvu2mgMlmYtnaVvw=MeS=>JdD7Og%_B^i9oycLX z;oE{+$m#rIigj>^0dgrECIJ8b=lT23p5=d-?0>s>4|hPWQu`$k /data/mender/dump ``` -Then, back to the shell on the host where you have the variables form the Get Started guide, run: +The last command will not terminate, but will wait for you to execute commands from the host. So back in the shell on the host where you have the variables form the Get Started guide, run: ``` ssh -p 8822 root@${IP_ADDRESS} cat /data/mender/dump > rootfs.ext4 @@ -36,6 +36,8 @@ mender-artifact write rootfs-image \ -f rootfs.ext4 ``` +While the command is running, you should see a progress bar in the device window. + From here, upload the `system-v1.mender` and continue with the rest of the steps in the guide. From ae36fd0f05e747017192b1d7dc8d2c0bf9c8c3dc Mon Sep 17 00:00:00 2001 From: Alan Date: Mon, 4 Nov 2024 16:08:52 +0100 Subject: [PATCH 8/8] WIP --- .../05.Orchestrator/02.Examples/docs.md | 241 +++++++++--------- 1 file changed, 118 insertions(+), 123 deletions(-) diff --git a/09.Add-ons/05.Orchestrator/02.Examples/docs.md b/09.Add-ons/05.Orchestrator/02.Examples/docs.md index 11b3d7013..e76284660 100644 --- a/09.Add-ons/05.Orchestrator/02.Examples/docs.md +++ b/09.Add-ons/05.Orchestrator/02.Examples/docs.md @@ -24,7 +24,6 @@ Run this on the **build host**: mkdir orch-install ORCH_EVAL_DIR=$(realpath orch-install) mkdir -p $ORCH_EVAL_DIR/bin -PATH=$PATH:$ORCH_EVAL_DIR/bin ``` @@ -58,8 +57,7 @@ cp mender-orchestrator-update-interfaces/inventory/mender-inventory-mender-orche Install the manifest artifact generator for the host: ```bash -cp mender-orchestrator-update-interfaces/modules/mender-orchestrator-manifest/module-artifact-gen/mender-orchestrator-manifest-gen \ - $ORCH_EVAL_DIR/bin +cp mender-orchestrator-update-interfaces/modules/mender-orchestrator-manifest/module-artifact-gen/mender-orchestrator-manifest-gen . ``` @@ -126,52 +124,51 @@ mender-update-orchestrator -h The real rootfs update artifacts will be created from runtime using the mender-artifact tool. -On the **host**. - -``` bash -USER="root" -DEVICE_TYPE="qemux86-64" -TYPE="rootfs-image" +On the **device** -mender-artifact write rootfs-image -f ssh://${USER}@${IP_ADDRESS} \ - -n gateway-v3 \ - -o gateway-v3.mender \ - -t $DEVICE_TYPE \ - --ssh-args '-p 8822' \ - --ssh-args '-o UserKnownHostsFile=/dev/null' \ - --ssh-args '-o StrictHostKeyChecking=no' +```bash +mkfifo /data/mender/dump +mender-snapshot dump > /data/mender/dump +# The last command will not terminate, but will wait for you to execute commands from the host. So back in the shell on the host where you have the variables form the Get Started guide, run: ``` -!!! If you hit an error looking like `runFsck error: fsck error: exit status 12`, please use the workaround described [here](../../../301.Troubleshoot/03.Mender-Client/docs.md#fsck-error-when-creating-a-mender-artifact-using-the-snapshot-fe) instead. **Remember to replace `system-v1` with `gateway-v3` before executing the snippet!** The same workaround applies to the commands below as well. +On the **host**. -Make two more artifacts that we will use in the subsequent examples: ``` bash USER="root" DEVICE_TYPE="qemux86-64" TYPE="rootfs-image" -mender-artifact write rootfs-image -f ssh://${USER}@${IP_ADDRESS} \ - -n gateway-v4 \ - -o gateway-v4.mender \ - -t $DEVICE_TYPE \ - --ssh-args '-p 8822' \ - --ssh-args '-o UserKnownHostsFile=/dev/null' \ - --ssh-args '-o StrictHostKeyChecking=no' -``` -``` bash -USER="root" -DEVICE_TYPE="qemux86-64" -TYPE="rootfs-image" -mender-artifact write rootfs-image -f ssh://${USER}@${IP_ADDRESS} \ - -n gateway-v5 \ - -o gateway-v5.mender \ - -t $DEVICE_TYPE \ - --ssh-args '-p 8822' \ - --ssh-args '-o UserKnownHostsFile=/dev/null' \ - --ssh-args '-o StrictHostKeyChecking=no' + +ssh -o UserKnownHostsFile=/dev/null \ + -o StrictHostKeyChecking=no \ + -p 8822 root@${IP_ADDRESS} \ + cat /data/mender/dump > rootfs.ext4 + +mender-artifact write rootfs-image \ + -f ssh://"${USER}@${IP_ADDRESS}" \ + -t "${DEVICE_TYPE}" \ + -n gateway-v3 \ + -o gateway-v3.mender \ + -f rootfs.ext4 + +mender-artifact write rootfs-image \ + -f ssh://"${USER}@${IP_ADDRESS}" \ + -t "${DEVICE_TYPE}" \ + -n gateway-v4 \ + -o gateway-v4.mender \ + -f rootfs.ext4 + +mender-artifact write rootfs-image \ + -f ssh://"${USER}@${IP_ADDRESS}" \ + -t "${DEVICE_TYPE}" \ + -n gateway-v5 \ + -o gateway-v5.mender \ + -f rootfs.ext4 ``` + Put the `gateway-v3.mender` artifact on the device: ```bash @@ -190,105 +187,33 @@ In these examples we will update the root filesystem of the gateway where the Or well as update two peripheral mock RTOS devices. The two RTOS devices are configured in the manifest so that one device installs the update before the gateway, and the other one after the gateway. -![System overview](system-overview.png) - -### Example: Standard system update - real gateway - offline - -When updating the gateway, any failure in applying the manifest, whether it happens before or after -updating the gateway, triggers a full rollback of all components mentioned in the manifest. Let's -try this functionality in practice by simulating a failure in the last component we update. Execute -this on the **device**: - -```bash -MOCK_DIR=/data/mender-update-orchestrator/mock_env -touch $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL -``` - -Then execute this to start the installation. - -``` bash -MOCK_DIR=/data/mender-update-orchestrator/mock_env -mender-update-orchestrator install $MOCK_DIR/system-core-v3/manifest.yaml -``` - -The Orchestrator installs the rootfs for the gateway and reboots. So far there are no failures, -since the failure simulation has not kicked in yet. - -Log in into the new system to resume the system update: - -``` bash -MOCK_DIR=/data/mender-update-orchestrator/mock_env -mender-update-orchestrator resume -mender-update-orchestrator show-provides ``` - -The orchestrator tries to continue, but the simulated failure happens, which makes it roll back -instead, and reboot once more. - -``` bash -MOCK_DIR=/data/mender-update-orchestrator/mock_env -mender-update-orchestrator resume -mender-update-orchestrator show-provides +# Remove HM for the system overview +# Hateway should have an ID ``` -This completes the rollback, and from the listed Provides data, we can see that the version we -wanted to install has not been installed. +![System overview](system-overview.png) -Now execute this to remove the failure simulation: +### Example: Standard system update - managed by Hosted Mender -```bash -MOCK_DIR=/data/mender-update-orchestrator/mock_env -rm -f $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL -``` +In this example we will perform the update of the system using Hosted Mender. -Then execute this to apply the manifest a second time: +This works just like a regular Mender deployment, including rolling back automatically if a failure +is detected. -``` bash -MOCK_DIR=/data/mender-update-orchestrator/mock_env -mender-update-orchestrator install $MOCK_DIR/system-core-v3/manifest.yaml ``` - -The system should reboot. When it is back online, log in and execute this: - -``` bash -MOCK_DIR=/data/mender-update-orchestrator/mock_env -mender-update-orchestrator resume -mender-update-orchestrator show-provides +TODO: success case of system update ``` -The orchestrator continues updating the rest of the system components and concludes the system update. - - -### Example: Standard system update - real gateway - Hosted Mender download - -In this case the artifact of the gateway is not available on the device but is present on Hosted Mender. -The RTOS artifacts are present on the device. - -The orchestrator will pull the gateway artifact form Hosted Mender using the authentication from the Mender client. - - -```bash -MOCK_DIR=/data/mender-update-orchestrator/mock_env -mender-update-orchestrator install $MOCK_DIR/system-core-v4/manifest.yaml -# Orchestrator installs the rootfs and reboots ``` -Log in into the new system - -``` bash -MOCK_DIR=/data/mender-update-orchestrator/mock_env -mender-update-orchestrator resume -mender-update-orchestrator show-provides +TODO +The manifest artifat is just a manifest without the component artifacts. +So it's not a bundle. +The client downloads the manifest. +And then triggers the orchestrator make the system compliant ``` -### Example: Standard system update - real gateway - managed by Hosted Mender - -In this example we will perform the same update as in the previous example, but we will do so using -a Hosted Mender deployment instead of installing the manifest using the command line. This makes use -of the `mender-orchestrator-manifest` update module that we installed earlier. - -This works just like a regular Mender deployment, including rolling back automatically if a failure -is detected. #### Prepare the deployment @@ -302,9 +227,10 @@ touch $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL We need to create a special artifact containing the new manifest we want to install. For this we use the `mender-orchestrator-manifest-gen` tool. On the **host**, execute this: + ```bash DEVICE_TYPE="qemux86-64" -mender-orchestrator-manifest-gen \ +./mender-orchestrator-manifest-gen \ -n manifest-v5 \ -o manifest-v5.mender \ -t $DEVICE_TYPE \ @@ -347,3 +273,72 @@ success this time. Take a look at the updated inventory for the device by clicking the "Devices" tab in the UI, then click on the device. Then click on "Software" and "Inventory" tabs to view the installed software and the rest of the inventory, respectively. + + +### Example: Standard system update - offline + +When updating the gateway, any failure in applying the manifest, whether it happens before or after +updating the gateway, triggers a full rollback of all components mentioned in the manifest. Let's +try this functionality in practice by simulating a failure in the last component we update. Execute +this on the **device**: + +```bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +touch $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL +``` + +Then execute this to start the installation. + +``` bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +mender-update-orchestrator install $MOCK_DIR/system-core-v3/manifest.yaml +``` + +The Orchestrator installs the rootfs for the gateway and reboots. So far there are no failures, +since the failure simulation has not kicked in yet. + +Log in into the new system to resume the system update: + +``` bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +mender-update-orchestrator resume +mender-update-orchestrator show-provides +``` + +The orchestrator tries to continue, but the simulated failure happens, which makes it roll back +instead, and reboot once more. + +``` bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +mender-update-orchestrator resume +mender-update-orchestrator show-provides +``` + +This completes the rollback, and from the listed Provides data, we can see that the version we +wanted to install has not been installed. + +Now execute this to remove the failure simulation: + +```bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +rm -f $MOCK_DIR/mock_instances/R456/ArtifactCommit.FAIL +``` + +Then execute this to apply the manifest a second time: + +``` bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +mender-update-orchestrator install $MOCK_DIR/system-core-v3/manifest.yaml +``` + +The system should reboot. When it is back online, log in and execute this: + +``` bash +MOCK_DIR=/data/mender-update-orchestrator/mock_env +mender-update-orchestrator resume +mender-update-orchestrator show-provides +``` + +The orchestrator continues updating the rest of the system components and concludes the system update. + +