Nitro Enclaves ACM Agent not working in version 1.4.0

[aleksy-zalenski]

Hello,

After upgrading to the latest version of aws-nitro-enclaves-acm (1.4.0) it stopped working properly.

For some reason it's stuck on launching the enclaves. After some further debugging, I was able to obtain the following log lines:

tail -f /var/log/nitro_enclaves/nitro_enclaves.log
[ec2-user@ip-10-100-1-206 ~]$ tail -f /var/log/nitro_enclaves/nitro_enclaves.log
[nitro-cli:2094][INFO][2024-11-08T14:50:46.550Z][src/main.rs:63] Start Nitro CLI
[nitro-cli:2094][INFO][2024-11-08T14:50:46.552Z][src/main.rs:106] Sent command: Run
[enc-xxxxxxx:2096][INFO][2024-11-08T14:50:46.552Z][src/enclave_proc/mod.rs:571] Enclave process PID: 2096
[enc-xxxxxxx:2096][INFO][2024-11-08T14:50:46.554Z][src/enclave_proc/mod.rs:479] Received command: Run
[enc-xxxxxxx:2096][INFO][2024-11-08T14:50:46.554Z][src/enclave_proc/mod.rs:272] Run args = RunEnclavesArgs { eif_path: "/usr/share/nitro_enclaves/p11ne/p11ne.eif", enclave_cid: None, memory_mib: 2048, cpu_ids: None, debug_mode: false, attach_console: false, cpu_count: Some(1), enclave_name: Some("p11ne") }
[enc-xxxxxxx:2096][INFO][2024-11-08T14:50:46.561Z][src/enclave_proc/resource_manager.rs:356] Allocating memory regions to hold 2147483648 bytes.
[enc-xxxxxxx:2096][INFO][2024-11-08T14:50:46.562Z][src/enclave_proc/resource_manager.rs:438] Allocated 1024 region(s): 1024 page(s) of 2 MB
[enc-xxxxxxx:2096][INFO][2024-11-08T14:50:46.831Z][src/enclave_proc/resource_manager.rs:678] Finished initializing memory.
[enc-xxxxxxx:2096][ERROR][2024-11-08T14:53:46.930Z][src/common/mod.rs:432] Waiting on enclave to boot failed with error VsockTimeoutError. Terminating the enclave...
[enc-xxxxxxx:2096][ERROR][2024-11-08T14:53:46.985Z][src/common/mod.rs:432] [ E36 ] Enclave boot failure. Such error appears when attempting to receive the `ready` signal from a freshly booted enclave. It arises in several contexts, for instance, when the enclave is booted from an invalid EIF file and the enclave process immediately exits, failing to submit the `ready` signal. In this case, the error backtrace provides detailed information on what specifically failed during the enclave boot process.

For more details, please visit https://docs.aws.amazon.com/enclaves/latest/user/cli-errors.html#E36

If you open a support ticket, please provide the error log found at "/var/log/nitro_enclaves/err2024-11-08T14:53:46.985630460+00:00.log"
[nitro-cli:2094][ERROR][2024-11-08T14:53:46.985Z][src/common/mod.rs:432] [ E39 ] Enclave process connection failure. Such error appears when the enclave manager fails to connect to at least one enclave process for retrieving the description information.

For more details, please visit https://docs.aws.amazon.com/enclaves/latest/user/cli-errors.html#E39

If you open a support ticket, please provide the error log found at "/var/log/nitro_enclaves/err2024-11-08T14:53:46.985940848+00:00.log"
[enc-xxxxxxx:2096][WARN][2024-11-08T14:53:46.988Z][src/enclave_proc/mod.rs:178] Received signal SIGTERM. The enclave process will now close.
[enc-xxxxxxx:2096][INFO][2024-11-08T14:53:46.989Z][src/enclave_proc/mod.rs:541] Enclave process 2096 exited event loop.

Direct logs:

[ec2-user@ip-10-100-1-206 ~]$ cat /var/log/nitro_enclaves/err2024-11-08T13:58:12.032674048+00:00.log
  Action: Run Enclave
  Subactions:
    Failed to execute command `Run`
    Failed to trigger enclave run
    Failed to run enclave
    Failed to create enclave
    Waiting on enclave to boot failed with error VsockTimeoutError
  Root error file: src/enclave_proc/resource_manager.rs
  Root error line: 597
  Version: 1.3.4

[ec2-user@ip-10-100-1-206 ~]$ cat /var/log/nitro_enclaves/err2024-11-08T13:58:12.032894740+00:00.log
  Action: Run Enclave
  Subactions:
    Failed to handle all enclave process replies
    Failed to connect to 1 enclave processes
  Root error file: src/enclave_proc_comm.rs
  Root error line: 358
  Version: 1.3.4

After downgrading to the following versions:

aws-nitro-enclaves-acm.aarch64 0:1.3.0-3.amzn2
aws-nitro-enclaves-cli.aarch64 0:1.3.4-0.amzn2

Things are working as expected. The instance I am using is a c7g.large one. Can you please investigate and check what's not working with the problematic newer version?

[foersleo]

When creating the rpms of release for aws-nitro-enclaves-acm 1.4.0 for Amazon Linux 2 and 2023, we accidentally included the wrong signed EIF image file in the aarch64 variant of the RPMs, which manifests as the issue described by you.

This issue is only affecting the aarch64 (ARM) variant. The x86_64 variant is not affected.

We are currently working towards releasing new variants of the RPMs through AL2/AL2023 and those should land in the respective repositories by 2024-12-09. Keep your eyes open for rpms aws-nitro-enclaves-acm-1.4.0-2 or later to get the fix.

As a stop-gap solution we have provided the same signed EIF files as are shipped with the fixed RPMs on the release page (https://github.com/aws/aws-nitro-enclaves-acm/releases/tag/v1.4.0). For the affected aarch64 hosts you can download eif_signed_aarch64.tar.gz from that release page and move the contained p11ne.eif and image-measurements.json files to /usr/share/nitro-enclaves/p11ne/ directory on your parent EC2 instace and restart the acm for nitro enclaves service like so:

$ tar xf eif_signed_aarch64.tar.gz
$ sudo cp p11ne.eif image-measurements.json /usr/share/nitro_enclaves/p11ne/                                                       
$ sudo systemctl restart nitro-enclaves-acm.service