polkavm-test-data

[jarkkojs]

Closes: #209

[jarkkojs]

I don't know how to fix build-and-test-windows.

[athei]

The windows runner doesn't seem to use a rustup managed cargo. So they are not able to download and use the toolchain specified in the rustup-toolchain.toml. So you might have to make sure that rustup is used on windows.

[koute]

I don't know how to fix build-and-test-windows.

This is weird; it's clearly using rustup because cargo test --all triggers a toolchain install:

>> cargo test (debug)
info: syncing channel updates for '1.75.0-x86_64-pc-windows-msvc'
info: latest update on 2023-12-28, rust version 1.75.0 (82e1608df 2023-12-21)

but when it's ran from inside of the build.rs it ignores the guest-programs/rust-toolchain.toml for some reason?

I'm not sure how to fix it; you'll have to fiddle with it. Maybe clearing the environment variables screws it up?

[athei]

This is weird; it's clearly using rustup because cargo test --all triggers a toolchain install:

Oops didn't see that and jumped to conclusions. Rustup seems to installed and is picking up the toolchain file in the repo root.

but when it's ran from inside of the build.rs it ignores the guest-programs/rust-toolchain.toml for some reason?

Yes it is weird. Maybe some platform differences regarding pathes?

I'm not sure how to fix it; you'll have to fiddle with it. Maybe clearing the environment variables screws it up?

We do the clearing in the other build scripts in polkadot-sdk, too. But I don't think it is ever tested on Windows. Yeah maybe removing the clearing and work from there. Maybe additional or different variables need to be forwarded on Windows.

[athei]

At least now it is failing everywhere? :D

[jarkkojs]

At least now it is failing everywhere? :D

Working on it... temporary glitch ;---)

[jarkkojs]

@koute, @athei, I'll go a few diff's now that I've fixed the stuff I know how to fix!

Diff 1:

diff --git a/crates/polkavm-test-data/build.rs b/crates/polkavm-test-data/build.rs
index 2def01d..4971115 100644
--- a/crates/polkavm-test-data/build.rs
+++ b/crates/polkavm-test-data/build.rs
@@ -21,7 +21,6 @@ fn build(project: &str, profile: &str, target: &str) -> Result<(), String> {
     let mut cmd = Command::new("cargo");
 
     cmd.current_dir(project_path)
-        .env_clear()
         .env("PATH", std::env!("PATH"))
         .env("RUSTFLAGS", rust_flags)
         .env("RUSTUP_HOME", std::env!("RUSTUP_HOME"))

Result:

~/work/github.com/paritytech/polkavm/crates/polkavm-test-data polkavm-test-data*
$ cargo b    
warning: /home/jarkko/work/github.com/paritytech/polkavm/Cargo.toml: unused manifest key: workspace.lints.rust.unexpected_cfgs.check-cfg
   Compiling polkavm-test-data v0.18.0 (/home/jarkko/work/github.com/paritytech/polkavm/crates/polkavm-test-data)
error: failed to run custom build command for `polkavm-test-data v0.18.0 (/home/jarkko/work/github.com/paritytech/polkavm/crates/polkavm-test-data)`

Caused by:
  process didn't exit successfully: `/home/jarkko/work/github.com/paritytech/polkavm/target/debug/build/polkavm-test-data-0ea5ae9020752356/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=OUT_DIR

  --- stderr
  Error: "error: the `-Z` flag is only accepted on the nightly channel of Cargo, but this is the `stable` channel\nSee https://doc.rust-lang.org/book/appendix-07-nightly-rust.html for more information about Rust release channels.\n"

Diff 2:

diff --git a/crates/polkavm-test-data/build.rs b/crates/polkavm-test-data/build.rs
index 2def01d..b637903 100644
--- a/crates/polkavm-test-data/build.rs
+++ b/crates/polkavm-test-data/build.rs
@@ -18,10 +18,9 @@ fn build(project: &str, profile: &str, target: &str) -> Result<(), String> {
 
     // Note that using `std::env!("CARGO")` fails the compilation since the
     // compilation requires nightly.
-    let mut cmd = Command::new("cargo");
+    let mut cmd = Command::new(std::env!("CARGO"));
 
     cmd.current_dir(project_path)
-        .env_clear()
         .env("PATH", std::env!("PATH"))
         .env("RUSTFLAGS", rust_flags)
         .env("RUSTUP_HOME", std::env!("RUSTUP_HOME"))

Result:

~/work/github.com/paritytech/polkavm/crates/polkavm-test-data polkavm-test-data* 24s
$ cargo b    
warning: /home/jarkko/work/github.com/paritytech/polkavm/Cargo.toml: unused manifest key: workspace.lints.rust.unexpected_cfgs.check-cfg
   Compiling polkavm-test-data v0.18.0 (/home/jarkko/work/github.com/paritytech/polkavm/crates/polkavm-test-data)
error: failed to run custom build command for `polkavm-test-data v0.18.0 (/home/jarkko/work/github.com/paritytech/polkavm/crates/polkavm-test-data)`

Caused by:
  process didn't exit successfully: `/home/jarkko/work/github.com/paritytech/polkavm/target/debug/build/polkavm-test-data-0ea5ae9020752356/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=OUT_DIR

  --- stderr
  Error: "error: the `-Z` flag is only accepted on the nightly channel of Cargo, but this is the `stable` channel\nSee https://doc.rust-lang.org/book/appendix-07-nightly-rust.html for more information about Rust release channels.\n"

Diff 3:

diff --git a/crates/polkavm-test-data/build.rs b/crates/polkavm-test-data/build.rs
index 2def01d..a901351 100644
--- a/crates/polkavm-test-data/build.rs
+++ b/crates/polkavm-test-data/build.rs
@@ -18,7 +18,7 @@ fn build(project: &str, profile: &str, target: &str) -> Result<(), String> {
 
     // Note that using `std::env!("CARGO")` fails the compilation since the
     // compilation requires nightly.
-    let mut cmd = Command::new("cargo");
+    let mut cmd = Command::new(std::env!("CARGO"));
 
     cmd.current_dir(project_path)
         .env_clear()

Result:

~/work/github.com/paritytech/polkavm/crates/polkavm-test-data polkavm-test-data* 19s
$ cargo b    
warning: /home/jarkko/work/github.com/paritytech/polkavm/Cargo.toml: unused manifest key: workspace.lints.rust.unexpected_cfgs.check-cfg
   Compiling polkavm-test-data v0.18.0 (/home/jarkko/work/github.com/paritytech/polkavm/crates/polkavm-test-data)
error: failed to run custom build command for `polkavm-test-data v0.18.0 (/home/jarkko/work/github.com/paritytech/polkavm/crates/polkavm-test-data)`

Caused by:
  process didn't exit successfully: `/home/jarkko/work/github.com/paritytech/polkavm/target/debug/build/polkavm-test-data-0ea5ae9020752356/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=OUT_DIR

  --- stderr
  Error: "error: the `-Z` flag is only accepted on the nightly channel of Cargo, but this is the `stable` channel\nSee https://doc.rust-lang.org/book/appendix-07-nightly-rust.html for more information about Rust release channels.\n"

How should I move forward?

[athei]

How should I move forward?

Debug this further. Probably reproducing in a Windows VM. Looking at the code nothing obvious comes to mind for me.

[jarkkojs]

How should I move forward?

Debug this further. Probably reproducing in a Windows VM. Looking at the code nothing obvious comes to mind for me.

I'm working on setting up a Windows VM. It is worth of investment as Windows could break also in future because POSIX does not always align that well in that environment.

So yeah, we're on the same page.

[jarkkojs]

I spent a lot of time on setting up a Windows environment that is usable. Now I have an image that logins automatically and boots to the terminal, and I can reach the host via ssh. Also environment has been setup.

This what happens:

@athei, @koute: So I guess this maps to replacing std::env!("HOME") with std::env::home_dir() doesn't it?

[jarkkojs]

I think it was useful spend couple of days to figure out effective way to use for Windows. After this I can reproduce bugs in no time. It was an investment ;-)

[jarkkojs]

Better to take a break with this one, nothing comes to mind ATM to try out with Windows build. Revisiting when new idea pop up.

[jarkkojs]

Next steps in the backlog:

• Environment capture when running cargo in/out build.rs. This because outside build.rs we get successful results even when cwd is crates/polkavm-test-data and the build script is also located in crates/polkavm-test-data
• Don't use env_clear() for Windows build (needs to be actually verified).

[jarkkojs]

Some new remarks.

I wrote a script yesterday (Sunday), which captures WMI system events, and matches process names per keyword when they are created. I can use that to capture environment variables from terminal and build.rs runs of cargo. I capture both cargo and rustup process spawns.

The script dumps environment variables to a text file.

Then as postmortem analysis I look at disjoint sets of these runs.

Probably env_clear() is a bit brutal action to take for any possible OS so that I'll take away completely.

My script has been only lightly tested. A backup plan is to use ProcMon from sysinternals: https://learn.microsoft.com/en-us/sysinternals/downloads/procmon

It has filters functionality that is similar to Wireshark, i.e. it can dump stuff too... Just wanted to make sure that I have at least couple of ways to approach this that do not require contaminate test target with mods.

[jarkkojs]

Just putting reminder here. To make WinDbg work:

1. Disable Secure Boot
2. bcdedit -debug on

Then you can attach WinDbg to NT kernel: