Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Windows tests are failing on hydra #1283

Closed
piotr-iohk opened this issue Jan 20, 2020 · 3 comments
Closed

Windows tests are failing on hydra #1283

piotr-iohk opened this issue Jan 20, 2020 · 3 comments
Assignees
@rvl

Comments

@piotr-iohk
Copy link
Contributor

Context

Integration and unit tests are now running in Hydra:

#703 (comment)

Although all integration and unit tests area passing on Windows when running locally:

#1115 (comment)

Some jobs are still failing on Hydra:

Unit:

Integration:

Finished in 0.3991 seconds
1120 examples, 132 failures, 983 pending
builder for '/nix/store/9w2gv6v9g8gj1q529mr0z08sqz8z884y-cardano-wallet-jormungandr-2020.1.14-test-integration-x86_64-pc-mingw32-check-x86_64-pc-mingw32.drv' failed with exit code 1

Test Case

Unit:

Integration:

Failure / Counter-example

Resolution

QA
@piotr-iohk piotr-iohk mentioned this issue Jan 20, 2020
Closed
@rvl rvl mentioned this issue Mar 5, 2020
Merged
This was referenced Mar 20, 2020
Merged
Merged
@rvl rvl mentioned this issue Apr 2, 2020
Merged
7 tasks
iohk-bors bot added a commit that referenced this issue Apr 6, 2020
@iohk-bors @rvl @KtorZ
Merge #1536
5d06ffc 
1536: Re-enable integration tests on Hydra r=rvl a=rvl

# Issue Number

Relates to #1283.

# Overview

- [x] Enable integration tests on Hydra for the Linux/macOS native builds. This marks the single failing test as pending.
- [x] Enable integration tests under wine on Hydra for the windows cross build.
- [x] Choose a unique pipe name acceptable to Windows.
- [x] Enable line buffering mode for stdout so that you can see the test case names as they are running. 
- [ ] ~Get `cardano-wallet-byron:test:integration` passing under wine.~
- [x] Tested manually on real windows, and in CI ⇒ #1411.
- [x] Remove a broken build job and put back others that I accidentally removed.

# Comments

- Hydra [required job](https://hydra.iohk.io/job/Cardano/cardano-wallet/required/latest-finished#tabs-constituents)

- Note that integration tests are only ever run on non-PR builds (i.e. bors or master branch).

- On GitHub CI windows, all integration tests pass.
- On my Windows 10 VM, all integration tests pass except one:
  ```
  Failures:

    test/integration/Test/Integration/Byron/Scenario/CLI/Transactions.hs:357:9:
    1) API Specifications, BYRON_TXS_CLI, TRANS_DELETE_01a - Can forget pending tx, still it resolves when it is OK
         expected: ExitSuccess
          but got: ExitFailure 1

    To rerun use: --match "/API Specifications/BYRON_TXS_CLI/TRANS_DELETE_01a - Can forget pending tx, still it resolves when it is OK/"
  ```

- On Wine they integration test suite never finishes and there are errors logged from cardano-node after every test case:
  ```
  [localhos:cardano.node.LocalErrorPolicy:Error:46] [2020-04-02 15:48:57.01 UTC] {"event":"ErrorPolicyUnhandledApplicationException (MuxError {errorType = MuxIOException withIODataPtr (writeHandle): invalid argument (Error 0x13d), errorMsg = \"(writeHandle errored)\", errorStack = []})","kind":"ErrorPolicyTrace","address":"LocalAddress {getFilePath = \"\\\\\\\\.\\\\pipe\\\\cw-byron-bd9df28726c96ffa\"}"}
  [cardano-wallet.network:Warning:691] [2020-04-02 15:48:57.01 UTC] Connection lost with the node. withIODataPtr (readHandle): resource vanished (Broken pipe.)
  [cardano-wallet.wallet-engine:Error:290] [2020-04-02 15:48:57.01 UTC] a2bd5324: Unexpected error following the chain: ExceptionInLinkedThread ThreadId 691 (MuxError {errorType = MuxIOException withIODataPtr (readHandle): resource vanished (Broken pipe.), errorMsg = "(readHandle errored)", errorStack = []})
  ```
  This is most likely due to `ouroboros-network` not fully working under Wine. So we have to skip these tests on Hydra.


Co-authored-by: Rodney Lorrimar <[email protected]>
Co-authored-by: KtorZ <[email protected]>
iohk-bors bot added a commit that referenced this issue Apr 7, 2020
@iohk-bors @rvl @KtorZ
Merge #1536
ad94340 
1536: Re-enable integration tests on Hydra r=rvl a=rvl

# Issue Number

Relates to #1283.

# Overview

- [x] Enable integration tests on Hydra for the Linux/macOS native builds. This marks the single failing test as pending.
- [x] Enable integration tests under wine on Hydra for the windows cross build.
- [x] Choose a unique pipe name acceptable to Windows.
- [x] Enable line buffering mode for stdout so that you can see the test case names as they are running. 
- [ ] ~Get `cardano-wallet-byron:test:integration` passing under wine.~
- [x] Tested manually on real windows, and in CI ⇒ #1411.
- [x] Remove a broken build job and put back others that I accidentally removed.

# Comments

- Hydra [required job](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-1536/required/latest-finished#tabs-constituents)

- Note that integration tests are only ever run on non-PR builds (i.e. bors or master branch).

- On GitHub CI windows, all integration tests pass.
- On my Windows 10 VM, all integration tests pass except one:
  ```
  Failures:

    test/integration/Test/Integration/Byron/Scenario/CLI/Transactions.hs:357:9:
    1) API Specifications, BYRON_TXS_CLI, TRANS_DELETE_01a - Can forget pending tx, still it resolves when it is OK
         expected: ExitSuccess
          but got: ExitFailure 1

    To rerun use: --match "/API Specifications/BYRON_TXS_CLI/TRANS_DELETE_01a - Can forget pending tx, still it resolves when it is OK/"
  ```

- On Wine they integration test suite never finishes and there are errors logged from cardano-node after every test case:
  ```
  [localhos:cardano.node.LocalErrorPolicy:Error:46] [2020-04-02 15:48:57.01 UTC] {"event":"ErrorPolicyUnhandledApplicationException (MuxError {errorType = MuxIOException withIODataPtr (writeHandle): invalid argument (Error 0x13d), errorMsg = \"(writeHandle errored)\", errorStack = []})","kind":"ErrorPolicyTrace","address":"LocalAddress {getFilePath = \"\\\\\\\\.\\\\pipe\\\\cw-byron-bd9df28726c96ffa\"}"}
  [cardano-wallet.network:Warning:691] [2020-04-02 15:48:57.01 UTC] Connection lost with the node. withIODataPtr (readHandle): resource vanished (Broken pipe.)
  [cardano-wallet.wallet-engine:Error:290] [2020-04-02 15:48:57.01 UTC] a2bd5324: Unexpected error following the chain: ExceptionInLinkedThread ThreadId 691 (MuxError {errorType = MuxIOException withIODataPtr (readHandle): resource vanished (Broken pipe.), errorMsg = "(readHandle errored)", errorStack = []})
  ```
  This is most likely due to `ouroboros-network` not fully working under Wine. So we have to skip these tests on Hydra.


Co-authored-by: Rodney Lorrimar <[email protected]>
Co-authored-by: KtorZ <[email protected]>
iohk-bors bot added a commit that referenced this issue Apr 7, 2020
@iohk-bors @rvl @paweljakubas
Merge #1411 #1530 #1543
07191ce 
1411: Add GitHub workflow for running Windows tests nightly r=rvl a=rvl

# Issue Number

Relates to #1283

# Overview

Runs the prebuilt Windows tests on Windows Server 2016.
This is run nightly for the master branch. The wine tests can be run on every PR.

# Comments

- There is a test failure which will disappear when we sort out #1373.
- Other tests failures were resolved by #1536.
- For testing purposes, I temporarily added the "push" build trigger. Roll back the temporary git commit before merging.
- The launcher unit tests fail on GitHub CI but pass locally on Windows. Not sure why.
- The test suites `cardano-wallet-launcher:unit` and `cardano-wallet-jormungandr:integration` are set to "continue-on-error" - meaning failures are ignored.


1530: Add latency benchmark for byron r=paweljakubas a=paweljakubas

# Issue Number

#1527 

# Overview

- [x] I have added first benchmark that works with just one scenario inside
- [x] I have extended scenario with all endpoints to be tested
- [x] I have enabled testing for both icarus and random wallets
- [x] I have added more benchmarks conditions (txs and utxos)
- [x] I have added the benchmark to nightly


# Comments

- Manual build on Buildkite: [cardano-wallet-nightly](https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds?branch=paweljakubas%2F1527%2Fadd-latency-benchmark-for-byron)


1543: Better error message when trying to generate address on icarus wallet. r=piotr-iohk a=piotr-iohk

# Issue Number

<!-- Put here a reference to the issue this PR relates to and which requirements it tackles -->


# Overview

- e3d6036
  Better error message when trying to generate address on icarus wallet.
# Comments

Was:
```
$ cardano-wallet-byron address create 3fe1acf86fefbf66b7481470ea81244eadde079d
Please enter your passphrase: *****************
It looks like something unexpected went wrong. Unfortunately I don't yet know how to handle this type of situation. Here's some information about what happened:
```

is:
```
$ cardano-wallet-byron address create 3fe1acf86fefbf66b7481470ea81244eadde079d
Please enter your passphrase: *****************
I cannot derive new unused address for this wallet type. Make sure to use Byron random wallet id.



Co-authored-by: Rodney Lorrimar <[email protected]>
Co-authored-by: Pawel Jakubas <[email protected]>
Co-authored-by: IOHK <[email protected]>
Co-authored-by: Piotr Stachyra <[email protected]>
iohk-bors bot added a commit that referenced this issue Apr 7, 2020
@iohk-bors @rvl @paweljakubas
Merge #1411 #1530 #1543
adaccff 
1411: Add GitHub workflow for running Windows tests nightly r=rvl a=rvl

# Issue Number

Relates to #1283

# Overview

Runs the prebuilt Windows tests on Windows Server 2016.
This is run nightly for the master branch. The wine tests can be run on every PR.

# Comments

- There is a test failure which will disappear when we sort out #1373.
- Other tests failures were resolved by #1536.
- For testing purposes, I temporarily added the "push" build trigger. Roll back the temporary git commit before merging.
- The launcher unit tests fail on GitHub CI but pass locally on Windows. Not sure why.
- The test suites `cardano-wallet-launcher:unit` and `cardano-wallet-jormungandr:integration` are set to "continue-on-error" - meaning failures are ignored.


1530: Add latency benchmark for byron r=paweljakubas a=paweljakubas

# Issue Number

#1527 

# Overview

- [x] I have added first benchmark that works with just one scenario inside
- [x] I have extended scenario with all endpoints to be tested
- [x] I have enabled testing for both icarus and random wallets
- [x] I have added more benchmarks conditions (txs and utxos)
- [x] I have added the benchmark to nightly


# Comments

- Manual build on Buildkite: [cardano-wallet-nightly](https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds?branch=paweljakubas%2F1527%2Fadd-latency-benchmark-for-byron)


1543: Better error message when trying to generate address on icarus wallet. r=piotr-iohk a=piotr-iohk

# Issue Number

<!-- Put here a reference to the issue this PR relates to and which requirements it tackles -->


# Overview

- e3d6036
  Better error message when trying to generate address on icarus wallet.
# Comments

Was:
```
$ cardano-wallet-byron address create 3fe1acf86fefbf66b7481470ea81244eadde079d
Please enter your passphrase: *****************
It looks like something unexpected went wrong. Unfortunately I don't yet know how to handle this type of situation. Here's some information about what happened:
```

is:
```
$ cardano-wallet-byron address create 3fe1acf86fefbf66b7481470ea81244eadde079d
Please enter your passphrase: *****************
I cannot derive new unused address for this wallet type. Make sure to use Byron random wallet id.



Co-authored-by: Rodney Lorrimar <[email protected]>
Co-authored-by: Pawel Jakubas <[email protected]>
Co-authored-by: IOHK <[email protected]>
Co-authored-by: Piotr Stachyra <[email protected]>
@KtorZ
Copy link
Member

KtorZ commented Apr 8, 2020

Still failing on Hydra:

job status
musl64.benchmarks.cardano-wallet-byron.latency.x86_64-linux ❌
musl64.benchmarks.cardano-wallet-byron.restore.x86_64-linux ❌
musl64.benchmarks.cardano-wallet-core.db.x86_64-linux ❌
musl64.benchmarks.cardano-wallet-jormungandr.latency.x86_64-linux ❌
musl64.cardano-node.x86_64-linux ❌
musl64.checks.cardano-wallet-byron.integration.x86_64-linux ❌
musl64.checks.cardano-wallet-byron.unit.x86_64-linux ❌
musl64.checks.cardano-wallet-cli.unit.x86_64-linux ❌
musl64.checks.cardano-wallet-core.unit.x86_64-linux ❌
musl64.checks.cardano-wallet-jormungandr.unit.x86_64-linux ❌
musl64.checks.cardano-wallet-launcher.unit.x86_64-linux ❌
musl64.db-converter.x86_64-linux ❌
musl64.migration-tests.x86_64-linux ❌
musl64.tests.cardano-wallet-byron.integration.x86_64-linux ❌
musl64.tests.cardano-wallet-byron.unit.x86_64-linux ❌
musl64.tests.cardano-wallet-cli.unit.x86_64-linux ❌
musl64.tests.cardano-wallet-core.unit.x86_64-linux ❌
musl64.tests.cardano-wallet-jormungandr.jormungandr-integration.x86_64-linux ❌
musl64.tests.cardano-wallet-jormungandr.unit.x86_64-linux ❌
musl64.tests.cardano-wallet-launcher.unit.x86_64-linux ❌
native.stackShell.x86_64-darwin ❌
stackShell.x86_64-darwin ❌
x86_64-w64-mingw32.stackNixRegenerate.x86_64-linux ❌

I don't quite know why do we have all the musl64.tests and musl64.benchmarks jobs in the first place though. As far as I understood, the musl64 category is for building static binaries on linux ?

And I also don't know what "stackShell" is for.

Any idea @rvl ?

iohk-bors bot added a commit that referenced this issue Apr 9, 2020
@iohk-bors @rvl
Merge #1466
5c3bc11 
1466: Fix jormungandr integration tests on Windows and Wine. r=rvl a=rvl

### Issue number

#1283 

### Overview

- [x] Fix stuck jormungandr process in launcher cli integration tests on windows.


Co-authored-by: Rodney Lorrimar <[email protected]>
@rvl rvl mentioned this issue Apr 14, 2020
Merged
6 tasks
@rvl
Copy link
Contributor

rvl commented Apr 14, 2020

The Hydra required job is passing, and this is what counts for CI status in GitHub and Bors. I put an explanation here: Hydra wiki page.

The stackShell attr is not documented where it is defined in default.nix, but is documented in the header of nix/stack-shell.nix.

The musl64 prefix is documented in the jobset file. I didn't concentrate on making these build because if the tests pass on linux then they are quite likely to pass on linux musl.

I have opened a PR to make all failing jobs either green or disabled ⇒ #1566.

iohk-bors bot added a commit that referenced this issue Apr 15, 2020
@iohk-bors @rvl
Merge #1566
eba4478 
1566: Make all hydra jobs green r=rvl a=rvl

# Issue Number

Relates to #1283

# Overview

Currently, all constituents of the [`required` job](https://hydra.iohk.io/job/Cardano/cardano-wallet/required/latest-finished#tabs-constituents) pass, but not all jobs in the main jobset pass.

This PR will either fix or disable the red jobs.

# Comments

[Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-1566#tabs-jobs)

Failing jobs which depend on SRE-83 / IntersectMBO/cardano-node#800:

- `musl64.benchmarks.cardano-wallet-byron.latency.x86_64-linux`
- `musl64.benchmarks.cardano-wallet-byron.restore.x86_64-linux`
- `musl64.cardano-node.x86_64-linux`
- `musl64.db-converter.x86_64-linux`
- `musl64.tests.cardano-wallet-byron.integration.x86_64-linux`
- `musl64.tests.cardano-wallet-jormungandr.jormungandr-integration.x86_64-linux`

Fixed jobs:

- [x] `native.stackShell.x86_64-darwin`
- [x] `musl64.benchmarks.cardano-wallet-jormungandr.latency.x86_64-linux`
- [x] `musl64.tests.cardano-wallet-jormungandr.jormungandr-integration.x86_64-linux`
- [x] `musl64.tests.cardano-wallet*`
- [x] `musl64.benchmarks.cardano-wallet*`

Disabled jobs:

- [x] `musl64.migration-tests.x86_64-linux`


Co-authored-by: Rodney Lorrimar <[email protected]>
@piotr-iohk
Copy link
Contributor Author

LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rvl rvl

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rvl @KtorZ @piotr-iohk