Update Haskell.nix #888

rvl · 2019-10-23T07:15:19Z

Relates to #703.

Overview

Updates the Haskell.nix version so that we can build for windows again.
Keeps more or less the same structure as before - i.e. no IFD, no niv. I don't want too much change at once.
Does however update nixpkgs from iohk fork of 18.09 → upstream 19.03.
Provides a big bundle of scripts and tests that can be run on Windows.

Hydra jobset
Hercules jobset
Link to download Windows testing bundle for PR #888
Link to download Windows testing bundle (master branch)

Comments

@angerman Please help

Fix failure for missing hsc2hs
Fix a previous mistake that affects cross builds (failing to cross-compile bash for some reason)
Worked around an evaluation error because weeder and ghci are no longer available from pkgs.haskellPackages.
Evaluation is taking a rather long time (5+ hours) in Hydra.
Some of the Buildkite pipeline steps don't work with the updated Haskell.nix.
Tests for windows are built not not executed. The tests should be run under wine.

angerman · 2019-10-23T07:43:10Z

I think you want to use https://github.com/input-output-hk/nixpkgs/archive/a8f81dc037a5977414a356dd068f2621b3c89b60.tar.gz for 19.03; especially as you want windows to work :-(
And

                (hackage: {
                    packages = {
                        "hsc2hs" = (((hackage.hsc2hs)."0.68.4").revisions).default;
                    };
                })

in the pkg-def-extras in the default.nix to get the hsc2hs issue resolved.

rvl · 2019-10-24T09:13:21Z

Thanks @angerman. I think there is just one more thing that needs to be done for this. That is, get it evaluating on Hydra. Currently it's taking a long time to evaluate and not succeeding. Do you know what could be the cause?

angerman

While my choice of ci.nix attribute naming is slightly different, that's mostly due to personal taste.

The additional injection of overlays and config, seems slightly complicated but I believe this to be a defect of haskell.nix; not sure I know of a better UI yet though.

KtorZ · 2019-10-28T17:53:57Z

bors r+

@angerman

888: Update Haskell.nix r=KtorZ a=rvl Relates to #703. # Overview - [x] Updates the Haskell.nix version so that we can build for windows again. - [x] Keeps more or less the same structure as before - i.e. no IFD, no niv. I don't want too much change at once. - [x] Does however update nixpkgs from iohk fork of 18.09 → upstream 19.03. - [x] Provides a big bundle of scripts and tests that can be run on Windows. [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-888) [Hercules jobset](https://hercules-ci.com/github/input-output-hk/cardano-wallet) [Link to download Windows testing bundle for PR #888](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-888/cardano-wallet-jormungandr-win64/latest) [Link to download Windows testing bundle (master branch)](https://hydra.iohk.io/job/Cardano/cardano-wallet/cardano-wallet-jormungandr-win64/latest) # Comments @angerman Please help - [x] Fix failure for missing `hsc2hs` - [x] Fix a previous mistake that affects cross builds (failing to cross-compile bash for some reason) - [x] Worked around an [evaluation error](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-888#tabs-errors) because weeder and ghci are no longer available from `pkgs.haskellPackages`. - [ ] Evaluation is taking a rather long time (5+ hours) in Hydra. - [ ] Some of the Buildkite pipeline steps don't work with the updated Haskell.nix. # Next PRs - [ ] Fix failures in sqlite tests on Windows. - [ ] Fix failure in launch command on Windows. - [ ] Fix the UTF-8 text issues on Windows ⇒ #914. - [ ] Add code which tells the windows command prompt to interpret ANSI color codes. - [ ] Identify other test failures and fix. Co-authored-by: Rodney Lorrimar <[email protected]>

iohk-bors · 2019-10-28T18:54:27Z

Build failed

buildkite/cardano-wallet

KtorZ · 2019-10-29T12:08:28Z

bors r+

@angerman

888: Update Haskell.nix r=KtorZ a=rvl Relates to #703. # Overview - [x] Updates the Haskell.nix version so that we can build for windows again. - [x] Keeps more or less the same structure as before - i.e. no IFD, no niv. I don't want too much change at once. - [x] Does however update nixpkgs from iohk fork of 18.09 → upstream 19.03. - [x] Provides a big bundle of scripts and tests that can be run on Windows. [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-888) [Hercules jobset](https://hercules-ci.com/github/input-output-hk/cardano-wallet) [Link to download Windows testing bundle for PR #888](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-888/cardano-wallet-jormungandr-win64/latest) [Link to download Windows testing bundle (master branch)](https://hydra.iohk.io/job/Cardano/cardano-wallet/cardano-wallet-jormungandr-win64/latest) # Comments @angerman Please help - [x] Fix failure for missing `hsc2hs` - [x] Fix a previous mistake that affects cross builds (failing to cross-compile bash for some reason) - [x] Worked around an [evaluation error](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-888#tabs-errors) because weeder and ghci are no longer available from `pkgs.haskellPackages`. - [ ] Evaluation is taking a rather long time (5+ hours) in Hydra. - [ ] Some of the Buildkite pipeline steps don't work with the updated Haskell.nix. # Next PRs - [ ] Fix failures in sqlite tests on Windows. - [ ] Fix failure in launch command on Windows. - [ ] Fix the UTF-8 text issues on Windows ⇒ #914. - [ ] Add code which tells the windows command prompt to interpret ANSI color codes. - [ ] Identify other test failures and fix. Co-authored-by: Rodney Lorrimar <[email protected]>

iohk-bors · 2019-10-29T13:10:06Z

Timed out

Fixes stack stack/nix build errors resulting from Haskell.nix update.

KtorZ · 2019-11-04T18:31:13Z

bors r+

@angerman

888: Update Haskell.nix r=KtorZ a=rvl Relates to #703. # Overview - [x] Updates the Haskell.nix version so that we can build for windows again. - [x] Keeps more or less the same structure as before - i.e. no IFD, no niv. I don't want too much change at once. - [x] Does however update nixpkgs from iohk fork of 18.09 → upstream 19.03. - [x] Provides a big bundle of scripts and tests that can be run on Windows. [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-888) [Hercules jobset](https://hercules-ci.com/github/input-output-hk/cardano-wallet) [Link to download Windows testing bundle for PR #888](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-888/cardano-wallet-jormungandr-win64/latest) [Link to download Windows testing bundle (master branch)](https://hydra.iohk.io/job/Cardano/cardano-wallet/cardano-wallet-jormungandr-win64/latest) # Comments @angerman Please help - [x] Fix failure for missing `hsc2hs` - [x] Fix a previous mistake that affects cross builds (failing to cross-compile bash for some reason) - [x] Worked around an [evaluation error](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-888#tabs-errors) because weeder and ghci are no longer available from `pkgs.haskellPackages`. - [ ] Evaluation is taking a rather long time (5+ hours) in Hydra. - [ ] Some of the Buildkite pipeline steps don't work with the updated Haskell.nix. - [ ] Tests for windows are built not not executed. The tests should be run under wine. # Next PRs - [ ] Fix failures in sqlite tests on Windows ⇒ #918. - [ ] Fix failure in launch command on Windows. - [ ] Fix the UTF-8 text issues on Windows ⇒ #914. - [ ] Add code which tells the windows command prompt to interpret ANSI color codes. - [ ] Identify other test failures and fix. 952: review golden tests for legacy transaction and generate witnesses using jcli r=KtorZ a=KtorZ # Issue Number  N/A # Overview  - [x] I have reviewed the legacy transaction golden tests, now fully using jcli to also generate the witnesses (this feature was missing when the golden tests were initially implemented). - [x] I have updated the comment on private key to reflect the real format expected by jcli # Comments  A bit disappointing to some extend since nothing really has changed.. It seems that we were already providing the right output :(  Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: Michael Bishop <[email protected]> Co-authored-by: KtorZ <[email protected]>

iohk-bors · 2019-11-04T19:17:30Z

Build failed (retrying...)

buildkite/cardano-wallet

@angerman

888: Update Haskell.nix r=KtorZ a=rvl Relates to #703. # Overview - [x] Updates the Haskell.nix version so that we can build for windows again. - [x] Keeps more or less the same structure as before - i.e. no IFD, no niv. I don't want too much change at once. - [x] Does however update nixpkgs from iohk fork of 18.09 → upstream 19.03. - [x] Provides a big bundle of scripts and tests that can be run on Windows. [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-888) [Hercules jobset](https://hercules-ci.com/github/input-output-hk/cardano-wallet) [Link to download Windows testing bundle for PR #888](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-888/cardano-wallet-jormungandr-win64/latest) [Link to download Windows testing bundle (master branch)](https://hydra.iohk.io/job/Cardano/cardano-wallet/cardano-wallet-jormungandr-win64/latest) # Comments @angerman Please help - [x] Fix failure for missing `hsc2hs` - [x] Fix a previous mistake that affects cross builds (failing to cross-compile bash for some reason) - [x] Worked around an [evaluation error](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-888#tabs-errors) because weeder and ghci are no longer available from `pkgs.haskellPackages`. - [ ] Evaluation is taking a rather long time (5+ hours) in Hydra. - [ ] Some of the Buildkite pipeline steps don't work with the updated Haskell.nix. - [ ] Tests for windows are built not not executed. The tests should be run under wine. # Next PRs - [ ] Fix failures in sqlite tests on Windows ⇒ #918. - [ ] Fix failure in launch command on Windows. - [ ] Fix the UTF-8 text issues on Windows ⇒ #914. - [ ] Add code which tells the windows command prompt to interpret ANSI color codes. - [ ] Identify other test failures and fix. Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: Michael Bishop <[email protected]>

iohk-bors · 2019-11-04T20:44:52Z

Build succeeded

buildkite/cardano-wallet

rvl · 2019-11-04T23:42:06Z

@cleverca22 @disassembler @KtorZ this commit d27da42 is a hack not a proper fix. It makes CI green but the problems are still there and will reappear. The underlying problems shouldn't be ignored and need to be fixed.

918: Fix Sqlite DB on Windows r=rvl a=rvl Relates to #703. Based on the branch of #888. # Overview The wallet failed with: NUL: openFile: does not exist (No such file or directory) This is due to runMigrationSilent which uses the silently library: http://hackage.haskell.org/package/silently-1.2.5.1/docs/src/System.IO.Silently.html#line-27 To fix, we swap `runMigrationQuiet` for `runMigrationSilent` on Windows. Co-authored-by: KtorZ <[email protected]> Co-authored-by: iohk-bors[bot] <43231472+iohk-bors[bot]@users.noreply.github.com> Co-authored-by: Rodney Lorrimar <[email protected]>

918: Fix Sqlite DB on Windows r=rvl a=rvl Relates to #703. Based on the branch of #888. # Overview The wallet failed with: NUL: openFile: does not exist (No such file or directory) This is due to runMigrationSilent which uses the silently library: http://hackage.haskell.org/package/silently-1.2.5.1/docs/src/System.IO.Silently.html#line-27 To fix, we swap `runMigrationQuiet` for `runMigrationSilent` on Windows. Co-authored-by: Rodney Lorrimar <[email protected]>

964: Add macOS tarball download to Hydra r=KtorZ a=rvl Relates to #704. # Overview - Adds a downloadable macOS tarball to the Hydra jobset. - Fixes closure size/caching issues which were hacked around in #888. (Halves the closure size of the nix-shell, ensures that Buildkite steps are also built in Hydra) - Adds placeholder for a musl libc fully static build. Tools team can help complete it. # Comments - I have tested the binaries on a MacBook Pro with `cardano-wallet-jormungandr launch --genesis-block ...` [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-964#tabs-jobs) [macOS tarball](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-964/cardano-wallet-jormungandr-macos64/latest) Co-authored-by: Rodney Lorrimar <[email protected]>

964: Add macOS tarball download to Hydra r=rvl a=rvl Relates to #704. # Overview - Adds a downloadable macOS tarball to the Hydra jobset. - Fixes closure size/caching issues which were hacked around in #888. (Halves the closure size of the nix-shell, ensures that Buildkite steps are also built in Hydra) - Adds placeholder for a musl libc fully static build. Tools team can help complete it. # Comments - I have tested the binaries on a MacBook Pro with `cardano-wallet-jormungandr launch --genesis-block ...` [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-964#tabs-jobs) [macOS tarball](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-964/cardano-wallet-jormungandr-macos64/latest) Co-authored-by: Rodney Lorrimar <[email protected]>

rvl requested a review from angerman October 23, 2019 07:15

rvl self-assigned this Oct 23, 2019

rvl force-pushed the rvl/703/update-haskell-nix branch 4 times, most recently from 70617f4 to ae5733c Compare October 24, 2019 09:13

rvl force-pushed the rvl/703/update-haskell-nix branch from 0da30d3 to 7ccc983 Compare October 25, 2019 02:44

rvl changed the title ~~wip: Update Haskell.nix~~ Update Haskell.nix Oct 25, 2019

rvl marked this pull request as ready for review October 25, 2019 02:45

angerman approved these changes Oct 25, 2019

View reviewed changes

This was referenced Oct 25, 2019

Fix text locale encoding issues on windows #914

Merged

Fix Sqlite DB on Windows #918

Merged

rvl force-pushed the rvl/703/update-haskell-nix branch from 7ccc983 to 717f1b5 Compare October 28, 2019 05:05

KtorZ force-pushed the rvl/703/update-haskell-nix branch from 717f1b5 to 317a9cc Compare October 28, 2019 17:53

KtorZ force-pushed the rvl/703/update-haskell-nix branch from 317a9cc to e38b407 Compare October 29, 2019 12:08

rvl mentioned this pull request Nov 1, 2019

Add allExes (like all) input-output-hk/haskell.nix#284

Closed

rvl force-pushed the rvl/703/update-haskell-nix branch 2 times, most recently from 1679e94 to a5fecf0 Compare November 4, 2019 00:49

cleverca22 force-pushed the rvl/703/update-haskell-nix branch 2 times, most recently from 4534ec2 to a591209 Compare November 4, 2019 15:17

disassembler force-pushed the rvl/703/update-haskell-nix branch from a591209 to 3c5af93 Compare November 4, 2019 15:35

Switch to Haskell.nix overlays release

2a4a064

rvl and others added 7 commits November 4, 2019 10:36

nix: Use upstreamed collectComponents

08def31

nix: Add cross builds to Hercules-CI

153be4d

nix: Cross build fixes

43ad716

nix: Work around evaluation error in shell

90306e1

nix: Include tests and updated .BAT files in the Windows bundle

8a2c104

Update to PR input-output-hk/haskell.nix#289 branch

6f425fa

Fixes stack stack/nix build errors resulting from Haskell.nix update.

improve CI times by pre-building stylish-haskell on hydra

d27da42

disassembler force-pushed the rvl/703/update-haskell-nix branch from 3c5af93 to d27da42 Compare November 4, 2019 15:36

iohk-bors bot merged commit d27da42 into master Nov 4, 2019

KtorZ mentioned this pull request Nov 5, 2019

Wallet Backend builds and tests on Windows #703

Closed

27 tasks

rvl deleted the rvl/703/update-haskell-nix branch November 5, 2019 05:50

rvl mentioned this pull request Nov 6, 2019

Add macOS tarball download to Hydra #964

Merged

KtorZ added this to the Usability & Compatibility milestone Nov 6, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Haskell.nix #888

Update Haskell.nix #888

rvl commented Oct 23, 2019 •

edited

Loading

angerman commented Oct 23, 2019

rvl commented Oct 24, 2019

angerman left a comment

KtorZ commented Oct 28, 2019

iohk-bors bot commented Oct 28, 2019

KtorZ commented Oct 29, 2019

iohk-bors bot commented Oct 29, 2019

KtorZ commented Nov 4, 2019

iohk-bors bot commented Nov 4, 2019

iohk-bors bot commented Nov 4, 2019

rvl commented Nov 4, 2019

Update Haskell.nix #888

Update Haskell.nix #888

Conversation

rvl commented Oct 23, 2019 • edited Loading

Overview

Comments

angerman commented Oct 23, 2019

rvl commented Oct 24, 2019

angerman left a comment

Choose a reason for hiding this comment

KtorZ commented Oct 28, 2019

iohk-bors bot commented Oct 28, 2019

Build failed

KtorZ commented Oct 29, 2019

iohk-bors bot commented Oct 29, 2019

Timed out

KtorZ commented Nov 4, 2019

iohk-bors bot commented Nov 4, 2019

Build failed (retrying...)

iohk-bors bot commented Nov 4, 2019

Build succeeded

rvl commented Nov 4, 2019

rvl commented Oct 23, 2019 •

edited

Loading