Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add nix-shell file for building with nix #578

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ahoneybun
Copy link
Member

Once you enter shell in the firmware-open repo it will basically run though the install-deps and coreboot-sdk scripts so that you can build your firmware right after it is done. I tested with the qemu firmware test to make sure that it did work.

I'm going to test on Pop!_OS next as this was done on NixOS to be sure that I have all of the depends.

@leviport
Copy link
Member

Can flashrom be added as well? I currently have to use flashrom in a Nix shell in order to use the ch341a to flash many of the newer BIOS chips, since the flashrom in Jammy is quite old.

@ahoneybun
Copy link
Member Author

Yep I can add that! The reason for adding the shellHooks would be to run the shell command once and have everything ready to build firmware. I do note that every time that you enter the shell it would pull the latest code and recompile which can be a lot. Another reason for that would be to not have another script (we could add logic for NixOS in the current ones thought).

@ahoneybun
Copy link
Member Author

Alright I was able to build and run the QEMU example on Pop!_OS 24.04 with nix now with the latest commit. I have not tested flashing the firmware yet.

@ahoneybun ahoneybun marked this pull request as ready for review September 10, 2024 17:49
@leviport
Copy link
Member

Can you build firmware for one of our machines? I'm getting this failure when building galp5.

Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make: *** [src/security/vboot/Makefile.mk:52: build/external/vboot_reference-romstage/vboot_fw.a] Error 2
make: *** Waiting for unfinished jobs....
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make: *** [src/security/vboot/Makefile.mk:54: build/external/vboot_reference-ramstage/vboot_fw.a] Error 2
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make: *** [src/security/vboot/Makefile.mk:55: build/external/vboot_reference-postcar/vboot_fw.a] Error 2
Makefile:303: *** Missing NSS. Please install libnss3-dev if it is not installed..  Stop.
make: *** [src/security/vboot/Makefile.mk:50: build/external/vboot_reference-bootblock/vboot_fw.a] Error 2

@ahoneybun
Copy link
Member Author

I'll test the galp5 and fix that.

@ahoneybun
Copy link
Member Author

Alright I was able to build firmware for the galp5, galp7 and addw3.

@leviport leviport requested review from a team September 10, 2024 19:25
leviport
leviport previously approved these changes Sep 10, 2024
Copy link
Member

@leviport leviport left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like it's working! In the Nix shell, the ch341a scripts don't throw "unknown chip" errors on models with XM25QU256C chips anymore, and the handful of random System76 models I tried building all built successfully.

@DrymarchonShaun
Copy link

Might want to include the hidapi package as well, ./ec/scripts/ectool.sh needs it, on NixOS at least.

@ahoneybun
Copy link
Member Author

Add that @DrymarchonShaun didn't cause any issues with building QEMU so it should be fine, does it fix your issue?

@DrymarchonShaun
Copy link

Yeah, that should do it.

Copy link
Member

@jacobgkau jacobgkau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting a couple of errors when I first enter the shell:

jacob@serw13:~/Work/firmware-open$ nix-shell
Updated Git hooks.
Git LFS initialized.
Enter passphrase for key '/home/jacob/.ssh/id_rsa':
make: Entering directory '/home/jacob/Work/firmware-open/coreboot'
Welcome to the coreboot cross toolchain builder v2024-02-18_732134932b

Building toolchain using 32 thread(s).

Target architecture is i386-elf
ERROR: Missing tool: Please install 'zlib (zlib1g-dev or zlib-devel)'. (eg using your OS packaging system)
make[2]: *** [Makefile:20: build_gcc] Error 1
make[1]: *** [Makefile:36: build-i386] Error 2
make: *** [util/crossgcc/Makefile.mk:32: crossgcc-i386] Error 2
make: Leaving directory '/home/jacob/Work/firmware-open/coreboot'
make: Entering directory '/home/jacob/Work/firmware-open/coreboot'
Welcome to the coreboot cross toolchain builder v2024-02-18_732134932b

Building toolchain using 32 thread(s).

Target architecture is x86_64-elf
ERROR: Missing tool: Please install 'zlib (zlib1g-dev or zlib-devel)'. (eg using your OS packaging system)
make[2]: *** [Makefile:20: build_gcc] Error 1
make[1]: *** [Makefile:39: build-x64] Error 2
make: *** [util/crossgcc/Makefile.mk:32: crossgcc-x64] Error 2
make: Leaving directory '/home/jacob/Work/firmware-open/coreboot'
make: Entering directory '/home/jacob/Work/firmware-open/coreboot'
make: Leaving directory '/home/jacob/Work/firmware-open/coreboot'

zlib1g-dev is already installed on the host, so not sure if that means something's missing from the nix-shell file? Building firmware still works, though (even after removing that package from the host). Are you also getting those errors/are they expected?

@ahoneybun
Copy link
Member Author

I have that package on my host system and I don't see that message. I do have zlib in the shell already but if I remove it from the shell.nix file I do see those errors. Very odd.

@ahoneybun
Copy link
Member Author

@jacobgkau can you check that you have the latest changes from this branch?

@jacobgkau
Copy link
Member

@jacobgkau can you check that you have the latest changes from this branch?

I'm on 6ccff2f, which is the last commit on this branch.

I do have zlib in the shell already but if I remove it from the shell.nix file I do see those errors.

Huh, I do see it in nix.shell. How are you checking that you have it in the shell?

@ahoneybun
Copy link
Member Author

ahoneybun commented Oct 9, 2024

I don't see that message if I have it in the shell.nix file which makes it seem that it is available in the shell.

I believe you can check like this:

[nix-shell:~/Projects/system76/firmware-open]$ ls /nix/store | grep zlib
0wkcfpw422yd52bvcf2lzp1vk31iaidi-zlib-1.3.1.drv
2ksxw3mkxlxr7l00fqvfgf3v1jnzzm7m-zlib-1.3.1.drv
3ds1n1s3s82cxbfd9wisqfm0cn7zqln6-zlib-1.3.1.drv
4rascx24j2597gjkcxvmwfhjld9x4pl6-zlib-1.3.1.drv
5cqxgjddsf882mhx6ghjn5c7j6jn6pm1-zlib-1.3.1.drv
6w6qnjzlibn0dcj85iv477ipqifw0kph-pytz-2024.1.tar.gz.drv
7hxzan6h0rcs79kq75j0a41yhxai0xm9-zlib-1.3.1.drv
83yyrk14n4bfavlravz91mycv8rlccil-zlib-ng-2.1.6.drv
8vmfhxc5w905jvkf4wfhb79jw8qkpq2i-zlib-1.3.1.drv
971g0fka30djkcww2ddpx7q66dr86c1q-zlib-1.3.1
9cda1wfcdsmx97gga5iflwqwf6vcirvf-zlib-1.3.1.drv
bk66m98652ywmgqbdc2wmjdmzlib69nc-foreign-types-shared-0.1.1.drv
dwrdqqi0qfl45wpwdavrasb3absqsrm3-zlib-ng-2.1.6
fxz846h7jams5rbvqi6ba55qwqvw7b20-zlib-1.3.1.drv
gyks6vvl7x0gq214ldjhi3w4rg37nh8i-zlib-1.3.1.tar.gz.drv
id7m774ya59gznvfkcn72522qmfncs4g-zlib-ng-2.1.6-bin
iicmmyl8n2sh8vwcrzyhqpmyvqsn5cwf-zlib-1.3.1.drv
j6701mn0rhnqjdmnv266kpacajm74rl7-zlib-1.3.1.drv
lv6nackqis28gg7l2ic43f6nk52hb39g-zlib-1.3.1
qj9byzfvh7dd61kk0aglj7cwqj1xqg6l-zlib-1.3.1-dev
qpn316xqda1pw401px37qmj9sv5gg6s2-zlib-1.3.1.drv
r1ywn1kcr07fvmj0lzdddhfmg4im8dvw-zlib-1.3.1.drv
rqs1zrcncqz3966khjndg1183cpdnqxs-zlib-1.3.1
w9f41v8cri38jqryfhdnpg1cf5yzijmw-zlib-ng-2.1.6-dev
xpbgwvh08302vfv6szbk90gvxk3yz72f-zlib-1.3.1.drv
ykh8zgm27qqhfxcq77jlap5ahrplrk54-zlib-1.3

@ahoneybun
Copy link
Member Author

I did a nix-collect-garbage -d then entered the shell again with nix-shell and it is still don't see that error.

@jacobgkau
Copy link
Member

Hmm. This is what I get when I run that command within the shell:

[nix-shell:~/Work/firmware-open]$ ls /nix/store | grep zlib
5cqxgjddsf882mhx6ghjn5c7j6jn6pm1-zlib-1.3.1.drv
6w6qnjzlibn0dcj85iv477ipqifw0kph-pytz-2024.1.tar.gz.drv
fw1jqi1ia1sinknci20m95mybpidrbhc-zlib-1.3.1-dev
gyks6vvl7x0gq214ldjhi3w4rg37nh8i-zlib-1.3.1.tar.gz.drv
j6701mn0rhnqjdmnv266kpacajm74rl7-zlib-1.3.1.drv
phnpfqk1j35nil4hqgaslqm9a1q2gffy-zlib-1.3.1
v2ny69wp81ch6k4bxmp4lnhh77r0n4h1-zlib-1.3.1
xpbgwvh08302vfv6szbk90gvxk3yz72f-zlib-1.3.1.drv

@ahoneybun
Copy link
Member Author

Some of my output might be from trying out the zlib-ng library instead which did not work and it needs zlib directly. After that did the error about zlib still appear?

@jacobgkau
Copy link
Member

jacobgkau commented Oct 9, 2024

Yes, that output is immediately following the error messages.

jacob@serw13:~/Work/firmware-open$ nix-shell
Updated Git hooks.
Git LFS initialized.
Enter passphrase for key '/home/jacob/.ssh/id_rsa':
make: Entering directory '/home/jacob/Work/firmware-open/coreboot'
Welcome to the coreboot cross toolchain builder v2024-02-18_732134932b

Building toolchain using 32 thread(s).

Target architecture is i386-elf
ERROR: Missing tool: Please install 'zlib (zlib1g-dev or zlib-devel)'. (eg using your OS packaging system)
make[2]: *** [Makefile:20: build_gcc] Error 1
make[1]: *** [Makefile:36: build-i386] Error 2
make: *** [util/crossgcc/Makefile.mk:32: crossgcc-i386] Error 2
make: Leaving directory '/home/jacob/Work/firmware-open/coreboot'
make: Entering directory '/home/jacob/Work/firmware-open/coreboot'
Welcome to the coreboot cross toolchain builder v2024-02-18_732134932b

Building toolchain using 32 thread(s).

Target architecture is x86_64-elf
ERROR: Missing tool: Please install 'zlib (zlib1g-dev or zlib-devel)'. (eg using your OS packaging system)
make[2]: *** [Makefile:20: build_gcc] Error 1
make[1]: *** [Makefile:39: build-x64] Error 2
make: *** [util/crossgcc/Makefile.mk:32: crossgcc-x64] Error 2
make: Leaving directory '/home/jacob/Work/firmware-open/coreboot'
make: Entering directory '/home/jacob/Work/firmware-open/coreboot'
make: Leaving directory '/home/jacob/Work/firmware-open/coreboot'

[nix-shell:~/Work/firmware-open]$ ls /nix/store | grep zlib
5cqxgjddsf882mhx6ghjn5c7j6jn6pm1-zlib-1.3.1.drv
6w6qnjzlibn0dcj85iv477ipqifw0kph-pytz-2024.1.tar.gz.drv
fw1jqi1ia1sinknci20m95mybpidrbhc-zlib-1.3.1-dev
gyks6vvl7x0gq214ldjhi3w4rg37nh8i-zlib-1.3.1.tar.gz.drv
j6701mn0rhnqjdmnv266kpacajm74rl7-zlib-1.3.1.drv
phnpfqk1j35nil4hqgaslqm9a1q2gffy-zlib-1.3.1
v2ny69wp81ch6k4bxmp4lnhh77r0n4h1-zlib-1.3.1
xpbgwvh08302vfv6szbk90gvxk3yz72f-zlib-1.3.1.drv

[nix-shell:~/Work/firmware-open]$ exit
exit

I tried adding zlib-ng to the shell.nix file immediately underneath zlib in the buildInputs section, but after doing that and running nix-shell again, I still get the errors and only get a few additional lines within /nix/store:

[nix-shell:~/Work/firmware-open]$ ls /nix/store | grep zlib
3imc56w0r4qwh5zlydjk020bvhdvfqfj-zlib-ng-2.2.1.drv
5cqxgjddsf882mhx6ghjn5c7j6jn6pm1-zlib-1.3.1.drv
6mbn7wv262sqgnix63qk9zcpzrafvd33-zlib-ng-2.2.1-bin
6w6qnjzlibn0dcj85iv477ipqifw0kph-pytz-2024.1.tar.gz.drv
fw1jqi1ia1sinknci20m95mybpidrbhc-zlib-1.3.1-dev
gyks6vvl7x0gq214ldjhi3w4rg37nh8i-zlib-1.3.1.tar.gz.drv
j6701mn0rhnqjdmnv266kpacajm74rl7-zlib-1.3.1.drv
nd1v3613ha43pr5pz751hs356is68a1s-zlib-ng-2.2.1
phnpfqk1j35nil4hqgaslqm9a1q2gffy-zlib-1.3.1
q24n0ar0z1b73s30fz5z88v4dsspq7kz-zlib-ng-2.2.1-dev
v2ny69wp81ch6k4bxmp4lnhh77r0n4h1-zlib-1.3.1
xpbgwvh08302vfv6szbk90gvxk3yz72f-zlib-1.3.1.drv

Testing on a fresh lab machine, I actually don't get the errors seen above on my work machine, but I do still get an error building MPFR v4.2.1.

@ahoneybun
Copy link
Member Author

The zlib-ng won't fix it at least it did not for me, zlib itself is correct. How did you install nix? Did you use the Multi-user installation command here?:

https://nixos.org/download/

@jacobgkau
Copy link
Member

I used the single-user install on that lab machine, and it looks like I used the multi-user install on my work machine.

@ahoneybun
Copy link
Member Author

Now I do see this error:

Building MPFR v4.2.1 for host ... failed. Check 'build-MPFR/build.log'.

so that is indeed odd that it is just happening now. I've been told to go the flake route but the official nix installer does not enable that and you would need to edit your config file for nix for it.

@matdibu
Copy link

matdibu commented Oct 21, 2024

if you want to simplify things a bit, you could simply use the coreboot-toolchain packaged in nixpkgs https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/tools/misc/coreboot-toolchain/default.nix

@ahoneybun
Copy link
Member Author

if you want to simplify things a bit, you could simply use the coreboot-toolchain packaged in nixpkgs https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/tools/misc/coreboot-toolchain/default.nix

Ah that is neat! I guess I can't run it like this?:

nix-shell -p coreboot-toolchain

@matdibu
Copy link

matdibu commented Oct 21, 2024

try

nix-shell -p coreboot-toolchain.i386

@matdibu
Copy link

matdibu commented Oct 21, 2024

this is what I use for one of my machines (sadly, the only machine in my possession which supports coreboot):
https://codeberg.org/mateidibu/coreboot-flake/src/commit/b72d8c1e21025646cc2ee9a65ef4d56ed86fa157/coreboot/default.nix

@ahoneybun
Copy link
Member Author

try

nix-shell -p coreboot-toolchain.i386

ahhh it's odd that it is .x64 instead of the normal x86_64.

@matdibu
Copy link

matdibu commented Oct 21, 2024

you're building x64 coreboot? I saw in the release notes of 24.05 that

Mark 64-bit support as stable

A significant amount of work has gone into fully supporting 64-bit coreboot builds. There are still additional pieces that are happening, but with SMM holding page tables itself, we can consider SMM support stable and safe enough for general use.

but I wasn't sure if it applied to all platforms and configs

@ahoneybun
Copy link
Member Author

you're building x64 coreboot? I saw in the release notes of 24.05 that

Mark 64-bit support as stable
A significant amount of work has gone into fully supporting 64-bit coreboot builds. There are still additional pieces that are happening, but with SMM holding page tables itself, we can consider SMM support stable and safe enough for general use.

but I wasn't sure if it applied to all platforms and configs

I would just think since the platform/device is x86 it should be building for that no? I am seeing this error though:

= note: rust-lld: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory

When I run the build script for qemu.

@matdibu
Copy link

matdibu commented Oct 21, 2024

64bit support for coreboot was added relatively recently
while, of course, in the boot process you would eventually transition to 64bit system, the different stages were still 32bit

I'm not a developer of coreboot, and I cannot speak on the behalf of either the project or the developers

@matdibu
Copy link

matdibu commented Oct 21, 2024

by the way, this is another interesting project that could serve as a source of inspiration https://codeberg.org/amjoseph/ownerboot

@ahoneybun
Copy link
Member Author

The goal of this PR is to just run nix-shell in the repo and have everything ready to build an image to flash to a system76 coreboot system. The main issue is that we're having mixed results on certain systems, I've been told to use flakes but I'm not sure about doing that.

@matdibu
Copy link

matdibu commented Oct 21, 2024

the only practical difference between flake and non-flake is that flakes provide a "standard" way of managing inputs
you could achieve the same results by using other input-managing tools, like https://github.com/nmattia/niv

regardless, if you want to read an interesting article on the subject: https://jade.fyi/blog/flakes-arent-real/

@matdibu
Copy link

matdibu commented Oct 21, 2024

ultimately, what would improve user experience is a pinned version of nixpkgs, which can be achieved in multiple ways (npins niv flakes)
otherwise, you'll just get the (somewhat) latest commit of nixpkgs from the branch you use as input (or the user's revision of nixpkgs)

@matdibu
Copy link

matdibu commented Oct 21, 2024

I like the idea of building it with a flake because it's a common way of managing dependencies, and builds are done in an offline sandbox (apart from fetchPhase which can access the internet). I even set strictDeps = true; in my build just to make it as "clean" as I can

@matdibu
Copy link

matdibu commented Oct 21, 2024

but remember that if you fully commit to the flake route you'll very likely run into the same issues that I did

namely, only fetchPhase can download files; buildPhase cannot

normally this is fine, but coreboot's build system fetches some of its submodules during build time, and performs additional git commands

this is why I had to do this in postPatch:

  1. get the submodules that are explicitly not pulled in a normal repo clone:
    cp -r ${inputs.intel-fsp}/*       ./3rdparty/fsp/
    cp -r ${inputs.intel-microcode}/* ./3rdparty/intel-microcode/
  1. get edk2 in place before starting the build
    mkdir -p payloads/external/edk2/workspace

    cp -r ${inputs.edk2-platforms} payloads/external/edk2/workspace/edk2-platforms
    chmod -R +w                    payloads/external/edk2/workspace/edk2-platforms

    cp -r ${inputs.edk2-mrchromebox} payloads/external/edk2/workspace/MrChromebox
    chmod -R +w                      payloads/external/edk2/workspace/MrChromebox
  1. this ugly patch to just nukes all the git commands done for edk2
    https://codeberg.org/mateidibu/coreboot-flake/src/branch/main/coreboot/0001-edk2-assume-both-EDK2_PATH-and-EDK2_PLATFORMS_PATH-a.patch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants