feat: Adding self-extracting file as an optional feature #31

prsabahrami · 2024-07-24T23:52:04Z

Motivation

Closes #4

Changes

Similar to conda-constructor, I have added a header.sh file. After the environment has been packed and archived, that archive is appended onto header.sh which later extracts it when it runs.

Packing an environment

pixi-pack pack --create-executable

Unpacking an environment

either directly run the .sh file created

sh environment.sh

You can use below flags

Unpacks an environment packed with pixi-pack

-f           no error if environment already exists
-h           print this help message and exit
-p ENV       environment prefix, defaults to $PREFIX
-i INSTALLER create the environment using the specified installer defaulting to $INSTALLER
-a           create an activation script to activate the environment

or you can directly use

pixi-pack unpack environment.sh

I will try to add better tests and error handling in the upcoming days. Meanwhile, let me know if you have any recommendations.

0xbe7a · 2024-07-25T21:42:24Z

One of the biggest use-cases for us for pixi-pack is the ability to build up archives on Linux for foreign targets, like Windows and to extract it there. Having a self-extracting archive there would be probably the most useful scenario. Your prepended shell script will only work for Unix systems, right?

Also after taking a (brief) look it appears that the extraction script is expecting a existing conda or micromamba executable on PATH. I think in these scenarios it's already an attractive option to manually extract the archive and create the environment using the included lockfiles. I think it would be very useful, if we could prepend a standalone extraction tool, that does not require any system tools to be available on the system.

prsabahrami · 2024-07-25T21:58:05Z

@0xbe7a
Yes, it is correct.
I was thinking I could add a bat self-extracting script to be used on Windows in a separate PR. Then through the passed --platform we can decide to generate sh script or bat script (or both).
Please let me know if this works for the mentioned scenario.

Also, in regards to the extraction tool used in the script, fair enough.
I was thinking to implement the logic used in unpack.rs onto the script(s) so we could get rid of conda and micromambda. Let me know what you think. I'm not sure how efficient the script will turn out.

0xbe7a · 2024-07-26T07:31:43Z

The part that involves extracting and linking the packages is highly non-trivial and probably not really feasible with a bash script. Instead, we should try using rattler as we did in pixi-pack itself. rattler supports all the platforms we target. The approach I am considering involves including an executable object with the required unpack functionality and using this when unpacking the archive to make it truly standalone.

Additionally, upon reviewing your pull request again, please note that micromamba shell activate already exists and can create the entire activation script for micromamba. For unpacking the archive using micromamba, you can use micromamba to create the environment and then generate the activation script to get the final prefix. This way, a single shell script can be used to activate into this environment, similar to what pixi-pack unpack would have produced.

prsabahrami · 2024-07-26T15:49:38Z

Sounds good.
I will include an executable using rattler onto the script. Which the script will use to properly extract the archive.

Also, in regards to the micromamba shell activate, I will try to use rattler_shell inside the executable to create the activation script so we can remove the necessitiy of conda or micromambda.

prsabahrami · 2024-08-05T14:24:22Z

Apologies for the delay. Got busy with school work.
I have added an extractor subproject which then is built and its binary is added to script.
The original commands still work but in case the -i conda is passed to the script, one can not create an activation script since I'm using micromamba or rattler to create the script. If needed, I can modify header.sh to extract extractor binary at all times but I thought if someone has conda installed on their devices, there is no need to take extra time and extract the binary.
Let me know what you think and then I can improve the verbosity and testing.
Note: The logic used in extractor is just a simpler version of what is in unpack.rs.

pavelzw · 2024-08-07T06:01:39Z

there is no need to take extra time and extract the binary

I think it would be better for reproducibility if we always unpack the same way, i.e. always use extractor

pavelzw · 2024-08-07T06:03:20Z

Could you fix the merge conflicts as well?

prsabahrami · 2024-08-07T17:49:27Z

Could you fix the merge conflicts as well?

Done.

This reverts commit 9591883, reversing changes made to 2c70aa1.

prsabahrami · 2024-08-31T17:53:22Z

Added the tests from unpack.rs to extractor.
One thing I don't like is that the dependencies of extractor are being defined separately and hence built and compiled twice during the build process. Usually, using [workspace] solves this in Cargo.toml but I couldn't make it work using [workspace]. Any suggestions? This is making the build somewhat slow.

prsabahrami · 2024-09-05T16:21:53Z

Fixed the merge conflict.
Any idea on the previous comment?

prsabahrami · 2024-09-06T01:07:59Z

As I mentioned above, the issue here is with the lockfile conflicts and incorrect dependencies between extractor and pixi-pack. I am not sure how to correctly take care of this but for now, I have updated the dependencies and made extractor compliant with rattler's latest version. But once again, if the lockfile of the main project changes, I am not sure how it will affect extractor.

extractor/Cargo.toml

src/header.sh

prsabahrami · 2024-09-23T05:30:41Z

@pavelzw I have added a header.ps1 which adds support for windows now.
Let me know what do you think!

prsabahrami · 2024-09-30T04:16:35Z

@pavelzw Any update on this?

pavelzw

This already looks very good @prsabahrami! Keep up the good work :)

Some minor comments

pavelzw · 2024-10-02T14:41:40Z

.pre-commit-config.yaml

@@ -17,6 +17,7 @@ repos:
        types: [text]
      - id: end-of-file-fixer
        name: end-of-file-fixer
+        exclude: src/header.(sh|ps1)


why do we do this?

src/header.sh

pavelzw · 2024-10-02T14:42:35Z

src/header.sh

we could add shellcheck as an additional pre-commit hook 🤔

pavelzw · 2024-10-02T14:45:32Z

src/header.sh

+-q, --quiet                 Decrease logging verbosity
+"
+# Parse command-line options
+while getopts ":hfvo:s:q" opt; do


getopts is not available everywhere. for example: docker run --rm -it ubuntu which getopts has exit code 1

wondering whether there is a nice alternative for this or whether we would need to implement this ourselves

pavelzw · 2024-10-02T14:50:43Z

src/header.sh

+      ;;
+  esac
+done
+shift $((OPTIND -1))


what does this do? (ideally also write a comment 😅)

pavelzw · 2024-10-02T16:09:35Z

src/header.ps1

+
+Unpacks an environment packed using pixi-pack
+
+-f, --force                 No error if environment already exists


see my comments in header.sh

pavelzw · 2024-10-02T16:10:30Z

src/header.ps1

+
+# Step 1: Extract the archive and pixi-pack executable, and decode them
+$scriptContent = Get-Content -Raw -Path $MyInvocation.MyCommand.Path
+$lines = $scriptContent -split "`r?`n"


is this \r\n? 😅

pavelzw · 2024-10-02T16:11:23Z

src/header.ps1

+}
+
+if (-not $headerLine -or -not $archiveLine) {
+    Write-Error "Markers __END_HEADER__ or __END_ARCHIVE__ not found."


i don't think we have a similar test in header.sh. do you think we can add it there as well?

pavelzw · 2024-10-02T16:18:08Z

src/header.ps1

+    $archivePath = "$TEMPDIR\archive.tar"
+    [System.IO.File]::WriteAllBytes($archivePath, $decodedArchive)
+} catch {
+    Write-Error "Failed to decode Base64 archive content: $_"


What does $_ do?

pavelzw · 2024-10-02T16:19:45Z

src/header.ps1

+# Finally, add the path to the archive
+$arguments += $archivePath
+
+& $pixiPackPath @arguments


Is & the powershell equivalent of exec?

Co-authored-by: Pavel Zwerschke <[email protected]>

pavelzw · 2024-10-18T09:19:29Z

tests/integration_test.rs

we added tests for reproducible pixi-packs in #43

pixi-pack/tests/integration_test.rs

Line 284 in c84abb8

async fn test_reproducible_shasum(options: Options) {

could you extend that test with create_executable as well?

Absolutely.
I'll try to respond to all of the previous messages this weekend as well.
I have been extremely busy this week.

Awesome! No worries, take your time 😄

prsabahrami added 5 commits July 23, 2024 12:46

Adding self-executable

58eda9c

Added header.sh

175ed05

Adding archive directly to shar

4ae4b35

Finished self-extracting files

e5171a0

Adding tests

cecfe4e

prsabahrami requested review from pavelzw, delsner and 0xbe7a as code owners July 24, 2024 23:52

prsabahrami and others added 2 commits July 24, 2024 21:09

Remove extra line in header.sh

447a6e2

fix ci

c2290a6

pavelzw changed the title ~~Adding self-extracting file as an optional feature~~ feat: Adding self-extracting file as an optional feature Jul 25, 2024

Removing extra print in header file

506ff30

Fixing extraction error in header.sh

9cf18a3

prsabahrami and others added 6 commits August 3, 2024 10:53

Initialized extractor

b23c1a9

Update chore.yml

4bed827

Merge branch 'Quantco:main' into main

cd71898

Update ci.yml

85d50eb

Added extractor

5990275

Fixed unpack shell script

78352cb

Merge branch 'main' into selfexec_support

34ece1e

prsabahrami and others added 2 commits August 21, 2024 00:33

Updated Cargo.toml and Cargo.lock

bab4334

Merge branch 'main' into selfexec_support

5a409cc

prsabahrami and others added 4 commits August 22, 2024 15:29

Merge branch 'main' into selfexec_support

9591883

Revert "Merge branch 'main' into selfexec_support"

aeb8873

This reverts commit 9591883, reversing changes made to 2c70aa1.

Fixed cargo.toml and added support for windows

0137e5b

Adding tests

04b3a45

prsabahrami added 2 commits September 5, 2024 12:14

Merge branch 'main' into selfexec_support

90c1014

Fixed merge conflict

62ec962

Update dependencies

b7abd97

pavelzw requested changes Sep 11, 2024

View reviewed changes

extractor/Cargo.toml Outdated Show resolved Hide resolved

src/header.sh Outdated Show resolved Hide resolved

src/header.sh Show resolved Hide resolved

pavelzw added the enhancement New feature or request label Sep 13, 2024

added download to pack

89a242d

prsabahrami marked this pull request as draft September 21, 2024 22:07

prsabahrami added 3 commits September 21, 2024 20:53

Added windows support

1bf7523

added tests

4157a22

added tests and run fmt

1109360

prsabahrami marked this pull request as ready for review September 22, 2024 15:16

prsabahrami requested a review from pavelzw September 22, 2024 15:16

prsabahrami added 2 commits September 22, 2024 11:18

remove extra function in unpack

5c06d2b

Remove extractor tests

76d41fc

pavelzw reviewed Oct 2, 2024

View reviewed changes

prsabahrami and others added 5 commits October 11, 2024 18:09

Update src/header.sh

db13b4a

Co-authored-by: Pavel Zwerschke <[email protected]>

Update src/header.sh

9d2c1ac

Co-authored-by: Pavel Zwerschke <[email protected]>

Update src/main.rs

e980ec1

Co-authored-by: Pavel Zwerschke <[email protected]>

Update tests/integration_test.rs

bc037b5

Co-authored-by: Pavel Zwerschke <[email protected]>

Update tests/integration_test.rs

cc6b42e

Co-authored-by: Pavel Zwerschke <[email protected]>

pavelzw reviewed Oct 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Adding self-extracting file as an optional feature #31

feat: Adding self-extracting file as an optional feature #31

prsabahrami commented Jul 24, 2024

0xbe7a commented Jul 25, 2024 •

edited

Loading

prsabahrami commented Jul 25, 2024

0xbe7a commented Jul 26, 2024 •

edited

Loading

prsabahrami commented Jul 26, 2024

prsabahrami commented Aug 5, 2024 •

edited

Loading

pavelzw commented Aug 7, 2024

pavelzw commented Aug 7, 2024

prsabahrami commented Aug 7, 2024

prsabahrami commented Aug 31, 2024

prsabahrami commented Sep 5, 2024

prsabahrami commented Sep 6, 2024

prsabahrami commented Sep 23, 2024

prsabahrami commented Sep 30, 2024

pavelzw left a comment

pavelzw Oct 2, 2024

pavelzw Oct 2, 2024

pavelzw Oct 2, 2024

pavelzw Oct 2, 2024

pavelzw Oct 2, 2024

pavelzw Oct 2, 2024

pavelzw Oct 2, 2024

pavelzw Oct 2, 2024

pavelzw Oct 2, 2024

pavelzw Oct 18, 2024

prsabahrami Oct 18, 2024

pavelzw Oct 18, 2024 •

edited

Loading


		Unpacks an environment packed using pixi-pack

		-f, --force No error if environment already exists

feat: Adding self-extracting file as an optional feature #31

Are you sure you want to change the base?

feat: Adding self-extracting file as an optional feature #31

Conversation

prsabahrami commented Jul 24, 2024

Motivation

Changes

Packing an environment

Unpacking an environment

0xbe7a commented Jul 25, 2024 • edited Loading

prsabahrami commented Jul 25, 2024

0xbe7a commented Jul 26, 2024 • edited Loading

prsabahrami commented Jul 26, 2024

prsabahrami commented Aug 5, 2024 • edited Loading

pavelzw commented Aug 7, 2024

pavelzw commented Aug 7, 2024

prsabahrami commented Aug 7, 2024

prsabahrami commented Aug 31, 2024

prsabahrami commented Sep 5, 2024

prsabahrami commented Sep 6, 2024

prsabahrami commented Sep 23, 2024

prsabahrami commented Sep 30, 2024

pavelzw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavelzw Oct 18, 2024 • edited Loading

Choose a reason for hiding this comment

0xbe7a commented Jul 25, 2024 •

edited

Loading

0xbe7a commented Jul 26, 2024 •

edited

Loading

prsabahrami commented Aug 5, 2024 •

edited

Loading

pavelzw Oct 18, 2024 •

edited

Loading