Upgrade packit.yaml config to have integration tests #804

r0x0d · 2022-11-14T17:09:34Z

This pull request introduces the execution of leapp integration tests as a packit job instead of the current behavior of using a GitHub Actions to trigger the tests by a comment.

Signed-off-by: Rodolfo Olivieri [email protected]

github-actions · 2022-11-14T17:09:47Z

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

review please to notify leapp developers of review request
/packit copr-build to submit a public copr build using packit

To launch regression testing public members of oamg organization can leave the following comment:

/rerun to schedule basic regression tests using this pr build and leapp-repository*master* as artifacts
/rerun 42 to schedule basic regression tests using this pr build and leapp-repository*PR42* as artifacts
/rerun-all to schedule all tests (including sst) using this pr build and leapp-repository*master* as artifacts
/rerun-all 42 to schedule all tests (including sst) using this pr build and leapp-repository*PR42* as artifacts

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please consider rerunning the CI by commenting leapp-ci build (might require several comments). If the problem persists, contact leapp-infra.

r0x0d · 2022-11-14T17:16:54Z

Maybe some targets will fail right now because of:

This is the first time I'm trying this without my demo repository
Packit will deploy the latest changes to production tomorrow, so some workflows will fail until then

r0x0d · 2022-11-16T12:18:30Z

/packit test

fernflower · 2022-11-18T12:44:48Z

.packit.yaml

@@ -54,7 +54,7 @@ jobs:
      distros: [RHEL-7.9-ZStream]
  identifier: tests-79to84
  tmt_plan: "^(?!.*upgrade_plugin)(?!.*c2r)(?!.*sap)(?!.*8to9)(?!.*morf)"
-  tf_post_install_script: "#!/bin/sh\nsudo sed -i s/.*ssh-rsa/ssh-rsa/ /root/.ssh/authorized_keys;curl https://copr.fedorainfracloud.org/coprs/g/oamg/leapp/repo/epel-7/group_oamg-leapp-epel-7.repo --output /etc/yum.repos.d/copr_build.repo; yum install -y leapp*master*"
+  tf_post_install_script: "#!/bin/sh\nsudo sed -i s/.*ssh-rsa/ssh-rsa/ /root/.ssh/authorized_keys; yum install -y leapp-repository*master*"


I remember our agreement about enabling tests with leapp-repositorymaster only now, but to have a plan for enabling testing with custom build from leapp-repository PR later.
I am just curious, how will this be done? Are we waiting for some feature to land in packit that will enable this?

Yeah, I think they implemented that already: packit/packit-service#1544

And here's the link for the docs: https://packit.dev/docs/testing-farm/#running-tests-with-builds-from-another-pull-request

I am wondering if executing /packit test <namespace>/<repo>#<pr_id> as recommended by packit will work fine if we have leapp-repositorymaster installation hardcoded as a tf_post_install_script..

Well, to be honest, that was a super recent change they made that may not be 100% working right now. In any way, I'm hoping to test that feature here instead of my demo repository, I figured that it would be nicer to have actual (real) results instead of the made up ones I have in my repo.

Since they included that command especially because we asked, I don't mind go to them and ask for an update on that if we see that the behavior is incorrect.

.packit.yaml

packit-as-a-service · 2022-11-18T14:10:40Z

Failed to load packit config file:

Cannot parse package config: ValidationError({'jobs': {0: {'trigger': ['Invalid enum member comment']}, 2: {'trigger': ['Invalid enum member comment']}, 3: {'trigger': ['Invalid enum member comment']}, 4: {'trigger': ['Invalid enum member comment']}, 5: {'trigger': ['Invalid enum member comment']}, 6: {'trigger': ['Invalid enum member comment']}, 7: {'trigger': ['Invalid enum member comment']}, 8: {'trigger': ['Invalid enum member comment']}, 9: {'trigger': ['Invalid enum member comment']}}}).

For more info, please check out the documentation or contact the Packit team.

r0x0d · 2022-11-18T14:49:28Z

/packit test oamg/leapp-repository#master

r0x0d · 2022-11-18T14:51:47Z

/packit test oamg/leapp-repository#989

r0x0d · 2022-11-18T17:12:49Z

/packit test oamg/leapp-repository#989

r0x0d · 2022-11-18T18:19:38Z

/packit test oamg/leapp-repository#988

r0x0d · 2022-11-25T12:27:45Z

/packit test oamg/leapp-repository#990

r0x0d · 2022-12-06T13:14:27Z

/packit test oamg/leapp-repository#990

r0x0d · 2022-12-08T12:38:47Z

/packit test

r0x0d · 2022-12-08T12:39:12Z

/packit test oamg/leapp-repository#990

r0x0d · 2022-12-08T18:03:28Z

/packit test oamg/leapp-repository#990

r0x0d · 2022-12-08T18:11:38Z

@fernflower the discovery of the rhel-7.9-rhui tests are failing right now because of this comparison distro == rhel-7.9, I think.

packit is passing the distro name in a slightly differently way --context distro=RHEL-7.9-rhui. If I manually change to when: distro == RHEL-7.9-rhui it works just fine.

I'm not sure if packit offers a way to pass a different context/distro name in the request. @TomasTomecek could you help us here?

If that's not possible, should we change the condition in the test then, @fernflower?

FrNecas · 2022-12-12T12:13:21Z

Hello @r0x0d , I first thought that this is a problem with the -rhui suffix, however it turns out that it is a problem with how fmf/tmt evaluate context. Currently, it is case-sensitive (this is undocumented), consider the following plan:

execute:
    how: tmt
    script: /bin/true
enabled: false
adjust:
    enabled: true
    when: distro == rhel-7.9

With tmt -c distro="rhel-7.9-rhui" run, the plan runs, whereas with tmt -c distro="RHEL-7.9-rhui" run it does not run.

I've filed an issue for fmf: teemtee/fmf#186 and we are going to dicuss this tomorrow at tmt hacking session with other tmt contributors. We may add some form of case-insensitivity to context matching. For packit, we also have packit/packit-service#1797 to provide custom context, however I do not think that passing a custom distro will/should be supported. The distro value depends on what you have in target in your config (and this is passed to Testing Farm as compose), it IMO does not make much sense to override this.

You could try using rhel-7.9-rhui as your target in the config but I am not sure if that will work, it depends on how Testing Farm matches composes (if it is case-sensitive or not). Alternatively, you could use a workaround such as when: distro == rhel-7.9 or distro == RHEL-7.9 in the meantime (before fmf is extended).

r0x0d · 2022-12-12T12:37:26Z

Hello @r0x0d , I first thought that this is a problem with the -rhui suffix, however it turns out that it is a problem with how fmf/tmt evaluate context. Currently, it is case-sensitive (this is undocumented), consider the following plan:
execute:
    how: tmt
    script: /bin/true
enabled: false
adjust:
    enabled: true
    when: distro == rhel-7.9
With tmt -c distro="rhel-7.9-rhui" run, the plan runs, whereas with tmt -c distro="RHEL-7.9-rhui" run it does not run.

I've filed an issue for fmf: teemtee/fmf#186 and we are going to dicuss this tomorrow at tmt hacking session with other tmt contributors. We may add some form of case-insensitivity to context matching. For packit, we also have packit/packit-service#1797 to provide custom context, however I do not think that passing a custom distro will/should be supported. The distro value depends on what you have in target in your config (and this is passed to Testing Farm as compose), it IMO does not make much sense to override this.

You could try using rhel-7.9-rhui as your target in the config but I am not sure if that will work, it depends on how Testing Farm matches composes (if it is case-sensitive or not). Alternatively, you could use a workaround such as when: distro == rhel-7.9 or distro == RHEL-7.9 in the meantime (before fmf is extended).

Hi, @FrNecas! Thank you for your reply and for all that explanation.

I will make sure that I follow both of those issues to keep updated with the current state of them. Also, for the workaround part, I will leave that decision to the @oamg/qe to see if it really fits in their use case or if they would like to go with some other route too. In any case, thanks again, I will wait a bit more for teemtee/fmf#186.

fernflower · 2022-12-12T12:50:31Z

@fernflower the discovery of the rhel-7.9-rhui tests are failing right now because of this comparison distro == rhel-7.9, I think.

packit is passing the distro name in a slightly differently way --context distro=RHEL-7.9-rhui. If I manually change to when: distro == RHEL-7.9-rhui it works just fine.

I'm not sure if packit offers a way to pass a different context/distro name in the request. @TomasTomecek could you help us here?

If that's not possible, should we change the condition in the test then, @fernflower?

Hi @r0x0d , if we change the condition in the test to 'distro == RHEL-7.9-rhui' then this test won't be run in case of a typical rhel-7.9 upgrade and we will need to duplicate the test, which is not really a favorable outcome because it becomes harder to maintain. It would be great to solve that on packit side imho, but of course if it doesn't work we can go back to discussing changes in the test.

r0x0d · 2022-12-12T13:04:11Z

@fernflower the discovery of the rhel-7.9-rhui tests are failing right now because of this comparison distro == rhel-7.9, I think.
packit is passing the distro name in a slightly differently way --context distro=RHEL-7.9-rhui. If I manually change to when: distro == RHEL-7.9-rhui it works just fine.
I'm not sure if packit offers a way to pass a different context/distro name in the request. @TomasTomecek could you help us here?
If that's not possible, should we change the condition in the test then, @fernflower?

Hi @r0x0d , if we change the condition in the test to 'distro == RHEL-7.9-rhui' then this test won't be run in case of a typical rhel-7.9 upgrade and we will need to duplicate the test, which is not really a favorable outcome because it becomes harder to maintain. It would be great to solve that on packit side imho, but of course if it doesn't work we can go back to discussing changes in the test.

Seems reasonable to me. Let's wait a few more to see the outcome in the linked issues, maybe the solution on their side will be quick.

r0x0d · 2022-12-13T14:16:54Z

@fernflower the discovery of the rhel-7.9-rhui tests are failing right now because of this comparison distro == rhel-7.9, I think.
packit is passing the distro name in a slightly differently way --context distro=RHEL-7.9-rhui. If I manually change to when: distro == RHEL-7.9-rhui it works just fine.
I'm not sure if packit offers a way to pass a different context/distro name in the request. @TomasTomecek could you help us here?
If that's not possible, should we change the condition in the test then, @fernflower?

Hi @r0x0d , if we change the condition in the test to 'distro == RHEL-7.9-rhui' then this test won't be run in case of a typical rhel-7.9 upgrade and we will need to duplicate the test, which is not really a favorable outcome because it becomes harder to maintain. It would be great to solve that on packit side imho, but of course if it doesn't work we can go back to discussing changes in the test.

@fernflower just a follow-up on that, me and @FrNecas were talking about this issue today, and we thought that leapp would benefit more from packit/packit-service#1797 as it will let us define custom contexts are you guys have currently in your GitHub actions testing farm workflow.

The main breakdown on all of this is that tmt will stick with being case-sensitive and will document that for future uses. So, looking for alternatives, we realized that waiting and going for packit/packit-service#1797 would be more clean and suitable for leapp right now.

r0x0d · 2023-01-13T12:09:45Z

Waiting for packit/packit-service#1797

This pull request introduces the execution of leapp integration tests as a packit job instead of the current behavior of using a GitHub Actions to trigger the tests by a comment. Signed-off-by: Rodolfo Olivieri <[email protected]>

Signed-off-by: Rodolfo Olivieri <[email protected]>

fernflower · 2023-03-15T10:09:08Z

/packit test

fernflower · 2023-03-15T12:17:52Z

Okay, looks like installation of leapp-repository via postinstall script doesn't work - the package gets removed during guest setup artifacts' installation :(
So we are back to figuring out how to build and install specific version (master by default) of leapp-repository.
I am gonna try the /packit test oamg/leapp-repository#master and then maybe reach out to packit team and reopen this discussion.

fernflower · 2023-03-15T12:18:23Z

/packit test oamg/leapp-repository#master

fernflower · 2023-03-15T12:18:38Z

/packit build oamg/leapp-repository#master

fernflower · 2023-03-15T12:31:10Z

/packit test oamg/leapp-repository#1052

.packit.yaml

Signed-off-by: Rodolfo Olivieri <[email protected]>

fernflower · 2023-03-21T14:58:15Z

/packit test oamg/leapp-repository#1052

.packit.yaml

fernflower · 2023-03-22T09:18:14Z

Short summary:
Bad news - leapp packit test runs will be complicated until packit/packit-service#1968 lands. As there is no leapp -> leapp-repository dependency, and /packit test is only capable of installing the-leapp-build-from-pr in the test environment, which is not enough for our tests. It is possible to use /packit test oamg/leapp-repository#SOME_LEAPP_REPOSITORY_PR_THAT_STILL_HAS_BUILD_ALIVE to install both leapp and leapp-repository in the test env, but it has the drawback that you have to make sure yourself that the selected leapp-repository build is indeed available as packit won't rebuild anything in case it's not.
When the issue #1968 is resolved, it will be possible to run the tests properly with /packit test oamg/leapp-repository#master.
Good news - the approach in this PR can be used for testing leapp-repository already, as there is a leapp-repository -> leapp dependency.

Signed-off-by: Rodolfo Olivieri <[email protected]>

fernflower · 2023-03-22T12:50:16Z

/packit test oamg/leapp-repository#1052

fernflower · 2023-03-22T12:50:37Z

/packit test oamg/leapp-repository#1052

stale

fernflower · 2023-03-22T13:43:39Z

/packit test oamg/leapp-repository#1052

Signed-off-by: Rodolfo Olivieri <[email protected]>

fernflower · 2023-03-24T11:19:49Z

/packit test oamg/leapp-repository#1052

fernflower · 2023-03-24T11:32:47Z

/packit test oamg/leapp-repository#1052

fernflower · 2023-04-03T08:13:28Z

Moving this to draft not to be accidentally merged until packit/packit-service#1972 is delivered.

fernflower · 2024-01-18T10:39:51Z

We have finally got all the necessary features from tft to be able to install leapp in test env for packit-triggered tests! So #827 has this PR's commits with some updates on top and should be merged soon.
Thanks a lot for doing the most important 80% of packit adoption!

r0x0d requested a review from fernflower November 14, 2022 17:09

r0x0d self-assigned this Nov 14, 2022

r0x0d force-pushed the integration-tests-with-packit branch from a77b3d5 to f603155 Compare November 14, 2022 17:11

oamg deleted a comment from packit-as-a-service bot Nov 14, 2022

fernflower reviewed Nov 18, 2022

View reviewed changes

pirat89 reviewed Nov 18, 2022

View reviewed changes

.packit.yaml Outdated Show resolved Hide resolved

r0x0d force-pushed the integration-tests-with-packit branch from 1410943 to 15c6b6f Compare November 18, 2022 14:15

r0x0d mentioned this pull request Nov 25, 2022

OAMG team onboarding: match their TF GitHub Workflow packit/packit-service#1524

Closed

5 tasks

Upgrade packit.yaml config to have integration tests

d5d998e

This pull request introduces the execution of leapp integration tests as a packit job instead of the current behavior of using a GitHub Actions to trigger the tests by a comment. Signed-off-by: Rodolfo Olivieri <[email protected]>

Change a few targets from epel-7 to epel-8

001d56b

Signed-off-by: Rodolfo Olivieri <[email protected]>

fernflower reviewed Mar 15, 2023

View reviewed changes

.packit.yaml Outdated Show resolved Hide resolved

Remove leapp-repository installation from post-install-script

4cae94c

Signed-off-by: Rodolfo Olivieri <[email protected]>

fernflower reviewed Mar 22, 2023

View reviewed changes

.packit.yaml Outdated Show resolved Hide resolved

r0x0d force-pushed the integration-tests-with-packit branch 2 times, most recently from 630ac68 to 6ff55fe Compare March 22, 2023 12:42

Update config as needed in oamg#819

e1ef75b

Signed-off-by: Rodolfo Olivieri <[email protected]>

r0x0d force-pushed the integration-tests-with-packit branch from 6ff55fe to e1ef75b Compare March 22, 2023 12:43

r0x0d requested a review from fernflower March 22, 2023 12:45

Add new environment variable to 8.8to9.2

d0893c0

Signed-off-by: Rodolfo Olivieri <[email protected]>

fernflower marked this pull request as draft April 3, 2023 08:13

fernflower marked this pull request as ready for review January 18, 2024 10:36

fernflower marked this pull request as draft January 18, 2024 10:37

fernflower closed this Jan 18, 2024

r0x0d deleted the integration-tests-with-packit branch January 19, 2024 01:35

Upgrade packit.yaml config to have integration tests #804

Upgrade packit.yaml config to have integration tests #804

Conversation

r0x0d commented Nov 14, 2022

github-actions bot commented Nov 14, 2022

Thank you for contributing to the Leapp project!

r0x0d commented Nov 14, 2022

r0x0d commented Nov 16, 2022

fernflower Nov 18, 2022 • edited Loading

Choose a reason for hiding this comment

r0x0d Nov 18, 2022

Choose a reason for hiding this comment

r0x0d Nov 18, 2022

Choose a reason for hiding this comment

fernflower Nov 18, 2022

Choose a reason for hiding this comment

r0x0d Nov 18, 2022

Choose a reason for hiding this comment

packit-as-a-service bot commented Nov 18, 2022

r0x0d commented Nov 18, 2022

r0x0d commented Nov 18, 2022

r0x0d commented Nov 18, 2022

r0x0d commented Nov 18, 2022

r0x0d commented Nov 25, 2022

r0x0d commented Dec 6, 2022

r0x0d commented Dec 8, 2022

r0x0d commented Dec 8, 2022

r0x0d commented Dec 8, 2022

r0x0d commented Dec 8, 2022

FrNecas commented Dec 12, 2022 • edited Loading

r0x0d commented Dec 12, 2022

fernflower commented Dec 12, 2022

r0x0d commented Dec 12, 2022

r0x0d commented Dec 13, 2022

r0x0d commented Jan 13, 2023

fernflower commented Mar 15, 2023

fernflower commented Mar 15, 2023

fernflower commented Mar 15, 2023

fernflower commented Mar 15, 2023

fernflower commented Mar 15, 2023

fernflower commented Mar 21, 2023

fernflower commented Mar 22, 2023 • edited Loading

fernflower commented Mar 22, 2023

fernflower commented Mar 22, 2023

fernflower commented Mar 22, 2023

fernflower commented Mar 24, 2023

fernflower commented Mar 24, 2023

fernflower commented Apr 3, 2023

fernflower commented Jan 18, 2024

fernflower Nov 18, 2022 •

edited

Loading

FrNecas commented Dec 12, 2022 •

edited

Loading

fernflower commented Mar 22, 2023 •

edited

Loading