Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Homogenize how we handle mirrored repositories on salt states #860

Open
srbarrios opened this issue Feb 15, 2021 · 12 comments
Open

Homogenize how we handle mirrored repositories on salt states #860

srbarrios opened this issue Feb 15, 2021 · 12 comments

Comments

@srbarrios
Copy link
Member

First of all, I would like that we agreed on something:

The repository URIs provided when we don't have set up the "mirror" grain, must be identical to the URIs provided when we have a "mirror" grain; differentiating them only by the domain name where sumaform request this content.

If we agree on that assumption, I would like to refactor some Jinja if statements, homogenizing the way we handle those mirrored repositories.

In my opinion, it will be less error-prone if we use this format:

http://{{ grains.get("mirror") | default("download.opensuse.org", true) }}/repositories/systemsmanagement:/Uyuni:/Master:/openSUSE_Leap_15-Uyuni-Client-Tools/openSUSE_Leap_15.0/

Instead of this other format, also found in the salt states:

  {% if grains.get('mirror') %}
    {% if 'beta' in grains.get('product_version') | default('', true) %}
    - baseurl: http://{{ grains.get("mirror") }}/repo/$RCE/SLES11-SP4-SUSE-Manager-Tools-Beta/sle-11-x86_64/
    {% else %}
    - baseurl: http://{{ grains.get("mirror") }}/repo/$RCE/SLES11-SP4-SUSE-Manager-Tools/sle-11-x86_64/
    {% endif %}
    {% else %}
    {% if 'beta' in grains.get('product_version') | default('', true) %}
    - baseurl: http://euklid.nue.suse.com/mirror/SuSE/build-ncc.suse.de/SUSE/Updates/SLE-SERVER/11-SP4-CLIENT-TOOLS-BETA/x86_64/update/
    {% else %}
    - baseurl: http://euklid.nue.suse.com/mirror/SuSE/build-ncc.suse.de/SUSE/Updates/SLE-SERVER/11-SP4-CLIENT-TOOLS/x86_64/update/
    {% endif %}
    {% endif %}

So, ideally, the above snipper could be refactored as:

{% if 'beta' in grains.get('product_version') | default('', true) %}
- baseurl: http://{{ grains.get("mirror") | default("euklid.nue.suse.com", true) }}/mirror/SuSE/build-ncc.suse.de/SUSE/Updates/SLE-SERVER/11-SP4-CLIENT-TOOLS-BETA/x86_64/update/
{% else %}
- baseurl: http://{{ grains.get("mirror") | default("euklid.nue.suse.com", true) }}/mirror/SuSE/build-ncc.suse.de/SUSE/Updates/SLE-SERVER/11-SP4-CLIENT-TOOLS/x86_64/update/
{% endif %}

Having in our minima mirror, on the http section, these lines:

http:
  - url: http://euklid.nue.suse.com/mirror/SuSE/build-ncc.suse.de/SUSE/Updates/SLE-SERVER/11-SP4-CLIENT-TOOLS-BETA
    archs: [x86_64]
  - url: http://euklid.nue.suse.com/mirror/SuSE/build-ncc.suse.de/SUSE/Updates/SLE-SERVER/11-SP4-CLIENT-TOOLS
    archs: [x86_64]

Note that in this example we are using a mirror as source origin, but please just don't pay attention to that detail. The general idea is that we have in our minima mirror exactly what we have as origin source, it doesn't matter for this refactor if it comes from OBS, IBS, or any other source.

@srbarrios
Copy link
Member Author

@juliogonzalez @rjmateus that make sense for you guys? I think this will be very benefitial to improve maintainability and readability.

@juliogonzalez
Copy link
Member

I don't think this is going to fly as you expect.

IIRC /repo/$RCE/SLES11-SP4-SUSE-Manager-Tools/sle-11-x86_64/ is the format provided by SCC, while /SuSE/build-ncc.suse.de/SUSE/Updates/SLE-SERVER/11-SP4-CLIENT-TOOLS/x86_64/update/ is the format provided by our internal mirror. NCC/SCC magic.

FMPOV there's something more urgent than this. We need a refactor of the way we manage the repositories for each OS. Having a single SLS file doesn't scale anymore. Then we can think about simplifying this as well (if possible).

@srbarrios
Copy link
Member Author

srbarrios commented Feb 15, 2021

I don't think this is going to fly as you expect.

IIRC /repo/$RCE/SLES11-SP4-SUSE-Manager-Tools/sle-11-x86_64/ is the format provided by SCC, while /SuSE/build-ncc.suse.de/SUSE/Updates/SLE-SERVER/11-SP4-CLIENT-TOOLS/x86_64/update/ is the format provided by our internal mirror. NCC/SCC magic.

FMPOV there's something more urgent than this. We need a refactor of the way we manage the repositories for each OS. Having a single SLS file doesn't scale anymore. Then we can think about simplifying this as well (if possible).

@juliogonzalez it doesn't matter the path used in the source origin, we can mirror any kind of url on minima, by using the example I provided and will mirror the content with exactly the same path.

But what we CAN'T afford is to use different content when we are using a mirror or not using a mirror, this will lead to misunderstandings, problems to maintain, unsync content, etcetera.
Our mirror must work "similar" as a transparent proxy-cache, with the difference that instead of requesting the content and then caching it for next time, we must populate the mirror with that content (using yml file and crons) so when we request that content, it's already there.
If you can't request content from SCC without a mirror, don't expect the mirror to provide you the content from SCC when you can use the mirror. Because then, the mirror becomes your SCC client, instead of a mirror. And this is not the purpose of the mirror.

I would like to clarify something about Minima mirror, this is how I see the Minima mirror usability:

  1. SCC section of Minima mirror yaml file: To be used by SUSE Manager server, using the disk mirror feature (configured on rhn.conf), when syncing products

  2. HTTP section of Minima mirror yaml file: To be used by Sumaform during the deployment and automatic configuration phase

Any repository used in Sumaform must be only temporary, using them to deploy and set up the environment, so none of these repositories must be used by SUMA or clients after the deployments. Also, all the repositories used for Uyuni must be accessible without our internal SUSE network.

@juliogonzalez
Copy link
Member

I don't recall exactly why this was done, and do not have time at this moment to research it again, but I seem to remember there was a reason to have this for products that come from "NCC". Please note that this "strange" thing happens only for SLE11 and (IIRC) RES6.

Maybe @rjmateus or @mcalmer remember better than me and know if your proposal will work.

@moio
Copy link
Contributor

moio commented Feb 15, 2021

@srbarrios minima will mirror an URL keeping its directory structure intact. If the server we are mirroring from has a different path than the URL served by SCC, then something has to bridge that gap.

Currently the gap is bridged in the SLS file, we could also move it to the minima configuration file, but the association still needs to be maintained somewhere.

So I would love to have one URL to rule them all and just change the host, but that is fundamentally not possible at least in some cases. I would not be against moving the mapping somewhere else sensible.

@srbarrios
Copy link
Member Author

srbarrios commented Feb 15, 2021

@srbarrios minima will mirror an URL keeping its directory structure intact. If the server we are mirroring from has a different path than the URL served by SCC, then something has to bridge that gap.

Currently the gap is bridged in the SLS file, we could also move it to the minima configuration file, but the association still needs to be maintained somewhere.

So I would love to have one URL to rule them all and just change the host, but that is fundamentally not possible at least in some cases. I would not be against moving the mapping somewhere else sensible.

@moio could you please tell me why the proposed change in the description will not work?

{% if 'beta' in grains.get('product_version') | default('', true) %}
- baseurl: http://{{ grains.get("mirror") | default("euklid.nue.suse.com", true) }}/mirror/SuSE/build-ncc.suse.de/SUSE/Updates/SLE-SERVER/11-SP4-CLIENT-TOOLS-BETA/x86_64/update/
{% else %}
- baseurl: http://{{ grains.get("mirror") | default("euklid.nue.suse.com", true) }}/mirror/SuSE/build-ncc.suse.de/SUSE/Updates/SLE-SERVER/11-SP4-CLIENT-TOOLS/x86_64/update/
{% endif %}
http:
  - url: http://euklid.nue.suse.com/mirror/SuSE/build-ncc.suse.de/SUSE/Updates/SLE-SERVER/11-SP4-CLIENT-TOOLS-BETA
    archs: [x86_64]
  - url: http://euklid.nue.suse.com/mirror/SuSE/build-ncc.suse.de/SUSE/Updates/SLE-SERVER/11-SP4-CLIENT-TOOLS
    archs: [x86_64]

Please let me know why we follow a strategy to use a different source of content when we use a mirror and when we don't use a mirror. In my understanding, as I described, we should use the mirror as a kind of proxy-cache, so we serve byte by byte the same content that we would download without the mirror.
If in this special case (I warn about to try to avoid getting stuck in this special case in the description too), we don't use IBS or OBS and use this other mirror "euklid.nue.suse.com", but we should treat this mirror just as the source of origin.
There is no need to distinguish between this and other sources, all sources must be treated equally, and mirror identically, all of them through "http section" of minima, and never by using "scc section", which in my humble option must be reserved to the use case I mentioned in my other comment, so only when SUMA server sync products through the disk mirror feature.
One of the reasons why I consider we shouldn't rely in SCC section of Mimina Mirror in Sumaform is to decouple this special feature of minima mirror from sumaform, so if a user want to use a different mirror implementation, can use it.

@mcalmer
Copy link
Contributor

mcalmer commented Feb 15, 2021

The problem is, that SCC has an "input" URL and an "output" URL which must not be the same.
This means for a mirror:

As long as we mirror everything from SCC/updates.suse.com directly, we get the output URL which then should be created also on the mirror. We can have 1 path independend of the mirror or scc used.

But if we use an internal mirror or the staging area, we get the "input" url which might be different to what SCC has as output URL.

This problem affects mostly SLES11 repos and maybe external repos like nvidia.

I would try to use https://updates.suse.de/ . Here you have the chance to get the URLs at the same place as SCC has it as output.

@srbarrios
Copy link
Member Author

srbarrios commented Feb 15, 2021

The problem is, that SCC has an "input" URL and an "output" URL which must not be the same.
This means for a mirror:

As long as we mirror everything from SCC/updates.suse.com directly, we get the output URL which then should be created also on the mirror. We can have 1 path independend of the mirror or scc used.

But if we use an internal mirror or the staging area, we get the "input" url which might be different to what SCC has as output URL.

This problem affects mostly SLES11 repos and maybe external repos like nvidia.

I would try to use https://updates.suse.de/ . Here you have the chance to get the URLs at the same place as SCC has it as output.

As I understand, this is not the problem I want that we address in this refactor. This is just another issue we have related to "this mirror that we are using as origin source" in some repos.

The refactor aims to simplify the code by using in-line jinja expression and by using always the same URI path with/without mirror. And as far we mirror what we use when we don't use a mirror, via http section of minima.yml, I can't see the technical issue.

@rjmateus
Copy link
Member

I agree that have it homogenize is a good thing. If what Michael mentioned is not a problem I think this would be great, and code would be simpler.

@moio
Copy link
Contributor

moio commented Feb 17, 2021

@srbarrios I do not disagree with the concept either - we are just warning you it might not be possible to apply it in all cases. We are all on the same page to homogenize to the extent posisble!

@srbarrios
Copy link
Member Author

@srbarrios I do not disagree with the concept either - we are just warning you it might not be possible to apply it in all cases. We are all on the same page to homogenize to the extent posisble!

I know I know, I just want to understand where it can not by applied, and the reason about it.
As far as I see the change, it can be applied everywhere. The main idea is to use the mirror as a mirror, and nothing else, so not trying to use the mirror as a by-pass of the SCC client issue, mentioned by MC.
For this case, we can use another minima mirror instance acting as a SCC client, but Sumaform shouldn't be aware of that infrastructure workaround, it should treat this other "mirror" just as tje source origin.
So, I think the description of the change still applying to all the cases.

If you want, I can make a draft, and we try it, not sure if this sprint, but sometime soon.

@moio
Copy link
Contributor

moio commented Feb 17, 2021

I am not sure I fully understand your idea, but feel free to open a PR.

I am in general fine with any PR that makes our life easier, provided it does not negatively impact current sumaform use cases.

As long as the sumaform mirror continues to work, feel free to shuffle URLs around to make code more readable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants