OKD Assisted Installation on Baremetal Cluster

[uvstudios]

Hello Everyone

Has anyone had success with installing a bare metal cluster through the assisted installer with OKD? Trying to determine the requirements to do this. I've been watching this video from RedHat about the assisted installer. Although from 2020, I believe the setup should be pretty much the same with generating an ISO to mount on the servers.

https://www.youtube.com/watch?v=BNC-5ISbxpU&t=729s

For OKD I need to host my own installer which I found a guide for in this documentation:

https://okd.io/docs/project/guides/sno-assisted-installer/

This seems to make sense, but my question is, is there any gotcha's with this configuration? I noticed that certain DNS records must be configured.

From looking at the openshift youtube video the API ip address can be configured via dhcp as well as the ingress ip. The one I am confused on is the wildcard one that must be set, what would you assign this to in a cluster?

There's also this in the documentation

https://docs.redhat.com/en/documentation/assisted_installer_for_openshift_container_[…]_container_platform_with_the_assisted_installer/prerequisites

But I am not completely clear about the 192.168.1.5 role is. From reading it's the load balancer, but does this need to be configured before the assisted installer?

I am a complete newbie when it comes to openshift. So maybe it's better to try and install a single node first and then jump into a cluster setup. Trying to learn some new technology and turn my proxmox cluster into something else.

Does anyone have a good tutorial or write up about the process? I'm looking for anything I could read or watch. Even if a book is helpful to learn more about the process.

Thanks for your guidance and help!

Chase

[titou10titou10]

Yes I had success installing latest version of OKD (v4.15, v4.16, v4.17) with AI, but now I use ABI (Agent based installer, internally used by AI) that I find easier to deal with as the "AI" component of ABI is running directly on the "rendez-vous" node without anything to do on my side, ie starting the AI somewhere and setup the cluster via the web

The installation of my OKD ABI UPI v4.17 clusters (3 masters + 3 workers) on Proxmox are 100% scripted with the prerequisites that

• a VM used as load balancer with either nginx or haproxy is up and running. I use Alma Linux for this and nginx for my v4.15 cluster and haproxy for v4.17 no particular reasons
• DNS is configured for the 6 nodes + LB node
• DHCP server is configured for the 6 nodes
• a ssh key is ready for the nodes
• "agent-config.yaml" and "install-config.yaml" files used by ABI/AI describing the cluster topology are ready

(Currently with the ABI/AI install of OKD-SCOS >= v4.16 I encountered 2 problems documented with workaround here :okd-project/okd-web#47 (comment))

Basically, the script download/extract the OKD installer, build the ABI iso install image from the agent-config.yaml/install-config.yaml config files. Then it creates the 6 vm on proxmox ("qm create...") with boot=iso image created previously and start the 6 nodes ("qm start...") and finally wait for the cluster to be up and running ("install agent wait-for bootstrap-complete..."
The install then proceed and after about 60-90 mins a cluster is up and running

And for "day-2" configuration, I have a few scripts that installs extra private CAs, set node labels, change the logo, install CSI NFS driver, cert manager, configure OAuth, setup API private certificate, ingress private certificates+http/2+proxy protocol, configure monitoring/alert manager, install tektoncd operator, velero, descheduler etc etc

Maybe I could document and make public all of this somewhere sometimes...
Maybe if you have specific question, you can contact me and I'll try to help you

As for some of your questions. With SNO you do not need a LB as everything will run in a single node/vm. API and web traffic will use the SNO IP
For a full flavour cluster, you need a lb in front of the cluster with 3 main functions

• route the incoming API traffic ("api.<cluster>.<domain>"+"api-int.<cluster>.<domain>", :6443, :22623) to the 3 master nodes
• route the incoming web traffic ("*.apps.<cluster>.<domain>", :80, :443) to the worker nodes where the ingress controller is running. This is on all two worker nodes by default (two first?)
• (optional) route the incoming "NodePort" traffic (I use "*.np.<cluster>.<domain>" to :30000-32767) to the worker nodes

All tcp traffic will go through the LB. The console installed as an app will be accessible at "console-openshift-console.apps.<cluster>.<domain>", so the DNS must be setup to point "api.<cluster>.<domain>", "api-int.<cluster>.<domain>", "*.apps.<cluster>.<domain>","*.np.<cluster>.<domain>" to the IP of your LB
In case of an SNO the DNS must assign all of them to the SNO node IP
And yes in the example, "192.168.1.5" is the address of the LB
You have an example of haproxy LB config for OKD/OCP in the OCP doc here: https://docs.openshift.com/container-platform/4.17/installing/installing_with_agent_based_installer/preparing-to-install-with-agent-based-installer.html#agent-install-load-balancing-none-example_preparing-to-install-with-agent-based-installer

[uvstudios]

Thanks so much for your answer and being so detailed. This definitely explains the process in more detail. From your response it seems like the best way to go about this is to use the ABI. Generate the configuration values with agent-config.yaml and install-config.yaml that will then create the desired ISO image. I'll have to do some additional reading on how to make this all happen. I did see some docs about creating the yaml files on the openshift documentation and what is required.

Is there anything in the OKD documentation about this?

Or is it okay to follow the documentation from openshift?
https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html-single/installing_an_on-premise_cluster_with_the_agent-based_installer/index#agent-based-installer-recommended-resources_preparing-to-install-with-agent-based-installer

I did see your documentation about the issues you faced with the AI. That might be my first step in this process just to get myself familiar with generating an install. See if I can get it to work for just a single node setup and then move onto creating a cluster.

If I can get everything worked out on my end I'd be happy to document the process of how it all comes together.

Once I get further along in the process maybe it would be okay to contact you and ask you a few questions. I don't want to take up too much of your time. I can tell you have a lot more experience with this than me. I've only deployed, proxmox, and Vmware hypervisors. With OKD there's a lot that can go wrong with creating the install unless you have everything defined perfectly.

The DNS setup makes much more sense now that the load balancer will handle all tcp request. Thank you for linked documentation.

I have a lot to digest and configure. Definitely gives me more of an idea of what to expect with the installation. Thank you for all your guidance and the time to you took to type it all out for me!

[titou10titou10]

Thanks so much for your answer and being so detailed.

yw

Or is it okay to follow the documentation from openshift? https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html-single/installing_an_on-premise_cluster_with_the_agent-based_installer/index#agent-based-installer-recommended-resources_preparing-to-install-with-agent-based-installer

Yes this is a good starting place

See if I can get it to work for just a single node setup and then move onto creating a cluster.

You can see an example of myagent-config.yamlandinstall-config.yamlfor an SNO installation here: #2015

Once I get further along in the process maybe it would be okay to contact you and ask you a few questions. I

Yep

Basic steps for creating the iso booting image

get the oc v4.17 client...put it on the path
mkdir install 
copy your agent-config.yaml and install-config.yaml to "install" dir
# override for "FCOS bootstrap node not pivoting"
export OPENSHIFT_INSTALL_OS_IMAGE_OVERRIDE=https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.17/4.17.0/rhcos-4.17.0-x86_64-live.x86_64.iso
oc adm release extract --command=openshift-install quay.io/okd/scos-release:4.17.0-okd-scos.1
./openshift-install agent create image --dir install --log-level=debug
cp install/agent.x86_64.iso ...somewhere for your vm to boot

then create and boot your SNO nodes...
follow the installation with:

./openshift-install --dir install agent wait-for bootstrap-complete --log-level=debug
./openshift-install --dir install agent wait-for install-complete --log-level=debug

be aware that the boot process checks the capabilities of your nodes. like the disk size (must be >100GB, the cpu latency, the RAM etc. I use 10vCPU and 40GB RAM for a SNO install but 8vCPU and 32GB RMA is ok)
if you do multiple attempts, it is mandatory to delete/recreate the install dir as the content is "consumed" by the openshift-install process
the password for the login with "kubeadmin" user will be ininstall/auth/kubeadmin-password and the kubeconfig to use for login ininstall/auth/kubeconfig: export KUBECONFIG=install/auth/kubeconfig