Merge pull request #507 from yussufsh/vpctgw

Add support for creating IBM Cloud services

docs/ocp_prereqs_powervs.md

@@ -61,10 +61,11 @@ Two options are available to enable communication over the private network.
 
 *Option 1*
 
-You can use the IBM Cloud CLI with the latest power-iaas plug-in (version 0.3.4 or later) to enable a private network communication.
-Refer: https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-managing-cloud-connections
+Now, the automation can create a [Cloud Connection](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-cloud-connections) for you. IBM Cloud connection creates a Direct Link (2.0) Connect instance to connect your Power Virtual Server instances to the IBM Cloud resources within your account.
 
-This requires attaching the private network to an IBM Cloud Direct Link Connect 2.0 connection.
+You can now use [IBM Cloud Transit Gateway](https://cloud.ibm.com/docs/transit-gateway?topic=transit-gateway-about) to manage the connection of your Power Virtual Server instances to the IBM Cloud resources. This feature is available when you set `use_ibm_cloud_services = true`. Skip the steps given below and please refer to [var.tfvars](./var.tfvars-doc.md#using-ibm-cloud-services) for more details.
+
+Use the IBM Cloud CLI with the latest power-iaas plug-in (version 0.3.4 or later) to enable a private network communication. This requires attaching the private network to an [IBM Cloud Direct Link Connect 2.0 connection](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-managing-cloud-connections).
 Perform the following steps to enable private network communication by attaching to the Direct Link Connect 2.0 connection.
 
 - Select a specific service instance
@@ -189,8 +190,12 @@ Click "**Continue**" to accept agreements, and then Click "**Submit case**".
 
 This usually takes a day to get enabled.
 
-## RHCOS and RHEL/CentOS 8.X Images for OpenShift
-RHEL image is used for bastion and RHCOS is used for the OpenShift cluster nodes.
+## RHCOS and RHEL/CentOS Images for OpenShift
+RHEL/CentOS (8.X or 9.X) image is used for bastion and RHCOS is used for the OpenShift cluster nodes.
+
+You can now use stock images for the bastion node. The automation will copy it to your PowerVS service instance if not already available.
+
+For RHCOS (OpenShift cluster nodes) you can let the automation import it for you from public COS bucket by setting the variable `rhcos_import_image = true`. Skip the steps given below for RHCOS and please refer to [var.tfvars](./var.tfvars-doc.md#openshift-cluster-details) for more details. Value of `rhcos_import_image_filename` can be refered from [rhcos-table.md](./rhcos-table.md) specific to the required OpenShit version.
 
 You'll need to create [OVA](https://en.wikipedia.org/wiki/Open_Virtualization_Format) formatted images for RHEL and RHCOS, upload them to IBM Cloud Object storage and then import these images as boot images in your PowerVS service instance.
 
@@ -206,6 +211,7 @@ Further, the image disk should be minimum of 120 GB in size.
   - RHCOS Qcow2 image is available at [latest stable](https://mirror.openshift.com/pub/openshift-v4/ppc64le/dependencies/rhcos/latest/rhcos-openstack.ppc64le.qcow2.gz) OR [pre-release](https://mirror.openshift.com/pub/openshift-v4/ppc64le/dependencies/rhcos/pre-release/latest/rhcos-openstack.ppc64le.qcow2.gz)
 
 Note: RHCOS image version is tied to the specific OCP release. For example RHCOS-4.6 image needs to be used for OCP 4.6 release.
+
 ### Uploading to IBM Cloud Object Storage
 
 - **Create IBM Cloud Object Storage service and bucket**

docs/var.tfvars-doc.md

@@ -124,7 +124,7 @@ Note that the boot images should have a minimum disk size of 120GB
 These set of variables should be provided when RHCOS image should be imported from public bucket of cloud object storage to your PowerVS service instance
 ```
 rhcos_import_image              = true                                                   # true/false (default=false)
-rhcos_import_image_filename     = "rhcos-411-85-202203181612-0-ppc64le-powervs.ova.gz"   # RHCOS boot image file name available in cloud object storage
+rhcos_import_image_filename     = "rhcos-415-92-202310310037-0-ppc64le-powervs.ova.gz"   # RHCOS boot image file name available in cloud object storage
 rhcos_import_image_storage_type = "tier1"                                                # tier1/tier3 (default=tier1) Storage type in PowerVS where image needs to be uploaded
 ```
 
@@ -236,20 +236,25 @@ Note: Once fips_compliant set to true it will enable FIPS on the OCP cluster and
 
 You can use IBM Cloud Internet Services (CIS) and Load Balancer services on VPC for running the OCP cluster. When this feature is enabled the services called `named` (DNS) and `haproxy` (Load Balancer) will not be running on the bastion/helpernode.
 
-Ensure you have setup [Cloud Connection](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-cloud-connections) or [DirectLink](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-ordering-direct-link-connect) with IBM Cloud VPC over the private network in cloud instance. Also, ensure you have registered a [DNS domain](https://cloud.ibm.com/docs/cis?topic=cis-about-ibm-cloud-internet-services-cis) and use it as given in `cluster_domain` variable.
+Ensure you have registered a [DNS domain](https://cloud.ibm.com/docs/cis?topic=cis-about-ibm-cloud-internet-services-cis) and use it as given in `cluster_domain` variable.
 
-**IMPORTANT**: This is an **experimental** feature at present. Please manually set variables `setup_snat = true` and `setup_squid_proxy = false` for using IBM Cloud services. This will allow the cluster nodes have public internet access without a proxy server.
 
 Below variables needs to be set in order to use the IBM Cloud services.
 
 ```
 use_ibm_cloud_services    = true
-ibm_cloud_vpc_name        = "ocp-vpc"
-ibm_cloud_vpc_subnet_name = "ocp-subnet"
+ibm_cloud_vpc_name        = "ocp-vpc"     # If empty a new VPC will be created in `iaas_vpc_region`.
+ibm_cloud_vpc_subnet_name = "ocp-subnet"  # If empty a new VPC Subnet will be created in the first AZ.
+ibm_cloud_resource_group  = "Default"     # Used for creating new VPC resources
 iaas_vpc_region           = "us-south" # the VPC region for accessing IBM Cloud services. If empty, will default to ibmcloud_region.
 ibm_cloud_cis_crn         = "crn:v1:bluemix:public:internet-svcs:global:a/<account_id>:<cis_instance_id>::" # CRN of the CIS instance where domain is registered.
+# Below are the variables required for setting up Transit Gateway and add VPC, PowerVS connections to it.
+ibm_cloud_tgw                   = ""  # Name of existing Transit Gateway where VPC and PowerVS targets are already added. If empty it will create a new Transit Gateway with VPC, PowerVS connected to it (includes support for PER enabled workspace as well).
+ibm_cloud_connection_name = ""  # Name of the cloud connection which is already attached to the above Transit Gateway. If empty a new cloud connection is created and added to above (or new) Transit Gateway. Not applicable for PER enabled workspaces.
 ```
 
+>**Note**: If you just need to add the DNS entries in the CIS domain zone without using IBM Cloud Services eg: VPC Load Balancer, then set `ibm_cloud_cis_crn` with the CIS domain CRN and keep `use_ibm_cloud_services = false`.
+
 ### Misc Customizations
 
 These variables provides miscellaneous customizations. For common usage scenarios these are not required and should be left unchanged.
@@ -339,8 +344,7 @@ chrony_config               = true
 chrony_config_servers       = [ {server = "0.centos.pool.ntp.org", options = "iburst"}, {server = "1.centos.pool.ntp.org", options = "iburst"} ]
 ```
 
-These set of variables are specific for cluster wide proxy configuration.
-Public internet access for the OpenShift cluster nodes is via Squid proxy deployed on the bastion.
+These set of variables are specific for cluster wide proxy configuration. Public internet access for the OpenShift cluster nodes is via Squid proxy deployed on the bastion. Ignored when `use_ibm_cloud_services = true`.
 ```
 setup_squid_proxy           = true
 ```
@@ -392,7 +396,7 @@ This variable is used to set the default Container Network Interface (CNI) netwo
 cni_network_provider       = "OVNKubernetes"
 ```
 
-This variable is used to enable SNAT for OCP nodes. When using SNAT, the OCP nodes will be able to access public internet without using a proxy
+This variable is used to enable SNAT for OCP nodes. When using SNAT, the OCP nodes will be able to access public internet without using a proxy. Ignored when `use_ibm_cloud_services = true`.
 
 ```
 setup_snat                 = true

modules/1_prepare/outputs.tf

@@ -54,3 +54,8 @@ output "bastion_external_vip" {
   depends_on = [null_resource.bastion_init]
   value      = local.bastion_count > 1 ? ibm_pi_network_port.bastion_internal_vip[0].public_ip : ""
 }
+
+output "cloud_connection_name" {
+  depends_on = [ibm_pi_cloud_connection.cloud_connection, time_sleep.wait_for_cc]
+  value      = var.create_cloud_connection ? ibm_pi_cloud_connection.cloud_connection[0].pi_cloud_connection_name : ""
+}

modules/1_prepare/prepare.tf

@@ -441,3 +441,22 @@ resource "ibm_pi_network_port" "bastion_internal_vip" {
   pi_network_name      = ibm_pi_network.public_network.pi_network_name
   pi_cloud_instance_id = var.service_instance_id
 }
+
+resource "ibm_pi_cloud_connection" "cloud_connection" {
+  count = var.create_cloud_connection ? 1 : 0
+
+  pi_cloud_instance_id                = var.service_instance_id
+  pi_cloud_connection_name            = "${var.cluster_id}-cc"
+  pi_cloud_connection_speed           = 100
+  pi_cloud_connection_global_routing  = true
+  pi_cloud_connection_transit_enabled = true
+  pi_cloud_connection_networks        = [data.ibm_pi_network.network.id]
+}
+# Give some time to change the status
+# after cc is created
+resource "time_sleep" "wait_for_cc" {
+  count = var.create_cloud_connection ? 1 : 0
+
+  depends_on      = [ibm_pi_cloud_connection.cloud_connection]
+  create_duration = "3m"
+}

modules/1_prepare/variables.tf

@@ -80,3 +80,5 @@ variable "volume_shareable" {}
 variable "setup_squid_proxy" {}
 variable "proxy" {}
 variable "fips_compliant" {}
+
+variable "create_cloud_connection" {}

modules/1_prepare/versions.tf

@@ -22,12 +22,15 @@ terraform {
   required_providers {
     ibm = {
       source  = "IBM-Cloud/ibm"
-      version = "~> 1.54.0"
+      version = "1.60.0"
     }
     null = {
       source  = "hashicorp/null"
       version = "~> 3.2"
     }
+    time = {
+      source = "hashicorp/time"
+    }
   }
   required_version = ">= 1.2.0"
 }

modules/4_nodes/nodes.tf

@@ -150,6 +150,10 @@ resource "ibm_pi_instance" "master" {
   pi_network {
     network_id = data.ibm_pi_network.network.id
   }
+
+  lifecycle {
+    ignore_changes = [pi_storage_pool_affinity]
+  }
 }
 resource "ibm_pi_instance_action" "master_stop" {
   count = var.master["count"]
@@ -216,6 +220,10 @@ resource "ibm_pi_instance" "worker" {
   pi_network {
     network_id = data.ibm_pi_network.network.id
   }
+
+  lifecycle {
+    ignore_changes = [pi_storage_pool_affinity]
+  }
 }
 resource "ibm_pi_instance_action" "worker_stop" {
   count = var.worker["count"]

modules/4_nodes/versions.tf

@@ -22,7 +22,7 @@ terraform {
   required_providers {
     ibm = {
       source  = "IBM-Cloud/ibm"
-      version = "~> 1.54.0"
+      version = "1.60.0"
     }
     ignition = {
       source  = "community-terraform-providers/ignition"

modules/5_install/versions.tf

@@ -26,7 +26,7 @@ terraform {
     }
     ibm = {
       source  = "IBM-Cloud/ibm"
-      version = "~> 1.54.0"
+      version = "1.60.0"
     }
 
   }

modules/7_ibmcloud/load_balancer.tf

@@ -32,16 +32,16 @@ locals {
 
 resource "ibm_is_lb" "load_balancer_internal" {
   name            = "${var.name_prefix}internal-loadbalancer"
-  resource_group  = data.ibm_is_vpc.vpc.resource_group
-  subnets         = [var.vpc_subnet_id]
+  resource_group  = local.resource_group_id
+  subnets         = [local.vpc_subnet_id]
   security_groups = [ibm_is_security_group.ocp_security_group.id]
   type            = "private"
 }
 
 resource "ibm_is_lb" "load_balancer_external" {
   name            = "${var.name_prefix}external-loadbalancer"
-  resource_group  = data.ibm_is_vpc.vpc.resource_group
-  subnets         = [var.vpc_subnet_id]
+  resource_group  = local.resource_group_id
+  subnets         = [local.vpc_subnet_id]
   security_groups = [ibm_is_security_group.ocp_security_group.id]
   type            = "public"
 }

modules/7_ibmcloud/outputs.tf

@@ -21,3 +21,7 @@
 output "load_balancer_hostname" {
   value = ibm_is_lb.load_balancer_external.hostname
 }
+
+output "vpc_cidr" {
+  value = local.vpc_subnet_cidr
+}

modules/7_ibmcloud/security_group.tf

@@ -21,14 +21,11 @@
 locals {
   tcp_ports = [22623, 6443, 443, 80]
 }
-data "ibm_is_vpc" "vpc" {
-  name = var.vpc_name
-}
 
 resource "ibm_is_security_group" "ocp_security_group" {
   name           = "${var.name_prefix}ocp-sec-group"
-  vpc            = data.ibm_is_vpc.vpc.id
-  resource_group = data.ibm_is_vpc.vpc.resource_group
+  vpc            = local.vpc_id
+  resource_group = local.resource_group_id
 }
 
 resource "ibm_is_security_group_rule" "inbound_ports" {

modules/7_ibmcloud/tgw.tf

@@ -0,0 +1,64 @@
+################################################################
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Licensed Materials - Property of IBM
+#
+# ©Copyright IBM Corp. 2023
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+################################################################
+
+data "ibm_tg_gateway" "transit_gateway" {
+  count = local.create_tgw ? 0 : 1
+
+  name = var.ibm_cloud_tgw
+}
+
+resource "ibm_tg_gateway" "transit_gateway" {
+  count = local.create_tgw ? 1 : 0
+
+  name           = "${var.cluster_id}-tgw"
+  location       = var.vpc_region
+  global         = true
+  resource_group = local.resource_group_id
+}
+
+resource "ibm_tg_connection" "tg_connection_vpc" {
+  count = local.create_vpc || local.create_tgw ? 1 : 0
+
+  gateway      = local.tgw_id
+  network_type = "vpc"
+  name         = "${var.cluster_id}-conn-vpc"
+  network_id   = local.vpc_crn
+}
+
+resource "ibm_tg_connection" "tg_connection_powervs" {
+  count = var.is_new_cloud_connection || local.create_tgw ? 1 : 0
+
+  gateway      = local.tgw_id
+  network_type = var.is_per ? "power_virtual_server" : "directlink"
+  name         = "${var.cluster_id}-conn-powervs"
+  network_id   = var.is_per ? var.ibm_cloud_tgw_net : data.ibm_dl_gateway.dl[0].crn
+}
+
+# If power workspace is given not required
+data "ibm_dl_gateway" "dl" {
+  count      = var.is_per ? 0 : 1
+  depends_on = [var.ibm_cloud_tgw_net]
+  name       = var.ibm_cloud_tgw_net
+}
+
+locals {
+  create_tgw = var.ibm_cloud_tgw == "" ? true : false
+  tgw_id     = local.create_tgw ? resource.ibm_tg_gateway.transit_gateway[0].id : data.ibm_tg_gateway.transit_gateway[0].id
+}

modules/7_ibmcloud/variables.tf

@@ -28,8 +28,14 @@ variable "name_prefix" {}
 variable "node_prefix" {}
 
 variable "vpc_name" {}
-variable "vpc_subnet_id" {}
+variable "vpc_subnet_name" {}
+variable "vpc_region" {}
+variable "ibm_cloud_resource_group" {}
 variable "ibm_cloud_cis_crn" {}
+variable "ibm_cloud_tgw" {}
+variable "ibm_cloud_tgw_net" {}
+variable "is_per" {}
+variable "is_new_cloud_connection" {}
 
 variable "bastion_count" {}
 variable "bootstrap_count" {}

modules/7_ibmcloud/versions.tf

@@ -22,7 +22,7 @@ terraform {
   required_providers {
     ibm = {
       source  = "ibm-cloud/ibm"
-      version = "~> 1.54.0"
+      version = "1.60.0"
     }
   }
   required_version = ">= 1.2.0"

modules/7_ibmcloud/vpc.tf

@@ -0,0 +1,64 @@
+################################################################
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Licensed Materials - Property of IBM
+#
+# ©Copyright IBM Corp. 2023
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+################################################################
+
+locals {
+  create_vpc        = var.vpc_name == ""
+  create_subnet     = var.vpc_name == "" || var.vpc_subnet_name == ""
+  vpc_id            = local.create_vpc ? ibm_is_vpc.vpc[0].id : data.ibm_is_vpc.vpc[0].id
+  vpc_crn           = local.create_vpc ? ibm_is_vpc.vpc[0].crn : data.ibm_is_vpc.vpc[0].crn
+  vpc_subnet_id     = local.create_subnet ? ibm_is_subnet.subnet[0].id : data.ibm_is_subnet.subnet[0].id
+  vpc_subnet_cidr   = local.create_subnet ? ibm_is_subnet.subnet[0].ipv4_cidr_block : data.ibm_is_subnet.subnet[0].ipv4_cidr_block
+  resource_group_id = local.create_vpc ? data.ibm_resource_group.group.id : data.ibm_is_vpc.vpc[0].resource_group
+}
+
+data "ibm_is_vpc" "vpc" {
+  count = local.create_vpc ? 0 : 1
+
+  name = var.vpc_name
+}
+
+data "ibm_is_subnet" "subnet" {
+  count = local.create_subnet ? 0 : 1
+
+  name = var.vpc_subnet_name
+}
+
+data "ibm_resource_group" "group" {
+  name = var.ibm_cloud_resource_group
+}
+
+resource "ibm_is_vpc" "vpc" {
+  count = local.create_vpc ? 1 : 0
+
+  name           = "${var.cluster_id}-vpc"
+  resource_group = local.resource_group_id
+  tags           = [var.cluster_id, "powervs-openshift"]
+}
+
+resource "ibm_is_subnet" "subnet" {
+  count = local.create_subnet ? 1 : 0
+
+  name                     = "${var.cluster_id}-subnet"
+  vpc                      = local.create_vpc ? ibm_is_vpc.vpc[0].id : data.ibm_is_vpc.vpc[0].id
+  resource_group           = local.resource_group_id
+  total_ipv4_address_count = 256
+  zone                     = "${var.vpc_region}-1"
+  tags                     = [var.cluster_id, "powervs-openshift"]
+}