Regarding stable release version of UPYOG

[nsrivastava2]

Q1. As per the discussions during the meetings we have understood that https://github.com/upyog Tag v2.0 is the the stable release and that is the one we shall be using for the FSM module.

We are currently using the UPYOG Devops Build - https://github.com/upyog/UPYOG-DevOps/tree/v2.0/deploy-as-code/egov-deployer

There were certain changes done by NIUA team to get the FSM module working but it is not stable and we are not clear that which code version was it used to get it working. Please guide us.

Q2. We need the Docker tags for pgr, fsm, business, and core to run version 2.0 applications. Can we use the Docker tag from the 2.0 release chart?

Q3. HRMS service from backend is up and running, but while validating the same from end it got crashed and no logs can be seen from the application logs, kindly confirm login credentials for employee login, as now I am using GRO with eGov@4321 password.

Our Deployed version is: https://upyog.velsof.com

[nsrivastava2]

Error showing while trying to save FSM request:
On this page when I choose SLUM as YES then this error comes.

org.egov.tracer.model.CustomException: [{code=BUSINESSSERVICE ERROR, message=No bussinessService object found for businessSerice: FSM_ZERO_PAY_SERVICE and tenantId: pg.citya, description=null, params=null}]
        at org.egov.fsm.workflow.WorkflowIntegrator.callWorkFlow(WorkflowIntegrator.java:141)
        at org.egov.fsm.service.FSMService.create(FSMService.java:125)
        at org.egov.fsm.web.controller.FSMController.create(FSMController.java:52)
        at sun.reflect.GeneratedMethodAccessor409.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:165)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.egov.tracer.http.filters.TracerFilter.doFilter(TracerFilter.java:109)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:109)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

[nsrivastava2]

As per our discussions over the call today this is what has been discussed:

1. We shall not use the Docker Images of UPYOG as UPYOG images are being updated constantly. We shall create our own Dockerhub and shall created every Docker image ourself.

2. Below mentioned are the URLs that has been confirmed to be used:

We shall use the code from:
https://github.com/upyog/UPYOG/tree/v2.0 - This is the tag which is security audited and tested with the FSM modules.

For the frontend digit-ui code is available here:
https://github.com/upyog/UPYOG/tree/v2.0/frontend/micro-ui

For Docker Images Creation use:
https://github.com/upyog/UPYOG/blob/master/build/maven/Dockerfile

For CI/CD setup using Jenkins use:
https://github.com/upyog/CIOps - For Groovy Script of Jenkins

For Deployment use:
https://github.com/upyog/UPYOG-DevOps/tree/master/ (The master branch is to be used)

https://upyog-docs.gitbook.io/upyog-v-1.0/upyog-1/platform/setup-upyog/ci-cd-setup/ci-cd-build-job-pipeline-setup - Documentation for CI/CD

[support-upyog]

Yes

[nsrivastava2]

ISSUES FACED and RESOLUTION

As per the discussion comments, we used MASTER branch

https://github.com/upyog/UPYOG-DevOps/tree/master
Branch - master

Error 1: Terraform version 0.14.10

Error 2:

Command - terraform init

Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/kubernetes: no available releases match the
│ given constraints ~> 1.11, >= 2.10.0

Solution: Changed from 1.11, >= 2.10.0

provider "kubernetes" {
  host                   = "${data.aws_eks_cluster.cluster.endpoint}"
  cluster_ca_certificate = "${base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)}"
  token                  = "${data.aws_eks_cluster_auth.cluster.token}"
  load_config_file       = false
  version                = "~> 2.10.0"
}

Error 3:

Error: Unsupported argument

│ 
│   on main.tf line 72, in module "eks":
│   72:   subnets         = "${concat(module.network.private_subnets, module.network.public_subnets)}"
│ 
│ An argument named "subnets" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│   on main.tf line 83, in module "eks":
│   83:   worker_groups_launch_template = [
│ 
│ An argument named "worker_groups_launch_template" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│   on main.tf line 96, in module "eks":
│   96:   map_users    = [
│ 
│ An argument named "map_users" is not expected here.

Solutions: Need to remove the whole section

-------------------------------------------------------------------------------
### Since the Master Branch was not working so we switched to RELEASE Branch.
https://github.com/upyog/UPYOG-DevOps/tree/master

Branch - release

Errors 1: Unsupported formatat

output "db_instance_name" {
  description = "The database name"
  value       = "${aws_db_instance.rds_postgres.name }" 
}

Solution - name -> id

Error 2:

Error: creating RDS DB Instance (upyog-eks-dev-db): InvalidParameterCombination: Cannot find version 11.13 for postgres
        status code: 400, request id: 9e5b71ec-5ab0-4d13-b625-a5f28375dca8

Solution - engine_version                = "11.13"   ## postgres version — > changed this to 11.20

Error 3:

db: // update the postgres db credentials
username: "upyogdevxxxx"
password: "upyogdevxxxx"
flywayUsername: "upyogdevxxxx"
flywayPassword: "upyogdevxxxx"
egov-notification-sms: // To work sms notification service add the sms provider details
username: demo

Solution - Change all // with #

Error 4:

/root/project/UPYOG-DevOps/config-as-code/helm/charts/cluster-configs
2023/10/22 03:59:01 ------------------------------------ DEPLOYING CLUSTER CONFIGS ------------------------------------
2023/10/22 03:59:01 helm template -f /root/project/UPYOG-DevOps/config-as-code/environments/egov-demo.yaml --output-dir /tmp/helm-904731347 -f /root/project/UPYOG-DevOps/config-as-code/environments/egov-demo-secrets.yaml .
2023/10/22 03:59:07 exit status 1: error: unable to recognize "ingress/root-ingress.yaml": no matches for kind "Ingress" in version "extensions/v1beta1"

panic: exit status 1: error: unable to recognize "ingress/root-ingress.yaml": no matches for kind "Ingress" in version "extensions/v1beta1"

Solution - Changed this to “networking.k8s.io/v1”

Error 5:

2023/10/22 04:13:43 ------------------------------------ DEPLOYING CLUSTER CONFIGS ------------------------------------
2023/10/22 04:13:43 helm template -f /root/project/UPYOG-DevOps/config-as-code/environments/egov-demo.yaml --output-dir /tmp/helm-3219531512 -f /root/project/UPYOG-DevOps/config-as-code/environments/egov-demo-secrets.yaml .
2023/10/22 04:13:46 exit status 1: error: error validating "ingress/root-ingress.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0]): missing required field "pathType" in io.k8s.api.networking.v1.HTTPIngressPath]; if you choose to ignore these errors, turn validation off with --validate=false

panic: exit status 1: error: error validating "ingress/root-ingress.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0]): missing required field "pathType" in io.k8s.api.networking.v1.HTTPIngressPath]; if you choose to ignore these errors, turn validation off with --validate=false

Solution - Replaced with bootcamp ingress file

Error 6:

2023/10/22 04:26:30 ------------------------------------ DEPLOYING egovio/fsm-calculator-db:fsm-services-dev-f4dc1171-9 ------------------------------------
2023/10/22 04:26:30 /root/project/UPYOG-DevOps/config-as-code/helm/charts/municipal-services/fsm-calculator
2023/10/22 04:26:30 Generating final manifests to directory : /tmp/helm-3189125796 
2023/10/22 04:26:30 Applying manifests to the cluster 
2023/10/22 04:26:31 exit status 1: Warning: spec.template.spec.containers[0].env[36].name: duplicate name "TRACER_OPENTRACING_ENABLED"
error: unable to recognize "fsm-calculator/templates/ingress.yaml": no matches for kind "Ingress" in version "extensions/v1beta1"

panic: exit status 1: Warning: spec.template.spec.containers[0].env[36].name: duplicate name "TRACER_OPENTRACING_ENABLED"
error: unable to recognize "fsm-calculator/templates/ingress.yaml": no matches for kind "Ingress" in version "extensions/v1beta1"

Solution - Replaced common ingress file with bootcamp branch

Error 7:

2023/10/22 04:31:14 /root/project/UPYOG-DevOps/config-as-code/helm/charts/backbone-services/nginx-ingress
2023/10/22 04:31:15 Generating final manifests to directory : /tmp/helm-859171832 
2023/10/22 04:31:15 Applying manifests to the cluster 
2023/10/22 04:31:17 exit status 1: unable to recognize "nginx-ingress/templates/clusterrole.yaml": no matches for kind "ClusterRole" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "nginx-ingress/templates/clusterrolebinding.yaml": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "nginx-ingress/templates/controller-role.yaml": no matches for kind "Role" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "nginx-ingress/templates/controller-rolebinding.yaml": no matches for kind "RoleBinding" in version "rbac.authorization.k8s.io/v1beta1"

panic: exit status 1: unable to recognize "nginx-ingress/templates/clusterrole.yaml": no matches for kind "ClusterRole" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "nginx-ingress/templates/clusterrolebinding.yaml": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "nginx-ingress/templates/controller-role.yaml": no matches for kind "Role" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "nginx-ingress/templates/controller-rolebinding.yaml": no matches for kind "RoleBinding" in version "rbac.authorization.k8s.io/v1beta1"

Solutions - ??

--------------------------------------------------------------
### We tried using Bootcamp-jenkins to get the CICD setup so that the deployments may work.

Bootcamp-jenkins

Error - Egov namespace was missing

Help Needed
We need to know that since the layout of UPYOG is different from DIGIT do you have Jenkins for UPYOG as well.

jobBuilder: # Add jobBuilder repo's, add repo's ssh link
repos: >-
'[email protected]:egovernments/core-services.git','[email protected]:egovernments/business-services.git'
deploymentJobs: # Setup deploymentJobs ACL

2. Need to talk about Jenkins-credentials secret missing

Egov-cicd

variable "kubernetes_version" {
  default = "1.20"   —--> 22
}

[nsrivastava2]

For the error related to Jenkins Secrets you have asked us to open the ci-secrets.yaml file. Screenshot of the same can be found here:

As explained by you, the solution is to provide the ClientID and ClientSecret. For these details we need to create an Auth App in Github and provide read access to the repos to this app. The ClientID and ClientSecret of this app is what needs to be supplied in the file mentioned above.

We will test it out and share the results.

Note: Whatever URL you areusing for the application in the egov-demo.yaml is the callback url we have been told need to provide while creating the Auth Token. Basically the URL on which you are setting up the UPYOG module example: upyog.yourdomain.com

[nsrivastava2]

ISSUES:
We've set up our CI/CD pipeline on AWS EKS, and everything seems fine with the pods. Our job builder also ran successfully and created a "build" folder with all the services. However, when we tried to build a specific service, like "digit UI, FSM" we are encountering the following problem: - It seems like it's trying to use a Google Cloud service account during the build, even though we have set up DockerHub credentials in our CI YAML file.

Error - 

Created Pod: jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-v95p4
[Normal][jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-v95p4][Scheduled] Successfully assigned jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-v95p4 to ip-192-168-73-54.ap-south-1.compute.internal
[Warning][jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-v95p4][FailedMount] MountVolume.SetUp failed for volume "service-account" : references non-existent secret key: gcpServiceAccount
[Warning][jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-v95p4][FailedMount] MountVolume.SetUp failed for volume "service-account" : references non-existent secret key: gcpServiceAccount
Still waiting to schedule task
‘builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-v95p4’ is offline
[Warning][jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-v95p4][FailedMount] MountVolume.SetUp failed for volume "service-account" : references non-existent secret key: gcpServiceAccount
[Warning][jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-v95p4][FailedMount] MountVolume.SetUp failed for volume "service-account" : references non-existent secret key: gcpServiceAccount
[Warning][jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-v95p4][FailedMount] MountVolume.SetUp failed for volume "service-account" : references non-existent secret key: gcpServiceAccount
[Warning][jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-v95p4][FailedMount] MountVolume.SetUp failed for volume "service-account" : references non-existent secret key: gcpServiceAccount
[Warning][jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-v95p4][FailedMount] MountVolume.SetUp failed for volume "service-account" : references non-existent secret key: gcpServiceAccount
[Warning][jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-v95p4][FailedMount] MountVolume.SetUp failed for volume "service-account" : references non-existent secret key: gcpServiceAccount


**Sometime also observed -** 

Created Pod: jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-7tljt
[Warning][jenkins/builds-upyog-frontend-micro-ui-digit-ui-1-gt04r-2710f-7tljt][FailedScheduling] 0/2 nodes are available: 2 Insufficient cpu.

But we have two K8 worker nodes with enough CPU.

[support-upyog]

Please join the call today

[nsrivastava2]

Thank you team for solving the GCP credentials problem. The issue was fixed after we removed the GCP part from the CIOPS repository. However, we're now encountering the following errors.

[Pipeline] node
Still waiting to schedule task
Waiting for next available executor

[support-upyog]

Please join the Today Technical Call @4-5 PM.