Docker Patch Library to support Docker re-release Automation #268

Divyaasm · 2023-08-01T19:11:47Z

Description

Addressing the Issue

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2023-08-01T19:30:40Z

Codecov Report

Merging #268 (b339016) into main (f184ed3) will increase coverage by 0.06%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main     #268      +/-   ##
============================================
+ Coverage     86.63%   86.69%   +0.06%     
  Complexity       27       27              
============================================
  Files            77       78       +1     
  Lines           202      203       +1     
  Branches         11       11              
============================================
+ Hits            175      176       +1     
  Misses           19       19              
  Partials          8        8

Files Changed	Coverage Δ
tests/jenkins/jobs/PatchDockerImage_Jenkinsfile	`100.00% <100.00%> (ø)`

Signed-off-by: Divya Madala <[email protected]>

gaiksaya · 2023-08-01T20:53:05Z

vars/patchDockerImage.groovy

+    sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.version"}}' ${docker_image} > versionnumber"""
+
+    sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.build-date"}}' ${docker_image} > time"""
+
+    sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.description"}}' ${docker_image} > number"""


Does it make sense to combine these shell commands into above sh""" block.
Also avoid using time as variable name. Can mess up actual clock sometimes.

Yeah but having all these statements in a single block isn't storing the output value in a variable. That's technically resulting an error. I'll change time variable name. Thanks

That doesn't sound right! This might be the culprit https://github.com/opensearch-project/opensearch-build-libraries/pull/268/files#diff-d42288632ebcbc435313fa808e87e49f58461165639396b6ec30cbb928596336R15-R17

I believe by default the shell is zsh and here we are running bash. Also those lines apply only to one command docker pull ${docker_image}. Try removing those and put it under default shell

gaiksaya · 2023-08-01T20:53:14Z

vars/patchDockerImage.groovy

+
+    docker pull ${docker_image}
+    """
+    sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.version"}}' ${docker_image} > versionnumber"""


Suggested change

sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.version"}}' ${docker_image} > versionnumber"""

sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.version"}}' ${docker_image} > versionNumber"""

Use sh("") if you are trying to do one liner unless you have other reasons.

gaiksaya · 2023-08-01T20:53:58Z

vars/patchDockerImage.groovy

+
+    sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.build-date"}}' ${docker_image} > time"""
+
+    sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.description"}}' ${docker_image} > number"""


Suggested change

sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.description"}}' ${docker_image} > number"""

sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.description"}}' ${docker_image} > buildNumber"""

gaiksaya · 2023-08-01T20:54:47Z

vars/patchDockerImage.groovy

+    if (artifactUrlX64 == null || artifactUrlARM64 ==  null) {
+    echo 'Skipping docker build, one of x64 or arm64 artifacts was not built.'
+    } else {


Please format the document for proper alignment.

Sure Sayali

gaiksaya · 2023-08-01T20:54:57Z

vars/patchDockerImage.groovy

+    if (artifactUrlX64 == null || artifactUrlARM64 ==  null) {
+    echo 'Skipping docker build, one of x64 or arm64 artifacts was not built.'
+    } else {
+        echo 'Trigger docker-build'


Suggested change

echo 'Trigger docker-build'

echo 'Triggering docker-build'

gaiksaya · 2023-08-01T20:55:41Z

vars/patchDockerImage.groovy

+            ]
+        }
+
+        echo 'Trigger docker-copy with tag build date '


Suggested change

echo 'Trigger docker-copy with tag build date '

echo 'Triggering docker-copy with tag build date '

gaiksaya · 2023-08-01T20:55:57Z

vars/patchDockerImage.groovy

+            }
+        }
+
+        echo "Trigger docker-scan for ${args.product} version ${inputManifest.build.version}${build_qualifier}"


Suggested change

echo "Trigger docker-scan for ${args.product} version ${inputManifest.build.version}${build_qualifier}"

echo "Triggering docker-scan for ${args.product} version ${inputManifest.build.version}${build_qualifier}"

gaiksaya · 2023-08-01T20:56:11Z

vars/patchDockerImage.groovy

+            ]
+        }
+
+        echo 'Trigger docker-promote'


Suggested change

echo 'Trigger docker-promote'

echo 'Triggering docker-promote'

gaiksaya · 2023-08-01T21:00:29Z

vars/patchDockerImage.groovy

+        }
+
+        echo 'Trigger docker-promote'
+        if(args.rerelease){


Do we want to do few more tests before pushing to Prod? For example we do not want to push to prod if high CVEs are detected in above docker scan. @bbarani @peterzhuamazon

Also @Divyaasm please add https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile#L440 wait:true for each job that is being triggered. I think default is true by default but better to mention it.

Think we have ignored CVE's presence in this implementation promote the image when Docker-Scan and Docker-Validation(enhancement) are successful . The idea is to re-run the Job every week to align with Base OS Image. @bbarani Correct me if I'm wrong.

I don't think we can do anything about any CVE at this point in time if it didn't get remediated as part of yum update as the fix needs to come from base OS team. My recommendation is to move forward with available fixes and patch the Docker images on regular basis.

peterzhuamazon

Hi @Divyaasm please read the comment here and considering not copy pasting the entire dockerbuild lib if possible.
I would suggest to re-use the lib or modify with new functions as we discussed before.
https://github.com/opensearch-project/opensearch-build-libraries/blob/main/vars/buildDockerImage.groovy

Thanks.

peterzhuamazon · 2023-08-07T17:28:52Z

vars/patchDockerImage.groovy

+    sh"""
+    #!/bin/bash


Use this:

sh """#!/bin/bash

Your original implementation will not force bash and will default back to sh.

vars/patchDockerImage.groovy

peterzhuamazon · 2023-08-07T17:30:09Z

vars/patchDockerImage.groovy

+
+    docker pull ${docker_image}
+    """
+    sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.version"}}' ${docker_image} > versionnumber"""


Use sh("") if you are trying to do one liner unless you have other reasons.

peterzhuamazon · 2023-08-07T17:30:28Z

vars/patchDockerImage.groovy

+    sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.build-date"}}' ${docker_image} > time"""
+
+    sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.description"}}' ${docker_image} > number"""


Why are we keep doing one liners here?

peterzhuamazon · 2023-08-07T17:31:37Z

vars/patchDockerImage.groovy

+
+    artifactUrlARM64 = "https://ci.opensearch.org/ci/dbc/distribution-build-${args.product}/${version}/${build_number}/linux/arm64/tar/dist/${args.product}/${args.product}-${version}-linux-arm64.tar.gz"
+
+    /*slice the time to get date value*/


vars/patchDockerImage.groovy

Signed-off-by: Divya Madala <[email protected]>

peterzhuamazon

Hi @Divyaasm this is much better and I add some comments on this.
Please take a look when you have time.

Thanks.

vars/buildDockerImage.groovy

vars/patchDockerImage.groovy

Signed-off-by: Divya Madala <[email protected]>

peterzhuamazon

Hi @Divyaasm please check more comments here.

Thanks.

peterzhuamazon · 2023-08-15T19:51:06Z

vars/buildDockerImage.groovy

@@ -6,11 +6,24 @@
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */
+/**
+@param Map[inputManifest] <Required> - path to Input Manifest.


Capital path.

vars/patchDockerImage.groovy

peterzhuamazon · 2023-08-15T19:55:21Z

vars/patchDockerImage.groovy

+    docker inspect --format '{{ index .Config.Labels "org.label-schema.version"}}' ${docker_image} > versionNumber
+    docker inspect --format '{{ index .Config.Labels "org.label-schema.build-date"}}' ${docker_image} > time
+    docker inspect --format '{{ index .Config.Labels "org.label-schema.description"}}' ${docker_image} > buildNumber
+    docker inspect --format '{{ index .Config.Labels "org.label-schema.version"}}' ${latest_docker_image} > latestVersionNumber


I didnt realize you are creating files here.
You can simply use this to save all the values?

GIT_COMMIT_EMAIL = sh ( script: 'git --no-pager show -s --format=\'%ae\'', returnStdout: true ).trim()

Possibly I miss it in the initial review?

Thanks.

Signed-off-by: Divya Madala <[email protected]>

peterzhuamazon

Great work, thanks.

Signed-off-by: Divya Madala <[email protected]>

Signed-off-by: Divya Madala <[email protected]> Signed-off-by: Divya Madala <[email protected]> (cherry picked from commit be4e5cd) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Divyaasm requested review from peterzhuamazon, prudhvigodithi, gaiksaya and rishabh6788 as code owners August 1, 2023 19:11

Divyaasm force-pushed the dockerpatch branch 2 times, most recently from c8c6f6e to 9aa11ae Compare August 1, 2023 19:40

Add Docker Patch Library

19d22b8

Signed-off-by: Divya Madala <[email protected]>

Divyaasm force-pushed the dockerpatch branch from 9aa11ae to 19d22b8 Compare August 1, 2023 19:55

gaiksaya requested changes Aug 1, 2023

View reviewed changes

peterzhuamazon requested changes Aug 7, 2023

View reviewed changes

Update Packer library

edaccb5

Signed-off-by: Divya Madala <[email protected]>

Divyaasm force-pushed the dockerpatch branch from 0db1c96 to edaccb5 Compare August 14, 2023 19:46

peterzhuamazon reviewed Aug 14, 2023

View reviewed changes

Resolve suggstions

5aa3b07

Signed-off-by: Divya Madala <[email protected]>

peterzhuamazon reviewed Aug 15, 2023

View reviewed changes

UPdatedocker patch library

3657264

Signed-off-by: Divya Madala <[email protected]>

peterzhuamazon reviewed Aug 18, 2023

View reviewed changes

peterzhuamazon approved these changes Aug 18, 2023

View reviewed changes

gaiksaya approved these changes Aug 18, 2023

View reviewed changes

Divyaasm and others added 2 commits August 18, 2023 11:58

Bump version to 5.6.0

8e70251

Signed-off-by: Divya Madala <[email protected]>

Merge branch 'main' into dockerpatch

b339016

Signed-off-by: Divya Madala <[email protected]>

gaiksaya merged commit be4e5cd into opensearch-project:main Aug 18, 2023
6 of 7 checks passed

gaiksaya added the backport 5.x label Aug 18, 2023

opensearch-trigger-bot bot mentioned this pull request Aug 18, 2023

[Backport 5.x] Docker Patch Library to support Docker re-release Automation #282

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker Patch Library to support Docker re-release Automation #268

Docker Patch Library to support Docker re-release Automation #268

Divyaasm commented Aug 1, 2023

codecov bot commented Aug 1, 2023 •

edited

Loading

gaiksaya Aug 1, 2023

Divyaasm Aug 1, 2023 •

edited

Loading

gaiksaya Aug 1, 2023 •

edited

Loading

gaiksaya Aug 1, 2023

peterzhuamazon Aug 7, 2023

gaiksaya Aug 1, 2023

gaiksaya Aug 1, 2023

Divyaasm Aug 1, 2023

gaiksaya Aug 1, 2023

gaiksaya Aug 1, 2023

gaiksaya Aug 1, 2023

gaiksaya Aug 1, 2023

gaiksaya Aug 1, 2023

Divyaasm Aug 1, 2023

bbarani Aug 1, 2023 •

edited

Loading

peterzhuamazon left a comment •

edited

Loading

peterzhuamazon Aug 7, 2023

peterzhuamazon Aug 7, 2023

peterzhuamazon Aug 7, 2023

peterzhuamazon Aug 7, 2023

peterzhuamazon left a comment

peterzhuamazon left a comment

peterzhuamazon Aug 15, 2023

peterzhuamazon Aug 15, 2023

peterzhuamazon left a comment

	sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.version"}}' ${docker_image} > versionnumber"""
	sh """docker inspect --format '{{ index .Config.Labels "org.label-schema.version"}}' ${docker_image} > versionNumber"""

	echo 'Trigger docker-copy with tag build date '
	echo 'Triggering docker-copy with tag build date '

	echo "Trigger docker-scan for ${args.product} version ${inputManifest.build.version}${build_qualifier}"
	echo "Triggering docker-scan for ${args.product} version ${inputManifest.build.version}${build_qualifier}"

	echo 'Trigger docker-promote'
	echo 'Triggering docker-promote'


		artifactUrlARM64 = "https://ci.opensearch.org/ci/dbc/distribution-build-${args.product}/${version}/${build_number}/linux/arm64/tar/dist/${args.product}/${args.product}-${version}-linux-arm64.tar.gz"

		/slice the time to get date value/

Docker Patch Library to support Docker re-release Automation #268

Docker Patch Library to support Docker re-release Automation #268

Conversation

Divyaasm commented Aug 1, 2023

Description

codecov bot commented Aug 1, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Divyaasm Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

gaiksaya Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bbarani Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

peterzhuamazon left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peterzhuamazon left a comment

Choose a reason for hiding this comment

peterzhuamazon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peterzhuamazon left a comment

Choose a reason for hiding this comment

codecov bot commented Aug 1, 2023 •

edited

Loading

Divyaasm Aug 1, 2023 •

edited

Loading

gaiksaya Aug 1, 2023 •

edited

Loading

bbarani Aug 1, 2023 •

edited

Loading

peterzhuamazon left a comment •

edited

Loading