Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Showcase conformance images options in the manifest #74808

Merged
merged 2 commits into from
Mar 2, 2019
Merged

Showcase conformance images options in the manifest #74808

merged 2 commits into from
Mar 2, 2019

Conversation

dims
Copy link
Member

@dims dims commented Mar 1, 2019 •
edited
Loading

The Dockerfile has a few more options, let's surface that in the
manifest. Also use the same defaults in the manifest as the Dockerfile
itself.

Change-Id: Ib7419cf7999430db15f39ac414c80ee362fcda76

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
The Dockerfile has a few more options, let's surface that in the
manifest. Also use the same defaults in the manifest as the Dockerfile
itself.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Mar 1, 2019
@dims
Copy link
Member Author

dims commented Mar 1, 2019

/test pull-kubernetes-conformance-image-test

@k8s-ci-robot k8s-ci-robot added sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/release Categorizes an issue or PR as relevant to SIG Release. release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Mar 1, 2019
@dims
Copy link
Member Author

dims commented Mar 1, 2019

/release-note-none

@dims
Copy link
Member Author

dims commented Mar 1, 2019

/priority important-soon

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Mar 1, 2019
@dims
Copy link
Member Author

dims commented Mar 1, 2019

/assign @brahmaroutu @timothysc

@dims
Copy link
Member Author

dims commented Mar 1, 2019

/area conformance

@k8s-ci-robot k8s-ci-robot added the area/conformance Issues or PRs related to kubernetes conformance tests label Mar 1, 2019
@dims dims force-pushed the showcase-the-available-options-in-manifest branch from 7e24960 to ee05682 Compare March 1, 2019 14:52
stevesloka
stevesloka reviewed Mar 1, 2019
View reviewed changes
cluster/images/conformance/conformance-e2e.yaml
- name: E2E_PROVIDER
value: "local"
- name: E2E_PARALLEL
value: "y"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current Dockerfile used a 1 for this arg. Setting to a y would mean we'd pass --nodes=25. Is that desired for the default?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stevesloka yes, i'd like to try this as the default (at least for the CI jobs). sonobuoy can still use the existing defaults.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we can do that seeing as there are [Serial] tests

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also at least on kind, tentative CI results suggest that running the non-serial tests in parallel is indeed flakier (and we've been running both for some time now)

https://testgrid.k8s.io/conformance-kind#kind,%20master%20(dev)%20%5Bnon-serial%5D

johnSchnake
johnSchnake reviewed Mar 1, 2019
View reviewed changes
cluster/images/conformance/conformance-e2e.yaml
value: "Pods should be submitted and removed"
value: "[Conformance]"
- name: E2E_SKIP
value: ""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we deliberately dont skip any tests when running conformance tests in CI?

Sonobuoy currently defaults to skip = 'Alpha|\[(Disruptive|Feature:[^\]]+|Flaky)\]' but I've not really known where that comes from historically. Off the top of my head I"m not sure how many (if any) fall into that category. I think some feature tests do though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we talked about this in kubernetes/test-infra#9104 but i mis-remembered the outcome. will switch up to use the skip list from there:
https://github.com/heptio/sonobuoy/blob/master/pkg/client/mode.go#L42

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@timothysc @spiffxp ^^

@dims
Showcase conformance images options in the manifest
da566ad 
The Dockerfile has a few more options, let's surface that in the
manifest. Also use the same defaults in the manifest as the Dockerfile
itself.

Change-Id: Ib7419cf7999430db15f39ac414c80ee362fcda76
@dims dims force-pushed the showcase-the-available-options-in-manifest branch from ee05682 to da566ad Compare March 1, 2019 21:39
timothysc
timothysc approved these changes Mar 1, 2019
View reviewed changes
Copy link
Member

@timothysc timothysc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 1, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@brahmaroutu
Copy link
Contributor

/lgtm

@spiffxp
Copy link
Member

spiffxp commented Mar 2, 2019 •
edited
Loading

/hold
can we strip out that skip list? if not, what tests need to be addressed? I want to live in a world where the conformance image focuses on conformance and that's it, none of this skipped-but-still-labeled-as-conformance nonsense

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 2, 2019
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 2, 2019
@spiffxp
Copy link
Member

spiffxp commented Mar 2, 2019

I tried dry running e2e.test with a focus on Conformance, with the skip list, and without. The number of tests didn't change. We can inspect runs once this merges to see if there are any that I missed.

Thanks @dims for letting me patch this in.

@spiffxp
Copy link
Member

spiffxp commented Mar 2, 2019

/lgtm
/hold cancel

@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Mar 2, 2019
@johnSchnake johnSchnake mentioned this pull request Mar 2, 2019
Open
@spiffxp spiffxp mentioned this pull request Mar 2, 2019
Closed
@brahmaroutu
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot merged commit 3bb22ed into kubernetes:master Mar 2, 2019
@johnSchnake johnSchnake mentioned this pull request Mar 4, 2019
Closed
6 tasks
@liggitt liggitt added this to the v1.14 milestone Jun 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@johnSchnake johnSchnake johnSchnake left review comments

@stevesloka stevesloka stevesloka left review comments

@BenTheElder BenTheElder BenTheElder left review comments

@spiffxp spiffxp spiffxp left review comments

@timothysc timothysc timothysc approved these changes

Assignees

@spiffxp spiffxp

@timothysc timothysc

@brahmaroutu brahmaroutu

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/conformance Issues or PRs related to kubernetes conformance tests cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note-none Denotes a PR that doesn't merit a release note. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/release Categorizes an issue or PR as relevant to SIG Release. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
v1.14
Development

Successfully merging this pull request may close these issues.
9 participants
@dims @k8s-ci-robot @brahmaroutu @spiffxp @timothysc @BenTheElder @stevesloka @johnSchnake @liggitt