[WIP] What: Cloud Native Guide Posts Issue #961 #997
Conversation
Why: * Capture milestones from successful projects * provide guidance and recommendations to projects seeking graduation * Provide criteria to guide TOC on determining when sandbox projects should depart from sandbox This change addresses the need by: * Modifying archiving.md to reflect Sandbox Departure link * Modify sandbox.md to set Sandbox Departure criteria * Modify README.md to capture Milestones link * create project milestones for community review, improvement, and feedback Signed-off-by: Emily Fox <[email protected]>
process/project_milestones.md
Outdated
| * A roadmap clearly defines initiatives that align with releases | ||
| * Releases begin to have establish a pattern of regularity | ||
| * Project has engaged a Technical Advisory Group for feedback on some portion of the project relevant to the domain | ||
| * Project has a clearly discoverable governance doc that covers the basics of decision making to include the commit bit |
There was a problem hiding this comment.
| * Project has a clearly discoverable governance doc that covers the basics of decision making to include the commit bit | |
| * Project has a clearly discoverable governance doc that covers the basics of decision making and how to earn the commit bit |
There was a problem hiding this comment.
"the commit bit" is probably not a well understood term. I don't know if the CNCF has a defined term ("maintainer" is definitely overloaded), but I think "how to earn permission to approve pull requests" or something similar would help.
(I assume the SCM for CNCF projects does not have to include GitHub and a more inclusive terms than Pull Requests might be appropriate)
There was a problem hiding this comment.
I've been thinking about Roadmaps. In the past I've been in favour of them as a requirement but I think I'm changing my mind because I'm not convinced they are as valuable as we'd like to think.
Background / food for thought: the Linux kernel does not have a published roadmap. Everyone involved knows what's going through mailing list discussions.
When we say things like a roadmap that's aligned with releases, combined with a regular cadence of releases, this feels dangerously close to requiring open source projects to make commitments that end users might come to think they can rely on.
And it doesn't seem very aligned with the ethos of open source to require developers to sign up to fixed commitments. Sure, in our community most people are being paid to work on projects but not everyone is. And if someone from Company A makes a commitment to deliver a particular feature in a certain release, but then Company A decides they should spend their time elsewhere, who from the project is supposed to pick up the slack to make sure the commitment is reached?
It might work for very large projects, but the more I think about it, the more it seems like a recipe for burnout amongst contributors, and disappointment amongst end users.
Projects generally use issues to communicate what features and fixes people are working on. A feature that meets some set of criteria (up to the project) makes it into the next release. Isn't that sufficient?
And if we require more, isn't that rather overstepping the idea that we don't impose any particular governance on projects?
There was a problem hiding this comment.
+100 - we really need to stop turning CNCF into a giant bureaucracy that treats all projects the same. This kills innovation stone dead and will put so many people off. Also, please stop imposing requirements on project maintainers that add a lot of work and will lead to burnout, without a very strong compensating reason and balance
There was a problem hiding this comment.
@lizrice Thanks for bringing this up, I dont disagree. We currently have roadmaps integrated through various portions of the lifecycle so we'll need to evaluate what the value is of these in both understanding the project's longer goals as well as the direction they are heading, however they are always point in time for evaluations along the lifecycle.
Your point on perceived commitment is spot on as well, and I'm not confident "messaging" here will resolve this disconnect. We'll need to consider more about how we can view the direction of the project at an evaluation, without dictating how the project expresses that.
There was a problem hiding this comment.
@monadic Thank you for the feedback. This PR on milestones does not establish new or additional requirements on projects, it is designed to serve as guiding points along their [project's] journey — from sandbox to beyond graduation which they may leverage. However, as @craigbox mentioned later, these may be seen as requirements. This is something will need to rework in this PR to avoid.
Regarding burnout, agreed. The CNCF lifecycle is structured to evaluate projects for stability, maturity, and adoption potential (to varying degrees) over the years, as we've learned more about project success, we're beginning to distill steps projects have taken along their path to success. Not as requirements, more as a catalog of what has and hasnt worked and more context around why.
Would providing more reasoning/clarity/what-it-does assist with the milestones? Such that as a project maintainer looking to move to incubation from sandbox (for instance) has ample information about how these may be beneficial to decide if documenting a roadmap is worthwhile for their project, or perhaps including a vision/direction, or neither, works for their needs?
There was a problem hiding this comment.
I've suggested different language for this point.
|
Preparing for graduation, we (Cilium) have been getting recommendations and requirements to do a lot of security-related audits and assessments:
They’re all good things on their own, but taken together they’re repetitive and overlapping, and are causing quite a lot of frustration ("we're drowning in checklists and paperwork".) It would be really helpful to have a clear steer on which of these are actually required. |
|
There's a lot here we can get better clarity on. The TAG Security Review/Assessment process was heavily reworked a few years ago to structure it such that projects may complete portions of the overall "Security Package" aligned with where they are at in the CNCF "lifecycle". This revision was designed to prevent the security review (which often doesn't have enough reviewers and as a result does take longer) from becoming a bottleneck for a project pursuing graduation. If projects completed the "self-assessment" during Sandbox, then optional "joint-review" during Incubation (well before considering graduation) these two security documents form a security package that can be leveraged by Security Auditors to assist in the formal audit. The joint-review builds on top of the self-assessment, but both documents do NOT replace a Security Audit, while joint-reviews do have some areas of overlap with Security Audits, the Security Audits go more in depth in other areas. They should be viewed as cumulatively complementary to one another, however a joint-review is NOT a requirement for graduation (nor is the self-assessment for incubation). I am not comfortable placing TAG Security and the joint-reviews in a position where all projects come to them b/c they are now the executors of a graduation requirement. It's not feasible with the amount of projects in the landscape and the amount of security reviewers we have. There as a been a lot of evolution in open source and supply chain security these past few years, with increasing maturity of various assessments that look at different portions of a project - from its architecture to its repository configuration and release processes. We've not yet revisited how these changes should be weighed or considered in the CNCF lifecycle and I believe this is worth pursuing by community through the Security TAG. |
Agreed |
|
Some practical things that I think would be nice to include are:
|
|
I had an excellent discussion with a community member yesterday about the amount of security asks of projects and the current throughput of Security TAG in reviews, specifically regarding the joint-reviews. What do others think about this as a milestone? |
|
I would caution that even though it might not be the stated intention, these may well end up interpreted as "unwritten criteria" for moving between maturity levels, especially in light of #992. |
@tomkerkhove this would be best in the graduation criteria/process documentation as this was a recent change. @amye is this something you can ensure is integrated?
Completely agree this is time consuming and inconsistent. @amye whatever updates we make for this should reduce duplicative effort. We the template for graduation with a lot of information in it, but perhaps its worth figuring out where this goes in a DD or a DD refresh versus initiating a process. |
Why: * Many community members have added or clarified milestones and intent. This change addresses the need by: * Inclusion of @xmulligan 's suggestions * tweaks to wording based on @monadic 's feedback * adjustments per @lizrice 's comments * Adjustments per @tomkerkhove 's comments * adding milestones per @raravena80 's feedback * include @craigbox 's suggestions Signed-off-by: Emily Fox <[email protected]>
Why: * Roadmap isnt a commitment by maintainers to do work, its subject to change This change addresses the need by: * providing explanation/examples on 'long term planning' Signed-off-by: Emily Fox <[email protected]>
|
Per discussion on today's TOC/TAG Call here is a google doc of the primary changes for better discussion/facilitation of this PR: https://docs.google.com/document/d/1aL0mxInyz-qtRT2QM7kdr9riM36uihILtgM4TR35cPQ/edit?usp=sharing |
Why: * received more community feedback This change addresses the need by: * resolves @leecalcote & @jberkus comments Signed-off-by: Emily Fox <[email protected]>
Why: * additional milestones for incubation suggested This change addresses the need by: * including suggested milestones from @raravena80 (with adjustment) Signed-off-by: Emily Fox <[email protected]>
Why: * removed per feedback and now in separate PR This change addresses the need by: * removed second sentence from line 38 in sandbox.md Signed-off-by: Emily Fox <[email protected]>
process/project_milestones.md
Outdated
| @@ -57,6 +55,9 @@ Incubation milestones are guiding points to highlight success areas a maturing p | |||
| * Project has engaged with a few Technical Advisory Groups to ensure they are considering best practices across multiple technical and governance facets | |||
| * Project has defined a process for managing the conduct of its community, either establishing its own group to do this, leveraging the existing leadership, or deferring to the CNCF | |||
| * Project maintains a dependency graph so adopters fully understand the complexity and risk of use | |||
| * Project has 3 or more adopters using the project in production to support completion of Due Diligence and adopter interviews | |||
| * Project plans activities to increase project adoption trajectory 3 adopters | |||
There was a problem hiding this comment.
| * Project plans activities to increase project adoption trajectory 3 adopters | |
| * Project plans activities to increase project adoption |
process/project_milestones.md
Outdated
| @@ -57,6 +55,9 @@ Incubation milestones are guiding points to highlight success areas a maturing p | |||
| * Project has engaged with a few Technical Advisory Groups to ensure they are considering best practices across multiple technical and governance facets | |||
| * Project has defined a process for managing the conduct of its community, either establishing its own group to do this, leveraging the existing leadership, or deferring to the CNCF | |||
| * Project maintains a dependency graph so adopters fully understand the complexity and risk of use | |||
| * Project has 3 or more adopters using the project in production to support completion of Due Diligence and adopter interviews | |||
| * Project plans activities to increase project adoption trajectory 3 adopters | |||
| * Project has more than 5 regular contributors, more than half of which are from different organizations | |||
There was a problem hiding this comment.
This one worries me A LOT. It clearly incentivizes projects and companies to do the wrong thing, in order to meet these "milestones".
For example, fearing that they are being seen to have "too many" contributions going into the open source project, the major backer of a project decides to hold their engineering efforts back into closed-source add-ons. Or use outsourcing companies and partners rather than hire people to work full-time on the project. Or decide not to contribute a project to the CNCF because it's going to be such an uphill struggle for the project to see itself as successful according to these criteria.
None of these outcomes are good for anyone in our community.
(Also, why arbitrarily 5? And what does "regular" mean? We could spend time debating these points - but that is time that could be better spent on actually progressing / supporting projects.)
There was a problem hiding this comment.
Makes sense, I suggested a slightly different variation of this here: https://docs.google.com/document/d/1aL0mxInyz-qtRT2QM7kdr9riM36uihILtgM4TR35cPQ. I think it's hard to determine a fixed number that can work for all the variety of different projects. IMO a more 'open' or 'flexible' language would work better.
There was a problem hiding this comment.
These are good counterpoints @lizrice . We also don't want to have a single entity (company or organization or individual) be the majority decision maker of a projects technical direction, if that entity is acquired or disappears, the project can be left out to dry, and in some case unable to recover because the primary pool of contribution has evaporated.
@raravena80 @lizrice - How do we strike a balance in these milestones — as guiding points projects could leverage to in order to become more resilient, mature, and widely adopted versions — that could help projects avoid the scenarios you present as well as a increase the likelihood of project's survivability with primary entity's complete absence? What actionable goals/ideals/structure/governance could be provided for projects to increase their resilience and survival likelihood in the event a core contributing entity ceases to be or continue?
Ricardo mentioned open and flexible language here, would the following adjustments meet the intent?
| * Project has more than 5 regular contributors, more than half of which are from different organizations | |
| * Project has multiple contributors who regularly participate in the growth and development of the project in accordance with the projects synchronous and asynchronous cadence | |
| * Project contribution governance defines timeliness and methods for recovering roles and permissions of contributors that drop from regular participation as _defined by the project_ (roles such as issue triage, community management, PR reviewer, maintainer, release lead, etc. and regularity such as participation in two of three releases, facilitation of 4 community meetups annually, etc.) |
There was a problem hiding this comment.
I can't say I'm keen on removing specifics from this. Folks in the community manager role for a project like having clear targets. Once a project gets to applying Graduated, if they don't have at least five regular contributors, the application isn't likely to go well.
Note that these are milestones, not updates to the specific requirements.
There was a problem hiding this comment.
👍 I believe all these can be covered in the March 21st TOC meeting.
|
Hi, I say this with care and respect for the intentions of the people on this thread. If the CNCF wants to continue to be a home for innovative and leading projects, then it needs to listen to people who work on them and make them successful. In my view, the kind of rules being proposed here will greatly narrow what sorts of projects can thrive in CNCF and make it ultimately unappealing. Please can we arrange a TOC call with appropriate representation to discuss. -alexis |
|
@monadic thank you for engaging, we can certainly schedule this for a discussion on a TOC call! It is apparent that the guiding points identified here are still being viewed as rules, requirements, or other forms of constraint which is not what they should be used for. I would love to hear more about your experience to see which, if any, of these describe the path for moving levels of the projects you've been involved in. Scaling that experience and making it approachable for other projects — if they would like to leverage it — would be beneficial for new projects and maybe even existing projects seeking to grow and move levels. When we have a date, I'll update here. Thank you again! |
|
thanks @TheFoxAtWork this is something we spent a lot of time thinking about when creating CNCF - what makes a successful OSS project? the answer is that they come from different places but a consistent theme is small companies innovating ** and sustaining ** them. this has numerous consequences. I love multipolar projects too. |
|
Looks like this actually already on the Agenda for the meeting next week (march 21st) @monadic (apologies for missing it earlier) |
|
8am PT? please can you advertise on toc & alums email lists |
The points under discussion today are being discussed as a criteria for graduation in #992. I've commented at length there. (I would love to be involved in an in-person discussion, but not enough to get up at 4am I'm afraid.) |
|
I have some concrete ideas for this milestones document, to de-couple the advice on growing a project from the TOC’s assessment of its maturity.
I’m aiming to be there for the discussion coming up in this week’s TOC meeting - looking forward to it.
As someone with experience on both the TOC and in community management for Cilium, I can tell you that no-one wants yet more checklists where it’s unclear how important they are - our experience with the myriad security assessments advised to us during the course of our graduation application is an example. If something looks like it might be considered, projects feel obliged to document them in their DD just in case it affects their application. And (sadly) people with some axe to grind will bring up the lack of some specific “milestone” during the public comment period. Neither of these outcomes are remotely helpful to the project or the user community - they just create additional work for the projects and for the TOC.
Most Sandbox projects have at least five regular contributors, so this number isn’t helpful as a target for graduation. What’s the right number? For massive (Kubernetes-esque) projects it’s probably one or two orders higher than this; for a small but critical security library, five might be totally appropriate. There is no “one size fits all” set of measurements that is appropriate for the needs of all projects. That’s why levelling assessments can’t be boiled down to a set of checkboxes, and it’s why previous incarnations of the TOC have deliberately resisted setting arbitrary thresholds, in favour of exercising judgement on a per-project basis. |
|
I'll also mention a few more boundary cases so they are on everyone's radar. Specifically, that it's hard to assess projects which are more specification focused given guidelines which are more often to assess an implementation. For example, when considering a specification project, should you count maintainers / approvers for the specification as those controlling entities? Should this also include the implementation maintainers? A healthy rate of churn for a successful and active specification may only involve a few changes a year. What if there are different compliant implementations from different groups that individually do not have a high degree of maintainer diversity? Would this be considered reasonable diversity, given one can always pick which group's implementation to use? |
Why: * Community provided tangible improvements that have high usability for projects This change addresses the need by: * Changing milestones to guide posts * removing references to moving levels * removing indicators of TOC consideration Signed-off-by: Emily Fox <[email protected]>
TheFoxAtWork
changed the title
Signed-off-by: Emily Fox <[email protected]>
|
Checking in on this PR to determine if any further changes suggested have not yet been addressed. Please take a few moments to review and determine if I've incorporated your changes. Thank you all in advance |
TheFoxAtWork
changed the title
TheFoxAtWork
changed the title
|
PR needs refreshed to align with recent changes in the TOC repo. Will update later. |
riaankleinhans
added
the
process-documentation
Why: * Capture milestones from successful projects provide guidance and recommendations to projects seeking to move levels * previous PR (cncf#997) was very out of date and had unresolvable conflicts due to repo changes Signed-off-by: Emily Fox <[email protected]>
Why:
This change addresses the need by:
Signed-off-by: Emily Fox [email protected]