Integration with Security Scorecard API will provide valuable security insights to Renovate users

[naveensrinivasan]

What would you like Renovate to be able to do?

I am writing to propose the integration of the Security Scorecard API (https://api.securityscorecards.dev/) into Renovate. This API offers a comprehensive view of the security posture of GitHub repositories, including information on vulnerabilities, and other security risks.

Incorporating this API into Renovate would provide users with valuable security insights, allowing them to easily assess the security of their dependencies and take necessary actions to address any issues. As software security becomes an increasingly important concern, the ability to quickly and accurately evaluate the security of projects is crucial.

I believe that the integration of the Security Scorecard API into Renovate would be a valuable addition to the platform and would be well-received by users.

If you have any ideas on how this should be implemented, please tell us here.

https://api.securityscorecards.dev/

Is this a feature you are interested in implementing yourself?

No

[naveensrinivasan]

@laurentsimon @azeemshaikh38 FYI...

[HonkingGoose]

OpenSSF Scorecard only scans projects hosted on GitHub repositories, quote from their README: ¹

The list of projects that are checked is available in the cron/internal/data/projects.csv file in this repository. If you would like us to track more, please feel free to send a Pull Request with others. Currently, this list is derived from projects hosted on GitHub ONLY. We do plan to expand them in near future to account for projects hosted on other source control systems.

We should explain this to Renovate users in the integration. This prevents questions from Renovate users about Scorecard results for other source hosts.

Footnotes

1. https://github.com/ossf/scorecard#public-data ↩

[HonkingGoose]

Summary/TLDR

• Proper badges may shrink down to unreadable size, and are not accessible
• I vote use plain text badges
• Different mockups of options
• Think about whether we want two badges or just a single Merge Confidence badge
• If possible and allowed: integrate OpenSSF Scorecard score into Merge Confidence algorithm and show a single Merge Confidence plain text badge

Badges on Dependency Dashboard problems

I tried getting our Merge Confidence badges working in a few mockups:

• add Merge Confidence badges behind pending upgrades on dashboard whitesource/merge-confidence#17

The problem was that badges rely on the browser viewport size. So when the viewport shrinks, the badges shrink with it. Eventually the badges are too small to read.

I can't reproduce the "badges shrinking down to unreadable"-problem anymore, so maybe it's fixed upstream somewhere (GitHub, Firefox browser). Edit: I can reproduce this when using the GitHub Mobile app. So shrinking badges are still an issue!

Options

1. Show a single badge that includes important metrics like OpenSSF -> only show Merge Confidence badge
2. Show badges for each thing separately -> one badge for OpenSSF, one for overall Merge Confidence
3. Pick "plain text badges" or "actual badge"

To help us see the different options, I've mocked things up. This is all "live" Markdown, so no screenshots! 😄

Mockups for just the OpenSSF Badge

Plain text

• build(deps): update dependency renovate to v34.73.1 | OpenSSF Scorecard: 7.1

Badge

• build(deps): update dependency renovate to v34.73.1

Mockup for OpenSSF Badge + Merge Confidence badge

Plain text

• build(deps): update dependency renovate to v34.73.1 | OpenSSF Scorecard: 7.1 | Merge confidence: neutral

Badge

• build(deps): update dependency renovate to v34.73.1

Mockup for Merge Confidence badge only

Plain text

• build(deps): update dependency renovate to v34.73.1 | Merge confidence: neutral

Badge

• build(deps): update dependency renovate to v34.73.1

My opinion

It's easy to "overload" the Dependency Dashboard or PR body text with too much information/badges.

I like the OpenSSF Scorecard idea, but I'm not sure adding a OpenSSF Scorecard badge is the way to go. If possible (and allowed by OpenSSF) I'd put the OpenSSF Scorecard score into the Merge Confidence badge algorithm. And only display a badge or plain text for the Merge Confidence final score.

I think we should use plain text badges as they always work, and are accessible. "Proper" badges don't work with screenreaders and may shrink down to unreadable sizes. Plus text is easy to debug. 😄

[naveensrinivasan]

OpenSSF Scorecard only scans projects hosted on GitHub repositories, quote from their README: 1

The list of projects that are checked is available in the cron/internal/data/projects.csv file in this repository. If you would like us to track more, please feel free to send a Pull Request with others. Currently, this list is derived from projects hosted on GitHub ONLY. We do plan to expand them in near future to account for projects hosted on other source control systems.

We should explain this to Renovate users in the integration. This prevents questions from Renovate users about Scorecard results for other source hosts.

Footnotes

1. https://github.com/ossf/scorecard#public-data ↩

Also Scorecard Github action gets results published into Scorecard API https://github.com/ossf/scorecard-action

[HonkingGoose]

@rarkins what do you think of integrating Security Scorecard API scores into Renovate's PRs? I think we have two options, both with pros and cons:

1. Add stand-alone badge/plain text for Security Score API result
2. Integrate Scorecard into the Merge Confidence algorithm

Edit: We also have a third option:

3. Close issue and do nothing.

[rarkins]

I'm not sure this is a good idea. Scorecard scores relate to a repo, and therefore 1 or more packages. Our PRs relate to version updates. The scores aren't about the updates, it would be confusing and misleading. Essentially it would be saying this: "Here is a PR which updates dependency foo from 1.0.0 to 1.1.0. FYI security scorecards indicates that foo is not very good".

[HonkingGoose]

Mixing Merge Confidence for the package, and OpenScorecard for the source repository is bound to confuse our users. We could add a note to explain the difference between the score, but then we're cluttering up the badges even more. 😞

I think there's value in knowing that a source repository is not following best practices, so I'm not against the Scorecard idea in general. I don't know of a clean way to integrate it into Renovate though.

[HonkingGoose]

Do you want to close this issue as "not planned" then @rarkins?