Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nanoFramework.IoT.Device dependency update process #1565

Open
networkfusion opened this issue Dec 16, 2024 · 6 comments
Open

nanoFramework.IoT.Device dependency update process #1565

networkfusion opened this issue Dec 16, 2024 · 6 comments

Comments

@networkfusion
Copy link
Member

networkfusion commented Dec 16, 2024

Description

All repos that use a multi project solution for nugets (such as M5Stack) candidates and especially
nanoFramework.IoT.Device currently runs nuget dependency updates using GitHub workflows in 5 stages.
In each stage, the projects are stepped through and update sequentially (with optional exclusions).
This takes a very long time (especially stage 4)!

Once all affected projects are updated, this is re-ran by azure-pipelines before PR merge
and again after merge to main/develop.

We need to:

  • work out how we can make this process as parallel as possible.
  • remove duplicated builds.

Potential fixes:

  • Should this be done in the repo (e.g. a single workflow yaml for each solution/project)?
  • Should nanodu be able to parallelize the builds?
  • Should dependency updates via GH workflow require the azure-pipeline check?
  • Should we run the update checks on a Monday and Wednesday, rather than a Wednesday and Friday (when more core members are available)?

Note that there could/should be a catch all workflow to collect and process ones missing.

Which leads to (aims):

  • Reduce build time (and associated costs).
  • Update quicker (daily).
  • Ensures users are not blocked by out of date nugets.
@torbacz
Copy link
Contributor

torbacz commented Dec 16, 2024

I've develop simple tool to keep .net pacakges up to date, because DependaBot is not working with .net well. Should be pretty easy to integrate with NF system. The key advantage is that I'm not using nuget.exe and also I've implemented in memory cache for packages thus it's pretty fast - updating 10 SLN with ~10 project each, took ~2 minutes on azure pipeline. If you are interested, I could extend it with NF and github support.
https://github.com/torbacz/DependencyUpdated

On the other hand, there is ongoing .NET rework in DependaBot, no ETA yet.
dependabot/dependabot-core#10865

@josesimoes
Copy link
Member

With the current workflow, I don't think we can't get past the exisiting "try to update" approach... Unless someone has a good idea on how to figure out if there is something to update. The current mechanism is already checking versions instead of performing the update, which was determined to be slightly quicker.

Having said that, the "obvious" performance upgrade to me seems to be firing parallel processes to tackle each repo.

I don't think there is a way out of the current staged run because of the interdependencies.

As for switching the the run schedule, nothing against it.

@torbacz the reason we went for a github action instead of azdo task was to take advantage of more availability simultaneously slots in github actions compared with limited azdo ones.
Still, haven't looked at your tool. Will do.
Question: anything "obvious" that you have there that could improve nanodu? 😉

@torbacz
Copy link
Contributor

torbacz commented Dec 18, 2024

@josesimoes
The main differences:

  • Nanodu is using nuget.exe tool, my tool is using REST API + XML parser (much faster)
  • I've got InMemory Cachce for packages versions

@Ellerbach
Copy link
Member

I've got InMemory Cachce for packages versions

you still have to build it once. Can we also expose it as a query table element? So anyone who would like to know which versions are related to which ones can have it?

@torbacz
Copy link
Contributor

torbacz commented Dec 18, 2024

That's right. Durring build process, so first one reaches to nuget rest api to all versions and then logic is applied (update policies and others)
https://github.com/torbacz/DependencyUpdated/blob/b98ef8e8ec96159d80fe18a1dbfa5f4be1595578/src/DependencyUpdated.Core/Updater.cs#L175

In case of IoT.Device that would be great improvement - instead of reaching to NuGet server each time, you will have to do it only once per run.

To be honset, I would not expose it. There is already info in nuspec file about it, it can be checked there for each package.

@Ellerbach
Copy link
Member

To be honset, I would not expose it. There is already info in nuspec file about it, it can be checked there for each package.

Yes, very fair!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants