[Spike] Investigate whether Kedro-Viz can display information from other open source Experiment Tracking APIs

[rashidakanchwala]

Description

Kedro-Viz currently supports experiment tracking through the Kedro's MetricsDataset or JSONDataset and it also uses Kedro sessions. However, with this we offer limited functionality to the user and this could possible be made better by replacing the backend to connect to more specialised/mainstream experiment tracking services such as MLFLow, W&B

The frontend for experiment tracking in Kedro-Viz is already in place, so the goal is to retain this while modifying the backend to fetch data from APIs of popular experiment tracking services.

Proposed Change:

• Keep the current frontend intact (Kedro-Viz UI for experiment tracking).
• Replace the backend to fetch experiment data (metrics, artifacts, etc.) from popular APIs that specialize in experiment tracking.
• The new backend will serve data from these APIs to the frontend.

Benefits:

• Utilising widely-used experiment tracking services would provide better features.
• Users will gain enhanced capabilities to store, visualise, and compare experiments.
• The decoupling of experiment tracking from Kedro's internal sessions and data versioning will allow us to evolve these features on Kedro Framework

Possible work:

• Backend development to integrate with these popular experiment tracking APIs.
• Minimal or no changes required on the frontend. The front-end would be read only.

Next Steps:

• Briefly assess the pain points of other experiment tracking UIs and confirm that the Kedro-Viz UI provides a better user experience.
• Assess the feasibility of integrating with popular experiment tracking APIs.
• Define the structure of the API calls to fetch experiment data.
• Implement the changes and ensure backward compatibility with the current setup.

Checklist

• Include labels so that we can categorise your feature request**

[rashidakanchwala]

@merelcht , @noklam , @astrojuanlu FYI.

[noklam]

Benefits:

• Utilising widely-used experiment tracking services would provide better features.
• Users will gain enhanced capabilities to store, visualise, and compare experiments.
• The decoupling of experiment tracking from Kedro's internal sessions and data versioning will allow us to evolve these features on Kedro Framework

My main question will be, what is the benefit of using kedro-viz instead of use mlflow for experiment tracking directly? Is the main value about providing an alternative UI for these toolings?

[astrojuanlu]

A couple of quick thoughts:

• We should do Make experiment tracking optional #2079 first and let the dust settle a bit
• Before embarking on this task, we should probably do some research (or look up what has been said) about current pain points with MLflow, and specifically its UI

[merelcht]

What would #2079 involve? Just separating the dependencies or also removing stuff from the UI? Do we have any sort of estimation of that work? I'm just wondering if it's worth the effort. Do we have any evidence that people don't use Viz because this feature exists, or that people find Viz is "too large" because of experiment tracking? Otherwise there won't be much user gain by just making it optional compared to having it there by default.

[astrojuanlu]

(Continuing that conversation there)

[rashidakanchwala]

My main question will be, what is the benefit of using kedro-viz instead of use mlflow for experiment tracking directly? Is the main value about providing an alternative UI for these toolings?

@noklam great question, I'm making two assumptions that we should evaluate:

1. The UI for MLflow and W&B is not as intuitive or effective, as several users have pointed out.
2. Kedro-Viz's experiment tracking has undergone extensive research and thoughtful design, with features like the timeline view and parallel coordinates, which may offer a better user experience.

Before embarking on this task, we should probably do some research (or look up what has been said) about current pain points with MLflow, and specifically its UI

For sure, I will add that as well to the ticket.

Added this point

Next Steps:

• Briefly assess the pain points of other experiment tracking UIs and confirm that the Kedro-Viz UI provides a better user experience.
  ... and the rest