[DEV TRACKER] Model Serving Requirements for Q4

[heyselbi]

From Req Document

Req 1: Model Storage

Users must be able to to deploy a model stored in (d.) AWS

• feature request: support custom CAs and ca-bundle from openshift Cluster Proxy Config in AWS S3 connectivity e.g. model serving puller modelmesh-serving#113

Req 2: Model Formats - Estimate: RHODS 1.36

Users must be able to serve models based on a variety of framework (a.) OOTB support for TensorFlow, PyTorch, scikit-learn models and (d.) Users must be able to serve models from Hugging Face without having to do any additional conversions or configurations

• OOTB support for HF models #86

Req 7: Deployment Rollouts

a. Ability to deploy new model versions & deploy % of traffic to new version (canary rollout)
b. Ability to do A/B testing on different model versions
c. Ability to test deployed endpoint directly in the product UI

• [STORY] Deployment Rollouts Req #98

Req 10: OOTB Deployed model performance metrics

Users must be able to access performance metrics for all deployed models (e.) CPU/GPU/memory utilization

• [SPIKE] Model Performance Metrics: CPU/GPU/memory utilization #99

Req 14: Model Serving Runtimes

b. OOTB support for Caikit/TGIS

• [STORY] Split caikit/TGIS ServingRuntime into three distinct images caikit#11 - RHODS 1.35
• Caikit/TGIS ADR caikit#13

c. OOTB support for NVIDIA Triton Inference Server

• [P0] [SPIKE] - OOTB support for NVIDIA Triton Inference Server modelmesh-serving#184

Req 15: Remote Deployment

eg. locations other than the cluster where model deployment is initiated (a.) Support models being deployed to remote (location other than where model deployment is initiated)

• [P0] [SPIKE] Remote model deployment modelmesh-serving#185
• [P0] [SPIKE] Caikit or TGIS update should have no impact on customer caikit#12

Req 17: Support options for KServe and/or ModelMesh - Estimate: RHODS 1.36

Support KServe - 1 model per pod or modelmesh - multiple models per pod (a.) RHODS admins should be able to configure whether they want to use KServe (single model serving + additional functionality), ModelMesh, or both

• [SPIKE] Assist UX/UI for KServe/ModelMesh experience modelmesh-serving#193
• Cannot deploy Kserve and Modelmesh together opendatahub-operator#573
• Test ModelMesh Upgrade to v0.11 modelmesh-serving#186

Other planned features

• [STORY] ModelMesh manifests transition readiness for ODH v2 modelmesh-serving#207 - RHODS 1.35
• [STORY] Enable Caikit Python API library caikit#15 - RHODS 1.36
• [SPIKE] storage controller need to support ca cert for self sign signed certificate odh-model-controller#61 - RHODS 1.36

Other planned enhancements

• Update OVMS to v2023.1 modelmesh-serving#137 - RHODS 1.35
• Upgrade MM to v0.11.0 in RHODS + Metrics hotfix modelmesh-serving#228 - RHODS 1.35
• Address High vulnerabilities in OVMS SNYK scan modelmesh-serving#192 - RHODS 1.35

Other planned bug fixes

• Readiness probe failed event observed after the rhods upgrade #77 - RHODS 1.35
• Error on reconciliation of odh-model-controller odh-model-controller#86 - RHODS 1.35

Resources

Model Serving Phase 2 Requirements doc
Model Serving Phase 2 Requirement Mapping spreadsheet

[israel-hdez]

Closing, as we are now tracking work on Jira.

[israel-hdez]

/close

[openshift-ci]

@israel-hdez: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.