* propagate trc bool across vllm init

Signed-off-by: Calvin Woo <[email protected]>
Signed-off-by: calvin d. woo <[email protected]>

* use args directly to avoid undefined var

Signed-off-by: Calvin Woo <[email protected]>
Signed-off-by: calvin d. woo <[email protected]>

* Remove trailing space

Signed-off-by: Dan Sun <[email protected]>
Signed-off-by: calvin d. woo <[email protected]>

* move params to newline

Signed-off-by: calvin d. woo <[email protected]>

---------

Signed-off-by: Calvin Woo <[email protected]>
Signed-off-by: calvin d. woo <[email protected]>
Signed-off-by: Dan Sun <[email protected]>
Co-authored-by: Dan Sun <[email protected]>

The KServe Python SDK README.md uses relative URLs that work well on GitHub but return a 404 error when visited on PyPI.

This change updates the README.md to use absolute URLs that work well on both GitHub and PyPI.

Signed-off-by: kevinbazira <[email protected]>

check empty model final.

Signed-off-by: HAO <[email protected]>
Co-authored-by: koshino17 <[email protected]>

* Fix No model ready error in multi model serving

- Fixes the regression introduced by kserve#3275

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Mark transformer model ready in init method

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Initial implementation of inference client

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add tests

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Use Inference client for e2e tests

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Upgrade pytest-asyncio to 0.23.4

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Fix mutable object initialization in default parameters

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Fix graph e2e tests

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Fix pmml test

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add explain, support dict response, use inference client for internal requests

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Fix inference graph test and grpc headers

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Remove v1 datamodels

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Introduce protocol in client config

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Support inference graph

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

remove logging configs

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Update default timeout to 60 seconds

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add retry config for grpc client

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Fix infer model_name parameter

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add tests for graph endpoint

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

debug

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

fix http client param mismatch

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

skip graph test

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

fix timeout in grpc client

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Fix url construction

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Fix explain

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* configure logger for e2e tests

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Rebase master

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Rebase master

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Fix grpc retry config

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Increase request timeout

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* configure logger for e2e tests

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Rebase master

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Rebase master

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Fix grpc retry config

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Increase request timeout

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Rebase

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Use fixtures for rest client

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix model name not properly parsed by inference graph

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Handle single string arg with excess whitespace

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Handle duplicate arguments

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>
Signed-off-by: Dan Sun <[email protected]>
Co-authored-by: Dan Sun <[email protected]>

empty commit

Signed-off-by: Spolti <[email protected]>

Use add_generation_rompt for chat template

Signed-off-by: Dattu Sharma <[email protected]>

* Deduplicate the names for the additional domain names

Signed-off-by: Vincent Hou <[email protected]>

* Refactoring the functions

Signed-off-by: Vincent Hou <[email protected]>

---------

Signed-off-by: Vincent Hou <[email protected]>

virtual service case insensitive

Signed-off-by: Andrews Arokiam <[email protected]>

* Install packages needed for model load

Signed-off-by: Gavrish Prabhu <[email protected]>

* make all apt get into a single line

Signed-off-by: Gavrish Prabhu <[email protected]>

---------

Signed-off-by: Gavrish Prabhu <[email protected]>

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

…serve#3789)

* Add readiness probe for mlserver in CI

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Increase memory limit for pmml test to prevent OOMKilled and read timeout error

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix logprobs

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix a scenario where stream completion fails if echo is true and logprobs is nil

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix a scenario where completion fails if the prompt is token_ids and echo is set to true

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Respect tokenizer revision

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add workaround for adding None to token_logprobs and top_logprobs

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

agent watcher unit test is always flaky so increase timeout to make it stable

Signed-off-by: jooho lee <[email protected]>

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add tests for vLLM

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* resolve comments

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Uncomment tests for fixed bugs

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

….3 (kserve#3812)

* Upgrade serving runtime python version to 3.11 and debian to bookworm

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Upgrade poetry to 1.8.3

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Upgrade openjdk to 17 for pmml

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix 'AS' casing warning

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix pmml server

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Bump vLLM to 0.5.3.post1

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Update makefile

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* approx probability comparison

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Set multiprocessing method to spawn

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

…se 'spawn' for mutiprocessing (kserve#3757)

* Refactor model server to let uvicorn handle multiple workers

- Refactored the ModelServer to let uvicorn handle multiple workers. This will remove the bottleneck of using 'fork' for multiprocessing

- Make FastAPI app instance easily accessible across the project so that users can easily add middlewares and custom exception handlers for custom models.

- Use uvloop eventpolicy

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add middleware example

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add e2e test

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Remove nest_asyncio in art explainer

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Remove uvloop

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Rebase master

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix python tests

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* revert art explainer

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Remove monkeypatch

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Remove redundant future exception logging

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Signed-off-by: Spolti <[email protected]>

* Make ray serve an optional dependency

Signed-off-by: Curtis Maddalozzo <[email protected]>
Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Unify the log configuration using kserve logger (kserve#3577)

* Configure logging for serving runtimes

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add pyyaml dependency

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* black format

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* fix pyproject.toml

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Rebase master

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* cleanup logger for e2e

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Modify logger format to include func name

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Log model download time.

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Rebase master

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Allow disabling logger configuration and deprecate logger related arg in model server

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Rebase master

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Resolve comments

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* pyyaml=^6.0.0 to fix build failure

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Remove logger related parameters from model server

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* import model_server

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix lint

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix linting

Signed-off-by: Curtis Maddalozzo <[email protected]>
Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Rebase, minor fixes and add e2e test

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Curtis Maddalozzo <[email protected]>
Signed-off-by: Sivanantham Chinnaiyan <[email protected]>
Co-authored-by: Curtis Maddalozzo <[email protected]>
Co-authored-by: Dan Sun <[email protected]>

* Update aif example

chore:	Update aif explainer example.
	- Bump KServer to 0.13.0, it will bring some library updates, plus, it fixes a few security alerts in this example.
	- update the scikit-learn package name

Signed-off-by: Spolti <[email protected]>

* move the local instructions to the README

Signed-off-by: Spolti <[email protected]>

* empty commit

Signed-off-by: Spolti <[email protected]>

---------

Signed-off-by: Spolti <[email protected]>

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

…ve#3737)

These changes introduce the possibility to configure KServe with its own Istio local gateway, to partially decouple KServe from the Knative local gateway.

Typically, it is OK to re-use the already configured Knative local gateway for KServe uses (as long as configs do not conflict). However, there are cases where having a dedicated local gateway for KServe is beneficial. Just to give some examples:
* To have the ability to use strict mTLS in Istio
* To reduce some pressure on the Knative local gateway by having a dedicated gateway deployment (it still would hit Knative gateway, but only once, rather than twice)
* To be able to configure TLS on cluster-local hostnames (Knative support is still experimental)

To have a dedicated Gateway in KServe, similar configurations to Knative are need to be done. At the very least, and if not having a dedicated gateway deployment, a v1/Service and an Istio Gateway resource need to be created for KServe. Such resources would need to be configured in _localGateway_ and _localGatewayService_. KServe still needs to rely on Knative routing for the KSVCs it creates. Thus, after handling an incoming request and resolving its target, it needs to be forwarded to be handled by Knative. This is the reason for introducing a new `knativeLocalGatewayService` in the ConfigMap.

The removed `ingressService` seems to be unused. Apparently, it became unused when the v1alpa1 API of the InferenceServices was deprecated and removed.

Signed-off-by: Edgar Hernández <[email protected]>

* Add support for Azure DNS zone endpoints

Signed-off-by: tjandy98 <[email protected]>

* Add test cases for Azure Blob and File Share URI pattern matching

Signed-off-by: tjandy98 <[email protected]>

* flake8

Signed-off-by: tjandy98 <[email protected]>

* black

Signed-off-by: tjandy98 <[email protected]>

---------

Signed-off-by: tjandy98 <[email protected]>

Signed-off-by: Dan Sun <[email protected]>

* Add logging request feature for vLLM

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add log request feature for huggingface

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Fix snyk image scan failure

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Update ROADMAP.md

Signed-off-by: Dan Sun <[email protected]>

* Add llm gateway

Signed-off-by: Dan Sun <[email protected]>

* Update ROADMAP.md

Signed-off-by: Dan Sun <[email protected]>

* Update ROADMAP.md

Signed-off-by: Dan Sun <[email protected]>

---------

Signed-off-by: Dan Sun <[email protected]>

* Image pull secrets in helm charts

Signed-off-by: [email protected] <[email protected]>

* fix README

Signed-off-by: [email protected] <[email protected]>

---------

Signed-off-by: [email protected] <[email protected]>

* fix for the rolling update strategy

Signed-off-by: Andrews Arokiam <[email protected]>

* codegen fix

Signed-off-by: Andrews Arokiam <[email protected]>

* codegen fix

Signed-off-by: Andrews Arokiam <[email protected]>

* codegen fix

Signed-off-by: Andrews Arokiam <[email protected]>

* go mod fix

Signed-off-by: Andrews Arokiam <[email protected]>

* fix for deployment patch

Signed-off-by: Andrews Arokiam <[email protected]>

* test for rolling update strategy

Signed-off-by: Andrews Arokiam <[email protected]>

---------

Signed-off-by: Andrews Arokiam <[email protected]>

…serve#3886)

fix tokens exceeding model limit error type in vllm server

Signed-off-by: [email protected] <[email protected]>

…rve#3685)

* Add support for binary data extension protocol and FP16 datatype support

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add FP16 validation

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Refactor, support request outputs and add tests

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add back numpy as data support

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add binary data extension support for inference client

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Increase timeout for gprc client

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Rebase master

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Apply suggestions from code review

Signed-off-by: Dan Sun <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>
Signed-off-by: Dan Sun <[email protected]>
Co-authored-by: Dan Sun <[email protected]>

* protobuf version upgrade 4.25.4

Signed-off-by: Andrews Arokiam <[email protected]>

* resolved merge conflicts on poetry

Signed-off-by: Andrews Arokiam <[email protected]>

---------

Signed-off-by: Andrews Arokiam <[email protected]>

* Adds optional labels to the controller

Signed-off-by: Souhaib Guitouni <[email protected]>

* Add annotation and podAnnotation

Signed-off-by: Dan Sun <[email protected]>

* Update helm doc

Signed-off-by: Dan Sun <[email protected]>

---------

Signed-off-by: Souhaib Guitouni <[email protected]>
Signed-off-by: Dan Sun <[email protected]>
Co-authored-by: Souhaib Guitouni <[email protected]>
Co-authored-by: Dan Sun <[email protected]>

…serve#3877)

Signed-off-by: jooho lee <[email protected]>

…re (kserve#3830)

"data": {
    "EventName": "s3:ObjectCreated:Put",
    "Key": "mnist/0.png",
    "Records": [
      {
        "eventVersion": "2.0",
        "eventSource": "minio:s3",
        "awsRegion": "",
        "eventTime": "2024-07-31T06:20:44.731Z",
        "eventName": "s3:ObjectCreated:Put",
        "userIdentity": {
          "principalId": "minio"
        },
        "s3": {
          "s3SchemaVersion": "1.0",
          "configurationId": "Config",
          "bucket": {
            "name": "mnist",
            "ownerIdentity": {
              "principalId": "minio"
            },
            "arn": "arn:aws:s3:::mnist"
          },
          "object": {
            "key": "0.png",
            "size": 324,
            "eTag": "ebed21f6f77b0a64673a3c96b0c623ba",
            "contentType": "image/png",
            "
            ...

Co-authored-by: jongs <[email protected]>

* support text embedding task in huggingfaceserver

Signed-off-by: Kevin Mingtarja <[email protected]>

* fix lint errors

Signed-off-by: Kevin Mingtarja <[email protected]>

* format code

Signed-off-by: Kevin Mingtarja <[email protected]>

* minor refactor

Signed-off-by: Kevin Mingtarja <[email protected]>

* add e2e test

Signed-off-by: Kevin Mingtarja <[email protected]>

* specify task explicitly in e2e test

Signed-off-by: Kevin Mingtarja <[email protected]>

* use approx() for comparing text embedding output

Signed-off-by: Kevin Mingtarja <[email protected]>

* fix output

Signed-off-by: Kevin Mingtarja <[email protected]>

* specify relative tolerance to approx

Signed-off-by: Kevin Mingtarja <[email protected]>

* fix e2e test output

Signed-off-by: Kevin Mingtarja <[email protected]>

* Update to use predict_isvc in test

Signed-off-by: Dan Sun <[email protected]>

* Fix output data assert

Signed-off-by: Dan Sun <[email protected]>

* Add abs toleration

Signed-off-by: Dan Sun <[email protected]>

---------

Signed-off-by: Kevin Mingtarja <[email protected]>
Signed-off-by: Dan Sun <[email protected]>
Co-authored-by: Dan Sun <[email protected]>

…serve#3827)

* Rename max_length parameter to max_model_len to be in sync with vLLM

Signed-off-by: Dattu Sharma <[email protected]>

* Add max_length back which also writes to max_model_len

Signed-off-by: Dattu Sharma <[email protected]>

* Fixup CLI helper message

Signed-off-by: Dattu Sharma <[email protected]>

* Also support max_model_len

Signed-off-by: datta0 <[email protected]>

* Cleanup and refactor to max_model_len

Signed-off-by: datta0 <[email protected]>

---------

Signed-off-by: Dattu Sharma <[email protected]>
Signed-off-by: datta0 <[email protected]>

Updated istio versions based on go version 1.21

Signed-off-by: mholder6 <[email protected]>

* Enrich isvc NotReady events for failed conditions

* adding metadata on requests

Signed-off-by: Gabe Cemaj <[email protected]>

* fix tests

Signed-off-by: Gabe Cemaj <[email protected]>

* switch to allow list system instead of prefix

Signed-off-by: Gabe Cemaj <[email protected]>

* encode meta as json and fix tests

Signed-off-by: Gabe Cemaj <[email protected]>

* expose options top level

Signed-off-by: Gabe Cemaj <[email protected]>

* expose options top level

Signed-off-by: Gabe Cemaj <[email protected]>

* Fix cloud event error handling

Signed-off-by: Dan Sun <[email protected]>

* import missing packages

Signed-off-by: Dan Sun <[email protected]>

* Rename to metadata headers

Signed-off-by: Dan Sun <[email protected]>

* Fix golint

Signed-off-by: Dan Sun <[email protected]>

* Generate for metadata header field

Signed-off-by: Dan Sun <[email protected]>

* fix conflicting go.sun

Signed-off-by: Gabe Cemaj <[email protected]>

---------

Signed-off-by: Gabe Cemaj <[email protected]>
Signed-off-by: Dan Sun <[email protected]>
Co-authored-by: Gabe Cemaj <[email protected]>
Co-authored-by: Dan Sun <[email protected]>

* Update 0.14.0-rc0 release

Signed-off-by: Dan Sun <[email protected]>

* Add security context

Signed-off-by: Dan Sun <[email protected]>

* Update helm doc

Signed-off-by: Dan Sun <[email protected]>

* Update crd

Signed-off-by: Dan Sun <[email protected]>

---------

Signed-off-by: Dan Sun <[email protected]>

Use API tokens for publishing package PyPI

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix sdlc

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add option for only installing deps in quick install

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Increase cpu & memory request for controller

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix 0.14.0-rc0 release

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* add security context and resources to rbac proxy container

Signed-off-by: Gavin Li <[email protected]>

* feedback

Signed-off-by: Gavin Li <[email protected]>

---------

Signed-off-by: Gavin Li <[email protected]>

Remove unwanted secret permissions

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* bump to vllm 0.5.5

Signed-off-by: Lize Cai <[email protected]>

* fix parse_and_batch_prompt import

Signed-off-by: Lize Cai <[email protected]>

---------

Signed-off-by: Lize Cai <[email protected]>

Signed-off-by: Jin Dong <[email protected]>

Signed-off-by: jooho lee <[email protected]>

* Implement health endpoint for vLLM backend

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add openai health endpoint

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

…vingruntimes (kserve#3917)

* Add security context for runtimes

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add security context for runtimes helm

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add security best practices for ig

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Disable service account secret auto mount

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* fmt

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Update test cases

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add user id for tensorflow, triton and torchserve

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Bump go 1.22

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Update generate scripts

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix breaking changes

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix deepcopy, client generation and bump controller-gen to 0.16.2

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* generate files

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Bump go version for CI and dockerfiles

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Bump go version for qpext

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Use slices.Contains

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Remove clientv1alpha1 dir

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix lint

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* go mod tidy

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* set min go version to 1.22.7

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* bump to vllm 0.6.0

Signed-off-by: yxia216 <[email protected]>

* lock

Signed-off-by: yxia216 <[email protected]>

---------

Signed-off-by: yxia216 <[email protected]>

…on (kserve#3885)

* Set the volume mount's readonly annotation based on the ISVC annotation

Signed-off-by: Hannah DeFazio <[email protected]>

* Add test case where readonly is unset, check values

Signed-off-by: Hannah DeFazio <[email protected]>

* Use StorageInitializerVolumeName constant

Signed-off-by: Hannah DeFazio <[email protected]>

* Set the readonly value for the storage-initializer

Signed-off-by: Hannah DeFazio <[email protected]>

* Add tests for direct pvc volume mount use case

Signed-off-by: Hannah DeFazio <[email protected]>

---------

Signed-off-by: Hannah DeFazio <[email protected]>
Co-authored-by: Spolti <[email protected]>

* add /dev/shm volume to hfserver.

Signed-off-by: Lize Cai <[email protected]>

* update helm chart docs

Signed-off-by: Lize Cai <[email protected]>

* add flag to enable devshm.

Signed-off-by: Lize Cai <[email protected]>

---------

Signed-off-by: Lize Cai <[email protected]>

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* new model cache cr

Signed-off-by: Jin Dong <[email protected]>

* update crd

Signed-off-by: Jin Dong <[email protected]>

* Fix genereted python tests

Signed-off-by: Jin Dong <[email protected]>

* Fix test failure

Signed-off-by: Jin Dong <[email protected]>

* Make nodegroup a list field in model cache cr

Signed-off-by: Jin Dong <[email protected]>

* fix lint

Signed-off-by: Jin Dong <[email protected]>

* minor updates to model cache cr

Signed-off-by: Jin Dong <[email protected]>

* Add usecase field to cluster storage container

Signed-off-by: Jin Dong <[email protected]>

* Fix test failures

Signed-off-by: Jin Dong <[email protected]>

* Change variable name

Signed-off-by: Jin Dong <[email protected]>

* Fix lint

Signed-off-by: Jin Dong <[email protected]>

* Fix default storage container cr

Signed-off-by: Jin Dong <[email protected]>

* fix defualt.yaml

Signed-off-by: Jin Dong <[email protected]>

* Remove storagelimit field from node group

Signed-off-by: Jin Dong <[email protected]>

* Fix python code

Signed-off-by: Jin Dong <[email protected]>

* Change some fields

Signed-off-by: Jin Dong <[email protected]>

* Rename crd

Signed-off-by: Jin Dong <[email protected]>

* Fix lint error in python test files

Signed-off-by: Jin Dong <[email protected]>

* Rename CR

Signed-off-by: Jin Dong <[email protected]>

* Add status to local model node group

Signed-off-by: Jin Dong <[email protected]>

* Add missing node status

Signed-off-by: Jin Dong <[email protected]>

* Remove files related to ClusterLocalNodeGroup

Signed-off-by: Jin Dong <[email protected]>

* Add default value for workload type

Signed-off-by: Jin Dong <[email protected]>

* Fix StorageContainerSpec WorkloadType default value

Signed-off-by: Jin Dong <[email protected]>

* nodegroups -> nodegroup

Signed-off-by: Jin Dong <[email protected]>

* Add comments

Signed-off-by: Jin Dong <[email protected]>

* Add back storageLimit

Signed-off-by: Jin Dong <[email protected]>

* Update charts/kserve-crd/templates/serving.kserve.io_clusterstoragecontainers.yaml

Signed-off-by: Jin Dong <[email protected]>

---------

Signed-off-by: Jin Dong <[email protected]>

Signed-off-by: Andrews Arokiam <[email protected]>

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Initial TLS bundle support

Signed-off-by: Rui Vieira <[email protected]>

* Move CA bundle volume and mount path to constants

Signed-off-by: Rui Vieira <[email protected]>

* Rename loggerConfigTls to loggerTLSConfig

Signed-off-by: Rui Vieira <[email protected]>

* Rename TlsCertName to CertName

Signed-off-by: Rui Vieira <[email protected]>

* Add Logger option skip TLS verification

Also:
- Fixed incorrect cert name argument name (`--log-tls-cert` is now `--logger-ca-cert-file`)

Signed-off-by: Rui Vieira <[email protected]>

* Correct case

Change CABundle and CACertfile to caBundle and caCertFile.

Signed-off-by: Rui Vieira <[email protected]>

* Fix linting errors

- Restore newline at the end of charts/kserve-resources/README.md
- Remove import of github.com/kserve/kserve/pkg/constants from `worker.go` and replace with local constant for the CA mount path
- `InsecureSkipVerify: logReq.TlsSkipVerify` was triggering gosec's G402 with "potential 'true' for `logReq.TlsSkipVerify`". Since this value is allowed to be true, this specific line was excluded from the checks and an explanatory comment added
- Remove import of `k8s.io/utils/ptr` and replace with a pointer `&optionalVolume`

Signed-off-by: Rui Vieira <[email protected]>

* Fix import sort order on `worker.go`

Signed-off-by: Rui Vieira <[email protected]>

---------

Signed-off-by: Rui Vieira <[email protected]>

* Fix explainer not working with path based routing

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add test

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add explainer e2e test for path based routing

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Rebase master

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

…serve#3944)

Fix broken ingress test and update go mod

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add HostIPC to ServingRuntimePodSpec

Signed-off-by: Jin Dong <[email protected]>

* Add hostIPC: true to huggingface server yaml

Signed-off-by: Jin Dong <[email protected]>

---------

Signed-off-by: Jin Dong <[email protected]>

Signed-off-by: jooho lee <[email protected]>

…3907)

* update fluid kserve demo to use huggingface servingruntime and other model.

Signed-off-by: Lize Cai <[email protected]>

* fix lint

Signed-off-by: Lize Cai <[email protected]>

* explicitly set custom servingruntime, update devshm.

Signed-off-by: Lize Cai <[email protected]>

* update the args in custom kserve hfserver

Signed-off-by: Lize Cai <[email protected]>

* address comments

Signed-off-by: Lize Cai <[email protected]>

* add return of line

Signed-off-by: Lize Cai <[email protected]>

---------

Signed-off-by: Lize Cai <[email protected]>

* bump to vLLM0.6.1post2

Signed-off-by: yxia216 <[email protected]>

* update kserve/poetry

Signed-off-by: yxia216 <[email protected]>

---------

Signed-off-by: yxia216 <[email protected]>

Add NodeDownloadPending status

Signed-off-by: Jin Dong <[email protected]>

kserve#3954)

Signed-off-by: Gregory Keith <[email protected]>

)

* initial commit for hugging face model download and load

Signed-off-by: Andrews Arokiam <[email protected]>

* bug fix on storage initializer

Signed-off-by: Andrews Arokiam <[email protected]>

* added hf_token and unittests

Signed-off-by: Andrews Arokiam <[email protected]>

* separate hf-storage-initializer image to reduce image size

Signed-off-by: Andrews Arokiam <[email protected]>

* review comment changes

Signed-off-by: Andrews Arokiam <[email protected]>

* snapshot download

Signed-off-by: Andrews Arokiam <[email protected]>

* use existing image for storage initializer

Signed-off-by: Andrews Arokiam <[email protected]>

* resolved merge conflicts

Signed-off-by: Andrews Arokiam <[email protected]>

* added hf storage uri validation

Signed-off-by: Andrews Arokiam <[email protected]>

* resolved merge conflicts

Signed-off-by: Andrews Arokiam <[email protected]>

---------

Signed-off-by: Andrews Arokiam <[email protected]>

Signed-off-by: Dan Sun <[email protected]>

* Consolidate into one commit

Signed-off-by: Jin Dong <[email protected]>

* Fix configmap format

Signed-off-by: Jin Dong <[email protected]>

* Fix configmap

Signed-off-by: Jin Dong <[email protected]>

* Log configmap read error

Signed-off-by: Jin Dong <[email protected]>

* fix naming

Signed-off-by: Dan Sun <[email protected]>

* Update comments

Signed-off-by: Jin Dong <[email protected]>

* Add enabled flag to configmap and avoid cluster resource check in isvc defaulter

Signed-off-by: Jin Dong <[email protected]>

* move client into the local model block

Signed-off-by: Dan Sun <[email protected]>

* Fix lint

Signed-off-by: Jin Dong <[email protected]>

---------

Signed-off-by: Jin Dong <[email protected]>
Signed-off-by: Dan Sun <[email protected]>
Co-authored-by: Dan Sun <[email protected]>

* Sync helm chart with kustomize

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Update manifest generation script to sync helm charts

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Make kserve-addressable-resolver role optional

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Prepare for 0.14.0-rc1 release

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Update release process

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Comment out crd sync script in make

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix helm template syntax

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* add a new API for multi-node/multi-gpu

Signed-off-by: jooho lee <[email protected]>

* fix gitaction

Signed-off-by: jooho lee <[email protected]>

* fix merging conflict

Signed-off-by: jooho lee <[email protected]>

* fix gitaction fail

Signed-off-by: jooho lee <[email protected]>

* regenerate codegen/manifests

Signed-off-by: jooho lee <[email protected]>

* Apply suggestions from code review

Co-authored-by: Dan Sun <[email protected]>
Signed-off-by: Jooho Lee <[email protected]>

* remove unnecessary comment

Signed-off-by: jooho lee <[email protected]>

* change the type of workerSpec in isvc to PodSpec

Signed-off-by: jooho lee <[email protected]>

* update controller-gen version

Signed-off-by: jooho lee <[email protected]>

* remove replicas from workerSpec

Signed-off-by: jooho lee <[email protected]>

* fix conflict merging

Signed-off-by: jooho lee <[email protected]>

* added size(replicas) for workerSpec again

Signed-off-by: jooho lee <[email protected]>

* add WorkerSpec to inferenceService

Signed-off-by: jooho lee <[email protected]>

* fix go linter

Signed-off-by: jooho lee <[email protected]>

---------

Signed-off-by: jooho lee <[email protected]>
Signed-off-by: Jooho Lee <[email protected]>
Signed-off-by: Jooho Lee <[email protected]>
Co-authored-by: Dan Sun <[email protected]>

…#3924)

* fix openapigen.sh that can be executed from kserve dir

Signed-off-by: jooho lee <[email protected]>

* regenerate codegen/manifests

Signed-off-by: jooho lee <[email protected]>

* Update go.sum

Signed-off-by: Dan Sun <[email protected]>

---------

Signed-off-by: jooho lee <[email protected]>
Signed-off-by: Dan Sun <[email protected]>
Co-authored-by: Dan Sun <[email protected]>

* Support python 3.12

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Update dependencies

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Update deps to support 3.12

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Remove python 3.8 support

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Remove skip for infer client test

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix port forward

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix sklearn pandas dep

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* skip pydantic v1 test for py 3.12

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add setuptools dep for pmml

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Fix lgb

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Include setuptools for paddle

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Include setuptools for huggingface

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Rebase

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Rebase

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

---------

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Bump version to 0.13.0-rc0 (kserve#3665)

Signed-off-by: Curtis Maddalozzo <[email protected]>
Signed-off-by: jordanyono <[email protected]>

* fixing docs

Signed-off-by: jordanyono <[email protected]>

* fix spelling mistake

Signed-off-by: jordanyono <[email protected]>

---------

Signed-off-by: Curtis Maddalozzo <[email protected]>
Signed-off-by: jordanyono <[email protected]>
Co-authored-by: Curtis Maddalozzo <[email protected]>

* Fix local testing

Signed-off-by: Dan Sun <[email protected]>

* Fix codegen

Signed-off-by: Dan Sun <[email protected]>

---------

Signed-off-by: Dan Sun <[email protected]>

Signed-off-by: Sivanantham Chinnaiyan <[email protected]>

* Add a flag for automount serviceaccount

Signed-off-by: Jin Dong <[email protected]>

* Set default to false

Signed-off-by: Jin Dong <[email protected]>

* Default to true

Signed-off-by: Jin Dong <[email protected]>

* Fix test error

Signed-off-by: Jin Dong <[email protected]>

* Update openapi generated.go

Signed-off-by: Jin Dong <[email protected]>

* Fix python lint

Signed-off-by: Jin Dong <[email protected]>

* Fix config loading

Signed-off-by: Jin Dong <[email protected]>

---------

Signed-off-by: Jin Dong <[email protected]>

…ainer (kserve#3985)

* Do not set security context on the storage initializer from user container

Signed-off-by: Jin Dong <[email protected]>

* Add securityContext to the default storage container in the helm chart

Signed-off-by: Jin Dong <[email protected]>

---------

Signed-off-by: Jin Dong <[email protected]>

This adds the model container as an init-container to mitigate a race
condition that would happen if the model container is not present on the
cluster-node. The race condition happens if the cluster is able to fetch
and start the runtime container before the modelcar is fetched. This
would lead to the runtime to terminate with error.

By configuring the model container as an init-container the runtime
won't start until the modelcar is fetched. Although there is still the
risk of a race condition when the cluster schedules the runtime
container first, the pod should stabilize after a few restarts of the
runtime container and should either prevent a CrashLoopBackOff event on
the pod, or the crash event would finish quickly.

This improves compatibility with the runtimes which can now stay
agnostic to the modelcar implementation, until better techniques (like
native sidecars, and oci volume mounts) become mature.

Signed-off-by: Edgar Hernández <[email protected]>