Sagemaker support for inference

[Tarun3679]

Hi, I am trying to test the throughout using vLLMs while inference. I am using amazon sagemaker. My typical notebook example is this one - https://github.com/huggingface/notebooks/blob/5ef609e9078e6248d73f28106e60ddafa9359db1/sagemaker/24_train_bloom_peft_lora/sagemaker-notebook.ipynb . Are there any resources which I can use as reference to deploy an endpoint using Vllm on sagemaker?

[lanking520]

There are ways to do this. Current DeepJavaLibrary support your use case

docker pull deepjavalibrary/djl-serving:deepspeed-nightly

Using this container with

serving.properties

engine=Python
option.model_id=openlm-research/open_llama_7b_v2
option.trust_remote_code=true
option.tensor_parallel_degree=max
option.rolling_batch=vllm
option.max_rolling_batch_size=32

requirements.txt

vllm==0.1.3
pandas

will work for your case. Tested with G5 and P4D instances.

https://github.com/deepjavalibrary/djl-demo/blob/master/aws/sagemaker/large-model-inference/BYOC_template_with_LMI_solution.ipynb

[Tarun3679]

Are we supposed to mention anything in the model.py?

[mariokostelac]

From what I see, https://docs.djl.ai/docs/demos/aws/sagemaker/large-model-inference/sample-llm/vllm_deploy_llama_13b.html contains the tutorial for doing that.

[ronyadgar]

Is there any options to use vLLM on the model.py file ? When I try this I got

RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!

%%writefile models2/model.py
from djl_python import Input, Output
from vllm import LLM
# Check whether CUDA (thus Nvidia GPU) is avaiable

# Define model and tokenizer function variable
client = None

# Model loader function
def load_model():
  global client
  client = LLM("mistralai/Mistral-7B-Instruct-v0.2", trust_remote_code=True, tensor_parallel_size=8)

# Handler function
def handle(input: Input):
  print('handler called', flush=True)

  # Check if input is empty
  if input.is_empty():
      return None

  input = input.get_as_json()
  print("ron input", input)
  input_prompt = str(input.get('prompt', ''))

  if len(input_prompt) < 1:
     return None

  # Load the model
  if client is None:
     load_model()

  output_words = client.generate(payloads, sampling_params=sampling_params)

  # Send result to output
  output = Output()
  output.add(output_words)

  return output

[AnirudhJadhav-toddleapp]

There are ways to do this. Current DeepJavaLibrary support your use case

docker pull deepjavalibrary/djl-serving:deepspeed-nightly

Using this container with

serving.properties

engine=Python
option.model_id=openlm-research/open_llama_7b_v2
option.trust_remote_code=true
option.tensor_parallel_degree=max
option.rolling_batch=vllm
option.max_rolling_batch_size=32

requirements.txt

vllm==0.1.3
pandas

will work for your case. Tested with G5 and P4D instances.

https://github.com/deepjavalibrary/djl-demo/blob/master/aws/sagemaker/large-model-inference/BYOC_template_with_LMI_solution.ipynb

hey @lanking520 , https://github.com/deepjavalibrary/djl-demo/blob/master/aws/sagemaker/large-model-inference/sample-llm/vllm_rollingbatch_deploy_customized_processing.ipynb
i used this library to deploy but it did not work. followed the same steps but i got the following error.
"Error hosting endpoint lmi-model-2024-03-04-10-30-51-503: Failed. Reason: The primary container for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint. "
there were other errors about session.wait_for_endpoint and session.create_endpoint.

[siddvenk]

@AnirudhJadhav-toddleapp

The sample that Qing has shared is quite old at this point. I recommend that you follow our guide here https://github.com/deepjavalibrary/djl-serving/blob/master/serving/docs/lmi/deployment_guide/deploying-your-endpoint.md#option-2-configuration---environment-variables. Replace HF_MODEL_ID with the huggingface model id you are trying deploy.

If you have a custom model, or artifacts stored in s3, we have some details on using sagemaker's support for uncompressed model artifacts here https://github.com/deepjavalibrary/djl-serving/blob/master/serving/docs/lmi/deployment_guide/deploying-your-endpoint.md#option-2-configuration---environment-variables.

Hope this helps.

[geraldstanje]

how can this be run without djl-serving?

can you run vllm/vllm-openai:latest container on aws sagemaker? if not what needs to be changed to make it work?

[JianyuZhan]

Yes, check my vllm-on-sagemaker: https://github.com/JianyuZhan/vllm-on-sagemaker; it runs latest vllm on sagemaker, with a thin fastapi server

[philmod-h]

I'm curious: why do you need a thin fastapi server on top of vLLM?

[vidhig]

When a model is deployed as a sagemaker endpoint using DLJ+vLLM, is it deployed as an openai compatible server or is it following offline inference within the endpoint?