Prompt_Declaration_Language_python.ipynb fails in nightly CI test

[bjhargrave]

See https://github.com/ibm-granite-community/granite-snack-cookbook/actions/runs/12630547162/job/35190601381

The Prompt_Declaration_Language_python.ipynb is now failing on the nightly CI test with the message:

PDLRuntimeError: Error during 'replicate/ibm-granite/granite-3.0-8b-instruct' model call: litellm.APIError: ReplicateException - can only join an iterable

[bjhargrave]

@deanwampler You worked on this recipe, can you take a look at this failure?

[deanwampler]

I just helped the PDL team in research. Assigning to @ink-pad and @amoghranavade.

[amoghranavade]

Hey Dean, I had worked on Entity Extraction notebook.

[deanwampler]

Sorry, my mistake.

[deanwampler]

A member of the PDL team suggested the following:

I think that the problem is that the container serving the model on Replicate is not running so the call to the model fails. For now, I see two alternatives:

1. Calling the models before running the tests to make sure that the container serving the model is up
2. Adding a fallback in the PDL code to indicate what to do when the model call fails.
   We will investigate if we can do something on the PDL side.

I notice the two notebooks in this folder hard-code the model: replicate/ibm-granite/granite-3.0-8b-instruct. Is it possible that is now obsolete? The runtime check suggested would be a good idea to add, too.

[bjhargrave]

Updating the model to granite 3.1 seems to solve the failure. So I made #103.

[bjhargrave]

Well it solved it locally for me, but the same error still occurs in the CI test

https://github.com/ibm-granite-community/granite-snack-cookbook/actions/runs/12694763150/job/35385274172?pr=103

So there is still some PDL runtime issue.

[bjhargrave]

I debugged this down into litellm Replicate support code.

https://github.com/BerriAI/litellm/blob/7584369fbe328a7794e35c6db365a5289b646227/litellm/llms/replicate/chat/handler.py#L199-L215

Despite the comment which says the result may be status=processing and being in a retry loop, the code never checks for processing and assumes the first response is successful. When it is processing the transform_response method raises an exception.

I will make a PR for litellm.

[bjhargrave]

I also opened IBM/prompt-declaration-language#264 since PDL has hard coded granite-3.0 and thus does not properly handle prompting for granite 3.1 (or later).

[bjhargrave]

I opened litellm issues BerriAI/litellm#7900 and PR BerriAI/litellm#7901 to fix the PDL recipe failures.

[krrishdholakia]

Thanks for the work here @bjhargrave - replied on the LiteLLM PR

[bjhargrave]

The litellm fix was merged into main

BerriAI/litellm@851b0c4