🗺️ Haystask Instrumentation

[mikeldking]

Instrumentation for Haystack

• create a tutorial for haystack #826
• add documentation on arize-phoenix #887
• [docs] document integration on haystack https://github.com/deepset-ai/haystack-integrations #822
• document integration on opentelemetry.io #823
• [haystack] support for openinference config #889
• support for supress tracing #882
• support for context attributes #886
• [haystack] [docs] better documentation on how to setup OI instrumentaiton without built-in tracing #814
• [haystack] default to chain span kind and capture IO as input / output semantic attributes #815
• [haystack] tool calling support #824
• [haystack] nix raw embeddings from being captured in retriever inputs #883
• [haystack] instrument router #825
• [BUG][haystack] Output messages is a string rather than a list of messages, causing pheonix to crash - this is using the quickstart #807
• [haystack] repeating components should still be instrumented #884
• [haystack] capture invocation parameters #888
• [haystack] removing dependancy on naming convention #885
• [haystack] add to conda forge #951
• [haystack] ensure haystack is not a runtime dependency #957

[mikeldking]

Unknown span kinds show up for instrumentation

[mikeldking]

Pipeline missing output

[mikeldking]

LLM call doesn't match OTEL syntax for the name:

[mikeldking]

Output messages is a string rather than a list of messages, causing pheonix to crash - this is using the quickstart

[Parker-Stafford]

input to retriever is embedding - might want to opt for the query string if possible here? not sure what the best thing to do is here, running this tutorial https://haystack.deepset.ai/tutorials/27_first_rag_pipeline

[Parker-Stafford]

Pipeline missing output

also happening on this tutorial https://haystack.deepset.ai/tutorials/27_first_rag_pipeline

[Parker-Stafford]

getting llmAttributes[LLMAttributePostfixes.output_messages]?.map is not a function this means we're expecting a list but getting something else when clicking on the llm span this tutorial:

https://haystack.deepset.ai/tutorials/27_first_rag_pipeline

[Parker-Stafford]

this might be correct, not sure how haystack pipelines work but there is no hierarchy in the spans here, typically i feel like there is a little bit of nesting here

[Jgilhuly]

Function / tool calling spans don't appear in Phoenix. Here's the trace for an invocation of the function calling tutorial at https://haystack.deepset.ai/tutorials/40_building_chat_application_with_function_calling

Need to add support for tool calling

[Parker-Stafford]

not sure what the best solution is here but this might need some work, there are no prompt template variables in the prompt_template block of attributes, there are retrieval documents though which do seem to be passed in to the template.
I would expect to see {{question}} in the vars at least and maybe the documents

[mikeldking]

Haystack component run shows up as leaf nodes (looks like it's OTEL).

[cjunkin]

opening OpenAIGenerator span causes error

[cjunkin]

Not sure if this is the desired formatting

[axiomofjoy]

• unknown span kinds
• clicking through on OpenAIGenerator results in C.map is not a function error
• missing outputs in spans
• ConditionalRouter and SerpDevWebSearch have no attributes

[cjunkin]

Empty/null attributes

[Parker-Stafford]

Query is lost in final llm call when running this example : https://docs.haystack.deepset.ai/docs/lostinthemiddleranker#in-a-pipeline

Templating is a little off but this is a larger product / design issue see screen shot (no variables / template but variables and template in the Prompt template tab)

Small note this reranker doesn't require a query because it assumes the documents are already ordered by relevance and just takes those documents and places the most important (first) ones at the edges of the context - this makes the ui look a little weird with the empty query

[mikeldking]

@axiomofjoy conda forge

[mikeldking]

Only thing remaining is opentelemetry.io closing