Suppport the Chunking DSL

[orpiske]

Describe the solution you'd like

Camel 4.8 introduced a new EIP (aka Chunking DSL) aimed at integration developers working on integrations for AI applications. The first implementation of this new DSL is based the LangChain4j tokenizer.

Looking at Camel Karavan I noticed it is not yet supported. This could be partially caused by CAMEL-21269 which had a few missing models. With Camel 4.9.0 and 4.8.1, the missing models will be present and tools should be able to generate YAML routes such as:

- route:
    id: route-1737
    from:
      id: from-2457
      uri: timer:template
      parameters:
        period: "1000"
      steps:
        - tokenizer:
            id: tokenizer-3980
            langChain4jCharacterTokenizer:
              maxOverlap: 10
              maxTokens: 1024
              tokenizerType: OPEN_AI

The following tokenizers are currently present:

• langChain4jCharacterTokenizer
• langChain4jParagraphTokenizer
• langChain4jWordTokenizer
• langChain4jLineTokenizer
• langChain4jSentenceTokenizer

Variant

Web Application

Container Management (if applicable)

None

Operating System (for VS Code Extension)

None