Added rest_server connector #2
Conversation
There was a problem hiding this comment.
General Suggestions:
It would be easier for me to review that if there was a simple small usage example. Also tests would not hurt since they are easily generated by LLMs.
Custom errors (ChatRESTServerError or smth) instead of ValueErrors would be better, but that requires writing an Exceptions module, which is quite easy to do.
A custom logger module with logs would not hurt too.
There was a problem hiding this comment.
With custom errors I agree, but it is ValueErrors that throws similar langchain classes in these situations. I thought about tests, but they are not possible, for the reason that it is an open library and I can't use any internal LLM url.
Logging tools for Runnables are built into langchain.
Co-authored-by: Jerzy Kamiński <[email protected]>
| messages) | ||
| } | ||
| response = requests.post( | ||
| url=f'{self.base_url}/v1/chat/completions', |
There was a problem hiding this comment.
Some older models use /v1/completions with manual tokens. Also, some newer models provide more control over prompt in raw /v1/completions mode, which could be extremely beneficial. I have no clue how to generalize that, but that's something to keep in mind, because come users may have a custom LLM that operates custom tokens. There are also other modes like tokenization and so on.
There was a problem hiding this comment.
I think it's for the LLM classes in langchain. ChatBaseModel classes assume chat exactly, not custom tokens. Maybe it makes sense to let the developer choose the endpoint, but then I think the whole point of this class is lost. There are too many additional settings and I would choose to write my own class then. But I don't know, it's debatable.
| stop: Optional[List[str]] = None, | ||
| **kwargs: Any, | ||
| ) -> Dict[str, Any]: | ||
| payload = { |
There was a problem hiding this comment.
Max_tokens, temperature and other parameters must be configurable by user, consider adding these fields to that dictionary.
There was a problem hiding this comment.
What we have on our server now does not support configuration of any parameters. When it will be available then I will add and test it.
Langchain connector for a typical REST server