forked from irmen/Tale
-
-
Notifications
You must be signed in to change notification settings - Fork 4
Streaming responses
Rickard Edén edited this page Jan 20, 2024
·
1 revision
The current implementation will also only work for prompts where the response is plain text (as opposed to json, which is more common). The descriptions of locations tend to be the longest, and they are just text.
As of now, it's off by default. To enable, set STREAM = True in backend_kobold_cpp.yaml or backend_llama_cpp.yaml