Incompatible with latest exllamav2: too many values to unpack (expected 3)

[davideuler]

When I run the API server, error shows on calling api:

exllamav2-openai-server/server.py", line 156, in _gen_single_token
    token, prob, eos = ExLlamaV2Sampler.sample(logits, gen_settings, self.sequence_ids[:1, :], random.random(), self.tokenizer, prefix_token)
ValueError: too many values to unpack (expected 3)

I fixed it by update the server.py, although I need to hack some of the exllamav2 source code to make the code works together.
vim server.py

        #token, prob, eos = ExLlamaV2Sampler.sample(logits, gen_settings, self.sequence_ids[:1, :], random.random(), self.tokenizer, prefix_token)
        result = ExLlamaV2Sampler.sample(logits, gen_settings, self.sequence_ids[:1, :], random.random(), self.tokenizer, None)
        token, _, _, prob, eos =  result

[bjj]

Yes, I have a PR open with exllamav2 to refactor that code to make it more reusable: turboderp/exllamav2#330 but I don't think there's interest on that end.

I was trying to avoid copying that class into this project, but I will probably end up doing that.

[davideuler]

Hopes the the openai server could work with exllamav2.
It's the only openai compatible api server for exllmav2 so far as I know. Keep going.

[DipperHide]

got similar error

Traceback (most recent call last):
  File "/root/user/exllamav2-openai-server/server.py", line 367, in inference_loop
    chunk, stopped, tokens = item.generator.stream()
  File "/root/miniconda3/envs/user/lib/python3.10/site-packages/exllamav2/generator/streaming.py", line 376, in stream
    chunk, eos, chunk_token_ids, probs, _, _, logits = self._stream()
  File "/root/miniconda3/envs/user/lib/python3.10/site-packages/exllamav2/generator/streaming.py", line 432, in _stream
    next_token, next_ptokens, next_pprobs, next_prob, eos, next_logits = self._gen_single_token(self.settings)
ValueError: not enough values to unpack (expected 6, got 3)

[bjj]

Yes, although requirements.txt says exllamav2>=0.0.11, it actually breaks above about 0.0.14. I'd like to make this work, but without some cooperation from exllamav2 I am stuck.