WIP; adding mac support

[dfl]

I began work on adding Mac / MPS support.

bitsandbytes doesn't yet support MPS, but until then the alternative is MLX. I think this PR is close to working, however streaming is not supported by MLX so I haven't finished integration. Maybe someone else can help?

[fblissjr]

so happy to see this, i had the exact same idea and wanted to check beforehand.

i wonder if we're better off just making it an openai spec for the llm. reason being that i'd probably run the SDXL model on a CUDA machine and the LLM on mlx.

[dfl]

@fblissjr hi! yes, that's the path I ended up going down but didn't complete... will push what I have

[dfl]

c.f. https://github.com/qnguyen3/chat-with-mlx/blob/main/chat_with_mlx/app.py

[fblissjr]

sweet! i converted the models here already. only did 8 bit because of the small nature of the models:

• https://huggingface.co/mlx-community/omost-phi-3-mini-128k-mlx-8bit
• https://huggingface.co/mlx-community/omost-dolphin-2.9-llama3-8b-mlx-8bit
• https://huggingface.co/mlx-community/omost-llama-3-8b-mlx-8bit

[dfl]

@fblissjr thanks! are you able to take a crack at finishing up the integration?

Additionally, I see you have a fork of mlx-outlines... seems like outlines could be a useful addition to the project, no? #84 (comment)

[fblissjr]

@fblissjr thanks! are you able to take a crack at finishing up the integration?

Additionally, I see you have a fork of mlx-outlines... seems like outlines could be a useful addition to the project, no? #84 (comment)

I didn't - got bogged down and now going to be out of pocket for a week. Most likely won't get to it unfortunately.

I did start outlines, forked from another fork. It was sorta working, but the project was just moving too fast and the fork too big of a drift that I abandoned it. Mostly was curious how far off it was from 'working' - which I discovered is still too far.

I think we need a proper mlx implementation of a constrained decoder & DSL rather than a retrofit of an older one.

[fblissjr]

@fblissjr thanks! are you able to take a crack at finishing up the integration?

Additionally, I see you have a fork of mlx-outlines... seems like outlines could be a useful addition to the project, no? #84 (comment)

@fblissjr thanks! are you able to take a crack at finishing up the integration?

Additionally, I see you have a fork of mlx-outlines... seems like outlines could be a useful addition to the project, no? #84 (comment)

@dfl just fyi, mlx adding logprobs, which will make stuff like outlines much easier to integrate: ml-explore/mlx-examples#826 (reply in thread)