Will "Prequant BitsAndBytes models with TP" be supported? #10117
-
|
when seve a BitsAndBytes model with --tensor-parallel-size N |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 3 replies
-
|
Getting same error. Btw, what is PP? |
Beta Was this translation helpful? Give feedback.
-
|
@chenqianfzh What difficulties are there with this support? |
Beta Was this translation helpful? Give feedback.
-
Michael and I had discussed about the lack of support of TP to prequant bnb models, in PR #8434. We agreed PP is a reasonable choice than TP.
|
Beta Was this translation helpful? Give feedback.
-
|
Could you please change the message? It's obvious to me what is tensor parallelizm, but seeing the message I had no idea was is PP vs. TP. This is such a minor change, but will save a lot of people I belive. |
Beta Was this translation helpful? Give feedback.
-
Make sense, I am handling this now |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
Michael and I had discussed about the lack of support of TP to prequant bnb models, in PR #8434. We agreed PP is a reasonable choice than TP.