1.19.0 fast-forward merge #542
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Random sampler warmup
This change allows to skip empty steps in multistep scenario. We are currently wasting host time on launching n-2 empty steps. This PR removes it. The gain will be visible after device time optimizations, as we are currently limited by HPU calculations inside multistep.
…ntext (#530) Co-authored-by: Konrad Zawora <[email protected]>
…on (#534) This PR is a follow-up to [https://github.com/HabanaAI/vllm-hpu-extension/pull/40](https://github.com/HabanaAI/vllm-hpu-extension/pull/40). It removes all bucketing logic that was moved to vllm-hpu-extension
Limit decode bucket size to num_hpu_blocks
Fixes issue with multi LoRA during `profile_run`.
We are seeing 10% performance regression in the llama-based model due to vllm-project#10239. The mark_step() function needs to be configured differently for each model to achieve the best performance. For some models, mark_step() for every decoder step would be optimal, but for other models, it's better to run it every n-th step. We are adding a counter to only register the hook for every n-th step, which can be configured with VLLM_CONFIG_HIDDEN_LAYERS
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.