Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

Upstream sync 2024 07 01 #350

Merged
merged 113 commits into from
Jul 3, 2024
Merged

Upstream sync 2024 07 01 #350

merged 113 commits into from
Jul 3, 2024

Commits on Jul 1, 2024

  1. [Distributed] Add send and recv helpers (vllm-project#5719)

    @andoorve @robertgshaw2-neuralmagic
    andoorve authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    54348ea View commit details
    Browse the repository at this point in the history

  2. [Bugfix] Add phi3v resize for dynamic shape and fix torchvision requi… 

    …rement (vllm-project#5772)
    @Isotr0py @robertgshaw2-neuralmagic
    Isotr0py authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    5ae4430 View commit details
    Browse the repository at this point in the history

  3. [doc][faq] add warning to download models for every nodes (vllm-proje… 

    …ct#5783)
    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    1601d82 View commit details
    Browse the repository at this point in the history

  4. [Doc] Add "Suggest edit" button to doc pages (vllm-project#5789)

    @mgoin @robertgshaw2-neuralmagic
    mgoin authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    20faeb6 View commit details
    Browse the repository at this point in the history

  5. [Doc] Add Phi-3-medium to list of supported models (vllm-project#5788)

    @mgoin @robertgshaw2-neuralmagic
    mgoin authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    62ecd68 View commit details
    Browse the repository at this point in the history

  6. [Bugfix] Fix FlexibleArgumentParser replaces _ with - for actual args (… 

    …vllm-project#5795)
    @CatherineSue @robertgshaw2-neuralmagic
    CatherineSue authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    ca8bc83 View commit details
    Browse the repository at this point in the history

  7. [ci] Remove aws template (vllm-project#5757) 

    Signed-off-by: kevin <[email protected]>
    @khluu @robertgshaw2-neuralmagic
    khluu authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    91b2d1d View commit details
    Browse the repository at this point in the history

  8. [Doc] Add notice about breaking changes to VLMs (vllm-project#5818)

    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    21450bc View commit details
    Browse the repository at this point in the history

  9. [Speculative Decoding] Support draft model on different tensor-parall… 

    …el size than target model (vllm-project#5414)
    @wooyeonlee0 @robertgshaw2-neuralmagic
    wooyeonlee0 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    1d55e23 View commit details
    Browse the repository at this point in the history

  10. [Misc] Remove useless code in cpu_worker (vllm-project#5824)

    @DamonFool @robertgshaw2-neuralmagic
    DamonFool authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    980c10b View commit details
    Browse the repository at this point in the history

  11. [Core] Add fault tolerance for RayTokenizerGroupPool (vllm-project#… 

    …5748)
    @Yard1 @robertgshaw2-neuralmagic
    Yard1 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    3b261da View commit details
    Browse the repository at this point in the history

  12. [doc][distributed] add both gloo and nccl tests (vllm-project#5834)

    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    8d6c12f View commit details
    Browse the repository at this point in the history

  13. [CI/Build] Add unit testing for FlexibleArgumentParser (vllm-project#… 

    …5798)
    @mgoin @robertgshaw2-neuralmagic
    mgoin authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    c3bc8c6 View commit details
    Browse the repository at this point in the history

  14. [Misc] Update w4a16 compressed-tensors support to include w8a16 (… 

    …vllm-project#5794)
    @dsikka @robertgshaw2-neuralmagic
    dsikka authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    a9e34b9 View commit details
    Browse the repository at this point in the history

  15. [Hardware][TPU] Refactor TPU backend (vllm-project#5831)

    @WoosukKwon @robertgshaw2-neuralmagic
    WoosukKwon authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    21f69d1 View commit details
    Browse the repository at this point in the history

  16. resolved

    @mawong-amd @robertgshaw2-neuralmagic
    mawong-amd authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    ece7c7f View commit details
    Browse the repository at this point in the history

  17. [Hardware][TPU] Raise errors for unsupported sampling params (vllm-pr… 

    …oject#5850)
    @WoosukKwon @robertgshaw2-neuralmagic
    WoosukKwon authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    e6935bd View commit details
    Browse the repository at this point in the history

  18. [CI/Build] Add E2E tests for MLPSpeculator (vllm-project#5791) 

    Signed-off-by: Thomas Parnell <[email protected]>
    @tdoublep @robertgshaw2-neuralmagic
    tdoublep authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    81a21d2 View commit details
    Browse the repository at this point in the history

  19. [Bugfix] Fix assertion in NeuronExecutor (vllm-project#5841)

    @aws-patlange @robertgshaw2-neuralmagic
    aws-patlange authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    f9775e9 View commit details
    Browse the repository at this point in the history

  20. [Core] Refactor Worker and ModelRunner to consolidate control plane c… 

    …ommunication (vllm-project#5408)

Signed-off-by: Stephanie Wang <[email protected]>
Signed-off-by: Stephanie <[email protected]>
Co-authored-by: Stephanie <[email protected]>
    @stephanie-wang @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    fb41934 View commit details
    Browse the repository at this point in the history

  21. [Misc][Doc] Add Example of using OpenAI Server with VLM (vllm-project… 

    …#5832)
    @ywang96 @robertgshaw2-neuralmagic
    ywang96 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    ce9da79 View commit details
    Browse the repository at this point in the history

  22. [bugfix][distributed] fix shm broadcast when the queue size is full (v… 

    …llm-project#5801)
    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    cb364ef View commit details
    Browse the repository at this point in the history

  23. [Bugfix] Fix embedding to support 2D inputs (vllm-project#5829)

    @WoosukKwon @robertgshaw2-neuralmagic
    WoosukKwon authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    9744700 View commit details
    Browse the repository at this point in the history

  24. [Bugfix][TPU] Fix KV cache size calculation (vllm-project#5860)

    @WoosukKwon @robertgshaw2-neuralmagic
    WoosukKwon authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    2f7eba7 View commit details
    Browse the repository at this point in the history

  25. [CI/Build] Refactor image test assets (vllm-project#5821)

    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    74952fd View commit details
    Browse the repository at this point in the history

  26. [Kernel] Adding bias epilogue support for cutlass_scaled_mm (vllm-p… 

    …roject#5560)

Co-authored-by: Chih-Chieh-Yang <[email protected]>
Co-authored-by: Lucas Wilkinson <[email protected]>
    @ProExpertProg @cyang49
    @LucasWilkinson @robertgshaw2-neuralmagic
    3 people authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    1d1929b View commit details
    Browse the repository at this point in the history

  27. [Frontend] Add tokenize/detokenize endpoints (vllm-project#5054)

    @sasha0552 @robertgshaw2-neuralmagic
    sasha0552 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    5095252 View commit details
    Browse the repository at this point in the history

  28. [Hardware][TPU] Support parallel sampling & Swapping (vllm-project#5855)

    @WoosukKwon @robertgshaw2-neuralmagic
    WoosukKwon authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    1653293 View commit details
    Browse the repository at this point in the history

  29. [Bugfix][TPU] Fix CPU cache allocation (vllm-project#5869)

    @WoosukKwon @robertgshaw2-neuralmagic
    WoosukKwon authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    e423b2c View commit details
    Browse the repository at this point in the history

  30. Support CPU inference with VSX PowerPC ISA (vllm-project#5652)

    @ChipKerchner @robertgshaw2-neuralmagic
    ChipKerchner authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    698f968 View commit details
    Browse the repository at this point in the history

  31. [doc] update usage of env var to avoid conflict (vllm-project#5873)

    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    182cdaa View commit details
    Browse the repository at this point in the history

  32. [Misc] Add example for LLaVA-NeXT (vllm-project#5879)

    @ywang96 @robertgshaw2-neuralmagic
    ywang96 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    750539c View commit details
    Browse the repository at this point in the history

  33. [BugFix] Fix cuda graph for MLPSpeculator (vllm-project#5875) 

    Co-authored-by: Abhinav Goyal <[email protected]>
    @njhill @abhigoyal1997 @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    7823612 View commit details
    Browse the repository at this point in the history

  34. [Doc] Add note about context length in Phi-3-Vision example (vllm-pro… 

    …ject#5887)
    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    0844ba8 View commit details
    Browse the repository at this point in the history

  35. [VLM][Bugfix] Make sure that multi_modal_kwargs is broadcasted prop… 

    …erly (vllm-project#5880)

Signed-off-by: Xiaowei Jiang <[email protected]>
    @xwjiang2010 @robertgshaw2-neuralmagic
    xwjiang2010 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    5855a8e View commit details
    Browse the repository at this point in the history

  36. [Model] Add base class for LoRA-supported models (vllm-project#5018)

    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    2102a46 View commit details
    Browse the repository at this point in the history

  37. [Bugfix] Fix img_sizes Parsing in Phi3-Vision (vllm-project#5888)

    @ywang96 @robertgshaw2-neuralmagic
    ywang96 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    f483510 View commit details
    Browse the repository at this point in the history

  38. [CI/Build] [1/3] Reorganize entrypoints tests (vllm-project#5526)

    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    684c441 View commit details
    Browse the repository at this point in the history

  39. [Model][Bugfix] Implicit model flags and reenable Phi-3-Vision (vllm-… 

    …project#5896)
    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    dcb8246 View commit details
    Browse the repository at this point in the history

  40. [doc][misc] add note for Kubernetes users (vllm-project#5916)

    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    db62aa3 View commit details
    Browse the repository at this point in the history

  41. [BugFix] Fix MLPSpeculator handling of num_speculative_tokens (vl… 

    …lm-project#5876)
    @njhill @robertgshaw2-neuralmagic
    njhill authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    0c7ef70 View commit details
    Browse the repository at this point in the history

  42. [BugFix] Fix min_tokens behaviour for multiple eos tokens (vllm-pro… 

    …ject#5849)
    @njhill @robertgshaw2-neuralmagic
    njhill authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    6e594ee View commit details
    Browse the repository at this point in the history

  43. [CI/Build] Fix Args for _get_logits_warper in Sampler Test (vllm-pr… 

    …oject#5922)
    @ywang96 @robertgshaw2-neuralmagic
    ywang96 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    81ddde3 View commit details
    Browse the repository at this point in the history

  44. [Model] Add Gemma 2 (vllm-project#5908)

    @WoosukKwon @robertgshaw2-neuralmagic
    WoosukKwon authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    c1d4964 View commit details
    Browse the repository at this point in the history

  45. [core][misc] remove logical block (vllm-project#5882)

    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    209a147 View commit details
    Browse the repository at this point in the history

  46. [Kernel][ROCm][AMD] fused_moe Triton configs v2 for mi300X (vllm-proj… 

    …ect#5932)
    @divakar-amd @robertgshaw2-neuralmagic
    divakar-amd authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    4d5e0b9 View commit details
    Browse the repository at this point in the history

  47. [Hardware][TPU] Optimize KV cache swapping (vllm-project#5878)

    @WoosukKwon @robertgshaw2-neuralmagic
    WoosukKwon authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    5f1316e View commit details
    Browse the repository at this point in the history

  48. [VLM][BugFix] Make sure that multi_modal_kwargs can broadcast prope… 

    …rly with ring buffer. (vllm-project#5905)

Signed-off-by: Xiaowei Jiang <[email protected]>
Co-authored-by: Roger Wang <[email protected]>
    @xwjiang2010 @ywang96 @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    74bf88f View commit details
    Browse the repository at this point in the history

  49. [Bugfix][Hardware][Intel CPU] Fix unpassed multi_modal_kwargs for CPU… 

    … runner (vllm-project#5956)
    @Isotr0py @robertgshaw2-neuralmagic
    Isotr0py authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    f177c04 View commit details
    Browse the repository at this point in the history

  50. [Core] Registry for processing model inputs (vllm-project#5214) 

    Co-authored-by: ywang96 <[email protected]>
    @DarkLight1337 @ywang96 @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    70af85d View commit details
    Browse the repository at this point in the history

  51. Unmark fused_moe config json file as executable (vllm-project#5960)

    @tlrmchlsmth @robertgshaw2-neuralmagic
    tlrmchlsmth authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    fd59ff4 View commit details
    Browse the repository at this point in the history

  52. [Hardware][Intel] OpenVINO vLLM backend (vllm-project#5379)

    @ilya-lavrenov @robertgshaw2-neuralmagic
    ilya-lavrenov authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    2e67191 View commit details
    Browse the repository at this point in the history

  53. [Bugfix] Better error message for MLPSpeculator when `num_speculative… 

    …_tokens` is set too high (vllm-project#5894)

Signed-off-by: Thomas Parnell <[email protected]>
    @tdoublep @robertgshaw2-neuralmagic
    tdoublep authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    0d4c0c6 View commit details
    Browse the repository at this point in the history

  54. [CI/Build] [2/3] Reorganize entrypoints tests (vllm-project#5904)

    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    1ce7d18 View commit details
    Browse the repository at this point in the history

  55. [Distributed] Make it clear that % should not be in tensor dict keys. (… 

    …vllm-project#5927)

Signed-off-by: Xiaowei Jiang <[email protected]>
    @xwjiang2010 @robertgshaw2-neuralmagic
    xwjiang2010 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    4b9894c View commit details
    Browse the repository at this point in the history

  56. [Spec Decode] Introduce DraftModelRunner (vllm-project#5799)

    @comaniac @robertgshaw2-neuralmagic
    comaniac authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    6664f2a View commit details
    Browse the repository at this point in the history

  57. [Bugfix] Fix compute datatype for cutlass 3.x epilogues (vllm-project… 

    …#5931)
    @tlrmchlsmth @robertgshaw2-neuralmagic
    tlrmchlsmth authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    42cdb40 View commit details
    Browse the repository at this point in the history

  58. [ Misc ] Remove fp8_shard_indexer from Col/Row Parallel Linear (Sim… 

    …plify Weight Loading) (vllm-project#5928)

Co-authored-by: Robert Shaw <rshaw@neuralmagic>
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic and Robert Shaw committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    7c1515e View commit details
    Browse the repository at this point in the history

  59. [ Bugfix ] Enabling Loading Models With Fused QKV/MLP on Disk with FP8 ( 

    vllm-project#5921)

Co-authored-by: Robert Shaw <rshaw@neuralmagic>
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic and Robert Shaw committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    9598197 View commit details
    Browse the repository at this point in the history

  60. Support Deepseek-V2 (vllm-project#4650) 

    Co-authored-by: Philipp Moritz <[email protected]>
    @zwd003 @pcmoritz @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    3441c30 View commit details
    Browse the repository at this point in the history

  61. [Bugfix] Only add Attention.kv_scale if kv cache quantization is en… 

    …abled (vllm-project#5936)
    @mgoin @robertgshaw2-neuralmagic
    mgoin authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    a5ef790 View commit details
    Browse the repository at this point in the history

  62. Unmark more files as executable (vllm-project#5962)

    @tlrmchlsmth @robertgshaw2-neuralmagic
    tlrmchlsmth authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    ccd94db View commit details
    Browse the repository at this point in the history

  63. [Bugfix] Fix Engine Failing After Invalid Request - AsyncEngineDeadEr… 

    …ror (vllm-project#5963)

Co-authored-by: Robert Shaw <rshaw@neuralmagic>
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic and Robert Shaw committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    f49047a View commit details
    Browse the repository at this point in the history

  64. [Kernel] Flashinfer for prefill & decode, with Cudagraph support for … 

    …decode (vllm-project#4628)

Co-authored-by: LiuXiaoxuanPKU <[email protected]>, bong-furiosa <[email protected]>
    @LiuXiaoxuanPKU @bong-furiosa @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    eeb9d99 View commit details
    Browse the repository at this point in the history

  65. [Bugfix][TPU] Fix TPU sampler output (vllm-project#5978)

    @WoosukKwon @robertgshaw2-neuralmagic
    WoosukKwon authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    026b28e View commit details
    Browse the repository at this point in the history

  66. [Bugfix][TPU] Fix pad slot id (vllm-project#5977)

    @WoosukKwon @robertgshaw2-neuralmagic
    WoosukKwon authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    f281c2e View commit details
    Browse the repository at this point in the history

  67. [Bugfix] fix missing last itl in openai completions benchmark (vllm-p… 

    …roject#5926)
    @mcalman @robertgshaw2-neuralmagic
    mcalman authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    b89416e View commit details
    Browse the repository at this point in the history

  68. [Misc] Extend vLLM Metrics logging API (vllm-project#5925) 

    Co-authored-by: Antoni Baum <[email protected]>
    @SolitaryThinker @Yard1 @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    acf1f76 View commit details
    Browse the repository at this point in the history

  69. [Kernel] Add punica dimensions for Granite 3b and 8b (vllm-project#5930) 

    Signed-off-by: Joe Runde <[email protected]>
    @joerunde @robertgshaw2-neuralmagic
    joerunde authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    b9acdae View commit details
    Browse the repository at this point in the history

  70. [Bugfix] Fix precisions in Gemma 1 (vllm-project#5913)

    @WoosukKwon @robertgshaw2-neuralmagic
    WoosukKwon authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    aa72bdc View commit details
    Browse the repository at this point in the history

  71. [Misc] Update Phi-3-Vision Example (vllm-project#5981) 

    Co-authored-by: Cyrus Leung <[email protected]>
    @ywang96 @DarkLight1337 @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    33fecd4 View commit details
    Browse the repository at this point in the history

  72. [Bugfix] Support eos_token_id from config.json (vllm-project#5954)

    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    00f60d2 View commit details
    Browse the repository at this point in the history

  73. [Core] Optimize SequenceStatus.is_finished by switching to IntEnum (v… 

    …llm-project#5974)
    @Yard1 @robertgshaw2-neuralmagic
    Yard1 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    270105d View commit details
    Browse the repository at this point in the history

  74. [Kernel] Raise an exception in MoE kernel if the batch size is larger… 

    … then 65k (vllm-project#5939)
    @comaniac @robertgshaw2-neuralmagic
    comaniac authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    b22f1be View commit details
    Browse the repository at this point in the history

  75. [ CI/Build ] Added E2E Test For Compressed Tensors (vllm-project#5839) 

    Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Robert Shaw <rshaw@neuralmagic>
    @robertgshaw2-neuralmagic @mgoin
    3 people committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    aa49ffe View commit details
    Browse the repository at this point in the history

  76. [CI/Build] Add TP test for vision models (vllm-project#5892)

    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    b481fe3 View commit details
    Browse the repository at this point in the history

  77. [ CI/Build ] LM Eval Harness Based CI Testing (vllm-project#5838) 

    Co-authored-by: Robert Shaw <rshaw@neuralmagic>
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic and Robert Shaw committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    47407b7 View commit details
    Browse the repository at this point in the history

  78. [Bugfix][CI/Build][Hardware][AMD] Install matching torchvision to fix… 

    … AMD tests (vllm-project#5949)
    @mawong-amd @robertgshaw2-neuralmagic
    mawong-amd authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    3d215cc View commit details
    Browse the repository at this point in the history

  79. [CI/Build] Temporarily Remove Phi3-Vision from TP Test (vllm-project#… 

    …5989)
    @ywang96 @robertgshaw2-neuralmagic
    ywang96 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    d0b7111 View commit details
    Browse the repository at this point in the history

  80. [CI/Build] Reuse code for checking output consistency (vllm-project#5988 

    )
    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    445b0d3 View commit details
    Browse the repository at this point in the history

  81. [CI/Build] [3/3] Reorganize entrypoints tests (vllm-project#5966)

    @DarkLight1337 @robertgshaw2-neuralmagic
    DarkLight1337 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    cea9f6b View commit details
    Browse the repository at this point in the history

  82. [ci][distributed] fix device count call 

    [ci][distributed] fix some cuda init that makes it necessary to use spawn (vllm-project#5991)
    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    4f7381a View commit details
    Browse the repository at this point in the history

  83. [Frontend]: Support base64 embedding (vllm-project#5935) 

    Co-authored-by: Cyrus Leung <[email protected]>
    @llmpros @DarkLight1337 @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    3ceed36 View commit details
    Browse the repository at this point in the history

  84. [Lora] Use safetensor keys instead of adapter_config.json to find une… 

    …xpected modules. (vllm-project#5909)

Co-authored-by: sang <[email protected]>
    @rkooo567 @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    51f3e3f View commit details
    Browse the repository at this point in the history

  85. [ CI ] Temporarily Disable Large LM-Eval Tests (vllm-project#6005) 

    Co-authored-by: [email protected] <rshaw@neuralmagic>
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic and [email protected] committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    9c74b00 View commit details
    Browse the repository at this point in the history

  86. [Misc] Fix get_min_capability (vllm-project#5971)

    @dsikka @robertgshaw2-neuralmagic
    dsikka authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    4153e58 View commit details
    Browse the repository at this point in the history

  87. [ Misc ] Refactor w8a8 to use process_weights_after_load (Simplify … 

    …Weight Loading) (vllm-project#5940)

Co-authored-by: Robert Shaw <rshaw@neuralmagic>
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic and Robert Shaw committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    27a711a View commit details
    Browse the repository at this point in the history

  88. format

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 1, 2024
    2 Configuration menu
    Copy the full SHA
    53655b2 View commit details
    Browse the repository at this point in the history

  89. isort

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    07abe05 View commit details
    Browse the repository at this point in the history

  90. format

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    0f0fec4 View commit details
    Browse the repository at this point in the history

  91. updated skipping

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    1cc7c46 View commit details
    Browse the repository at this point in the history

  92. added skipping to compressed-tensors

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    a699814 View commit details
    Browse the repository at this point in the history

  93. updated

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    9a4be7f View commit details
    Browse the repository at this point in the history

  94. format

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 1, 2024
    1 Configuration menu
    Copy the full SHA
    b4eec34 View commit details
    Browse the repository at this point in the history

  95. [misc][cuda] use nvml to avoid accidentally cuda initialization (vllm… 

    …-project#6007)
    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    08dedd5 View commit details
    Browse the repository at this point in the history

  96. [Speculative Decoding 2/2 ] Integrate typical acceptance sampler into… 

    … Spec Decode Worker (vllm-project#5348)
    @sroy745 @robertgshaw2-neuralmagic
    sroy745 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    dac4bb3 View commit details
    Browse the repository at this point in the history

  97. [ CI ] Re-enable Large Model LM Eval (vllm-project#6031)

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    87a4288 View commit details
    Browse the repository at this point in the history

  98. [doc][misc] remove deprecated api server in doc (vllm-project#6037)

    @youkaichao @robertgshaw2-neuralmagic
    youkaichao authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    81e1c3e View commit details
    Browse the repository at this point in the history

  99. [Misc] update benchmark backend for scalellm (vllm-project#6018)

    @zhyncs @robertgshaw2-neuralmagic
    zhyncs authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    2c3c43b View commit details
    Browse the repository at this point in the history

  100. [doc][misc] further lower visibility of simple api server (vllm-proje… 

    …ct#6041)

Co-authored-by: Simon Mo <[email protected]>
    @youkaichao @simon-mo @robertgshaw2-neuralmagic
    2 people authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    cf4e758 View commit details
    Browse the repository at this point in the history

  101. [Bugfix] Use RayActorError for older versions of Ray in RayTokenizerG… 

    …roupPool (vllm-project#6039)
    @Yard1 @robertgshaw2-neuralmagic
    Yard1 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    9c7608c View commit details
    Browse the repository at this point in the history

  102. [Bugfix] adding chunking mechanism to fused_moe to handle large inputs ( 

    vllm-project#6029)
    @avshalomman @robertgshaw2-neuralmagic
    avshalomman authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    1b7245f View commit details
    Browse the repository at this point in the history

  103. add FAQ doc under 'serving' (vllm-project#5946)

    @llmpros @robertgshaw2-neuralmagic
    llmpros authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    fa05042 View commit details
    Browse the repository at this point in the history

  104. [Bugfix][Doc] Fix Doc Formatting (vllm-project#6048)

    @ywang96 @robertgshaw2-neuralmagic
    ywang96 authored and robertgshaw2-neuralmagic committed Jul 1, 2024
    1 Configuration menu
    Copy the full SHA
    484a2e3 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. Update conftest.py

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Jul 2, 2024
    1 Configuration menu
    Copy the full SHA
    99f1474 View commit details
    Browse the repository at this point in the history

  2. format

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 2, 2024
    2 Configuration menu
    Copy the full SHA
    afb93b9 View commit details
    Browse the repository at this point in the history

  3. make _ImageAssets type hint py3.8 compatible

    @derekk-nm
    derekk-nm committed Jul 2, 2024
    1 Configuration menu
    Copy the full SHA
    fcb4dd3 View commit details
    Browse the repository at this point in the history

  4. fixed sampler

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    ceaf019 View commit details
    Browse the repository at this point in the history

  5. switched to compressed tensors instrad of sparseml

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    655389d View commit details
    Browse the repository at this point in the history

  6. formatted

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 2, 2024
    2 Configuration menu
    Copy the full SHA
    206af82 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. fix sampler test again

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    cd2aa72 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into upstream-sync-2024-07-01

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Jul 3, 2024
    Configuration menu
    Copy the full SHA
    f43cb06 View commit details
    Browse the repository at this point in the history

  3. Remove errant __commit__ definition

    @dbarbuzzi
    dbarbuzzi authored Jul 3, 2024
    1 Configuration menu
    Copy the full SHA
    7a45bfa View commit details
    Browse the repository at this point in the history