[WIP] Integrating vllm>=0.7.0

[ZSL98]

This PR aims to integrate vllm>=0.7.0 and preserve:
Backward compatibility: 0.3.1, 0.4.2, 0.5.4, 0.6.3 are still supported
Forward compatibility: Future versions of vllm (>= 0.7.0) will be supported without requiring manual maintenance for each new release.

The integration up to now only supports the fsdp backend. We have some rough numbers (vllm generation time in seconds in an iteration) for comparison. We are happy to see that with eager mode off, the generation time is shorten with the help of cuda graph. The sleep mode of vllm helps to reduce GPU memory pressure.

	vllm 0.6.3	vllm 0.7.0 (eager=true)	vllm 0.7.0 (eager=false)
Qwen2-7b	16	13.5	12
Qwen2-7b_rm	14.5	11.5	10
Qwen2-7b_seq_balance	99	97	72

To reproduce

1. Install vllm with pip3 install vllm==0.7.0 or pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly.
2. Apply the following minor modifications (I have posted as an issue [Bug]: Some issues in integrating vllm with verl vllm-project/vllm#12782):

• vllm/distributed/parallel_state.py: Comment out the assertion below:

    if (world_size
            != tensor_model_parallel_size * pipeline_model_parallel_size):
        raise RuntimeError(
            f"world_size ({world_size}) is not equal to "
            f"tensor_model_parallel_size ({tensor_model_parallel_size}) x "
            f"pipeline_model_parallel_size ({pipeline_model_parallel_size})")

• vllm/executor/uniproc_executor.py: change local_rank = rank to local_rank = int(os.environ["LOCAL_RANK"])

3. Try a bash run_qwen2-7b.sh under examples/ppo_trainer

Note

The test cases are currently limited, and bug-free operation is not guaranteed at this stage.

[PeterSH6]

I wonder why we need this function? It seems that vLLM UniProcExecutor already perform allreduce MIN.

[ZSL98]

I have removed this file.