LLM 部署(3)——vLLM CPU 和 GPU 模式部署大语言模型

vLLM 官网源代码地址：https://github.com/vllm-project/vllm

vLLM 支持目前主流大模型，详细列表见官网：https://docs.vllm.ai/en/latest/models/supported_models.html

1 vLLM 环境准备

vLLM目前只支持Linux操作系统。

1.1 安装 Miniconda 包管理工具

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh

Shell 上下文激活Miniconda命令：

~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh
 
source ~/.bashrc
source ~/.zshrc

1.2 设置 Miniconda 国内镜像

Miniconda配置文件路径：~/.condarc，一般情况下配置文件不存在，可以创建并初始化它：conda config --set show_channel_urls yes

然后打开配置文件，设置依赖包镜像渠道：

show_channel_urls: true
channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/msys2/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
  - defaults
custom_channels:
  conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud

保存配置文件，查看配置是否生效：conda info

1.3 设置 Python 版本和虚拟环境

通过Minicodan安装 Python 虚拟环境：conda create --name vLLM python=3.10 -y

激活环境：conda activate vLLM

2 下载模型权重文件

2.1 Git 方式下载

• 模型权重文件比较大，需要用到 Git 大文件系统，因此需要提前安装好：

sudo apt-get install git-lfs

• 下载模型

# 创建目录
mkdir -p ~/ModelSpace && cd ~/ModelSpace

# 下载文件
git lfs install
git clone https://www.modelscope.cn/qwen/qwen2-0.5b.git Qwen2-0.5B

下载过程中，如果因网络等原因中断，可以继续断点下载：

cd ~/ModelSpace/Qwen2-0.5B
git lfs pull`

2.2 SDK 和命令行方式下载

• SDK 下载

pip install modelscope
 
from modelscope import snapshot_download
model_dir = snapshot_download('qwen/qwen2-0.5b')

• 命令行下载

pip install modelscope
modelscope download --model qwen/qwen2-0.5b

3 使用 vLLM 部署大模型

vLLM的依赖包默认支持 GPU 部署和推理，如果使用CPU推理，需要根据vLLM源代码重新编译打包。

• GPU 部署和推理

通过 PIP 直接安装依赖包即可：

pip install vLLM

• CPU 部署和推理

需要下载vLLM源代码，自己编译打包和安装。

3.1 CPU 通过vLLM源代码编译打包装

• 下载vLLM源代码

mkdir -p ~/CodeSpace
cd ~/CodeSpace

git clone https://github.com/vllm-project/vllm.git vllm-project

• 安装vLLM的依赖

cd ~/CodeSpace/vllm-project
 
pip install --upgrade pip
pip install wheel packaging ninja "setuptools>=49.4.0" numpy
pip install -v -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu

• vLLM打包安装了：

cd ~/CodeSpace/vllm-project
VLLM_TARGET_DEVICE=cpu python setup.py install

3.2 vLLM 本地大模型部署

# Qwen2-vLLM-Local.py
import os
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
 
# 设置环境变量
os.environ['VLLM_TARGET_DEVICE'] = 'cpu'
 
# 模型ID：我们下载的模型权重文件目录
model_dir = '/home/obullxl/ModelSpace/Qwen2-0.5B'
 
# Tokenizer初始化
tokenizer = AutoTokenizer.from_pretrained(
    model_dir,
    local_files_only=True,
)
 
# Prompt提示词
messages = [
    {'role': 'system', 'content': 'You are a helpful assistant.'},
    {'role': 'user', 'content': '天空为什么是蓝色的？'}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
 
# 初始化大语言模型
llm = LLM(
    model=model_dir,
    tensor_parallel_size=1,  # CPU无需张量并行
    device='cpu',
)
 
# 超参数：最多512个Token
sampling_params = SamplingParams(temperature=0.7, top_p=0.8, repetition_penalty=1.05, max_tokens=512)
 
# 模型推理输出
outputs = llm.generate([text], sampling_params)
 
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
 
    print(f'Prompt提示词: {prompt!r}, 大模型推理输出: {generated_text!r}')

3.3 发布 API 服务和调用推理

• 通过vLLM把本地大模型部署成 OpenAI API 服务

python -m vllm.entrypoints.openai.api_server --model ~/ModelSpace/Qwen2-0.5B

• 默认情况下，API 服务端口为8000，可通过 --port 参数设置服务端口；同时，可通过--host 参数设置服务地址

python -m vllm.entrypoints.openai.api_server --model /home/obullxl/ModelSpace/Qwen2-0.5B --port 8000 --host 0.0.0.0

• 通过 CURL 命令验证服务

# Qwen2-vLLM-CURL.py
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "/home/obullxl/ModelSpace/Qwen2-0.5B",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "天空为什么是蓝色的？"}
  ],
  "temperature": 0.7,
  "top_p": 0.8,
  "repetition_penalty": 1.05,
  "max_tokens": 512
}'

• 也可以通过 Python 客户端调用API 访问服务：pip install openai

# Qwen2-vLLM-OpenAI.py
from openai import OpenAI
 
# OpenAI初始化
client = OpenAI(
    api_key='EMPTY',
    base_url='http://localhost:8000/v1',
)
 
chat_response = client.chat.completions.create(
    model='/home/obullxl/ModelSpace/Qwen2-0.5B',
    messages=[
        {'role': 'system', 'content': 'You are a helpful assistant.'},
        {'role': 'user', 'content': '天空为什么是蓝色的？'},
    ],
    temperature=0.7,
    top_p=0.8,
    max_tokens=512,
)
 
print('Qwen2推理结果:', chat_response)

• 还可以通过 WebUI 访问我们部署的 API 服务

pip install gradio

# Qwen2-vLLM-WebUI.py
import argparse
import json
 
import gradio as gr
import requests
 
 
def http_bot(prompt):
    headers = {"User-Agent": "vLLM Client"}
    pload = {
        "prompt": prompt,
        "stream": True,
        "max_tokens": 128,
    }
    response = requests.post(args.model_url,
                             headers=headers,
                             json=pload,
                             stream=True)
 
    for chunk in response.iter_lines(chunk_size=8192,
                                     decode_unicode=False,
                                     delimiter=b"\0"):
        if chunk:
            data = json.loads(chunk.decode("utf-8"))
            output = data["text"][0]
            yield output
 
 
def build_demo():
    with gr.Blocks() as demo:
        gr.Markdown("# vLLM text completion demo\n")
        inputbox = gr.Textbox(label="Input",
                              placeholder="Enter text and press ENTER")
        outputbox = gr.Textbox(label="Output",
                               placeholder="Generated result from the model")
        inputbox.submit(http_bot, [inputbox], [outputbox])
    return demo
 
 
if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--host", type=str, default=None)
    parser.add_argument("--port", type=int, default=8001)
    parser.add_argument("--model-url",
                        type=str,
                        default="http://0.0.0.0:8000/generate")
    args = parser.parse_args()
 
    demo = build_demo()
    demo.queue().launch(server_name=args.host,
                        server_port=args.port,
                        share=True)

启动 WebUI 服务：python Qwen2-vLLM-WebUI.py

浏览器打开 WebUI 界面：http://localhost:8001(如是Windows WSL子系统，可以通过ifconfig命令、或者直接通过ifconfig | grep eth0 -n1 | grep inet | awk '{print $3}'命令获取 WSL 的 IP 地址)

通过 WebUI 就可以与大模型对话了：

Running on local URL:  http://127.0.0.1:8001

如果是Windows WSL子系统，那么需要把 WebUI 设置为共享模式，否则会有如下提示：

Running on local URL:  http://127.0.0.1:8001
 
Could not create share link. Missing file: /home/obullxl/miniconda3/envs/vLLM/lib/python3.10/site-packages/gradio/frpc_linux_amd64_v0.2.
 
Please check your internet connection. This can happen if your antivirus software blocks the download of this file. You can install manually by following these steps:
 
1. Download this file: https://cdn-media.huggingface.co/frpc-gradio-0.2/frpc_linux_amd64
2. Rename the downloaded file to: frpc_linux_amd64_v0.2
3. Move the file to this location: /home/obullxl/miniconda3/envs/vLLM/lib/python3.10/site-packages/gradio

frpc_linux_amd64文件默认在 HF 上，需要通畅的网络，百度网盘

链接: https://pan.baidu.com/s/1CYDcbkUEhhhCEuzb5z8xXA?pwd=LNTX
提取码: LNTX

3.4 GPU 多卡部署和推理大模型

通过tensor_parallel_size参数启用 GPU 多卡分布式并行推理能力。

from vllm import LLM, SamplingParams
 
# ...... 省略部分
 
llm = LLM(
  model=model_dir,
  tensor_parallel_size=4,
)
 
# ....... 其他省略

也可以通过 --tensor-parallel-size 参数部署并发布 API 服务：

python -m vllm.entrypoints.api_server --model /home/obullxl/ModelSpace/Qwen2-0.5B --tensor-parallel-size 4

客户端的使用方法，与之前完全一样

https://gitee.com/obullxl/SunningTX/tree/master/PythonTX/Qwen-vLLM