Chatbox 连接 vLLM Qwen3

Chatbox 连接 vLLM Qwen3

flyfish

vLLM加载Qwen3

vllm serve Qwen3-14B

INFO 06-14 10:17:13 [__init__.py:239] Automatically detected platform cuda.
INFO 06-14 10:17:14 [api_server.py:1034] vLLM API server version 0.8.4
INFO 06-14 10:17:14 [api_server.py:1035] args: Namespace(subparser='serve', model_tag='Qwen3-14B', config='', host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen3-14B', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config=None, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=None, guided_decoding_backend='auto', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=None, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, enable_reasoning=False, reasoning_parser=None, disable_cascade_attn=False, disable_chunked_mm_input=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7b2744d29760>)
INFO 06-14 10:17:20 [config.py:689] This model supports multiple tasks: {'classify', 'embed', 'score', 'reward', 'generate'}. Defaulting to 'generate'.
INFO 06-14 10:17:20 [config.py:1901] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 06-14 10:17:24 [__init__.py:239] Automatically detected platform cuda.
INFO 06-14 10:17:26 [core.py:61] Initializing a V1 LLM engine (v0.8.4) with config: model='Qwen3-14B', speculative_config=None, tokenizer='Qwen3-14B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=Qwen3-14B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
/home/user/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py:29: UserWarning: Failed to get the IP address, using 0.0.0.0 by default.The value can be set by the environment variable VLLM_HOST_IP or HOST_IP.
  get_ip(), get_open_port())
WARNING 06-14 10:17:27 [utils.py:2444] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x79cbdc7c17f0>
INFO 06-14 10:17:27 [parallel_state.py:959] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 06-14 10:17:27 [cuda.py:221] Using Flash Attention backend on V1 engine.
INFO 06-14 10:17:27 [gpu_model_runner.py:1276] Starting to load model Qwen3-14B...
INFO 06-14 10:17:27 [topk_topp_sampler.py:44] Currently, FlashInfer top-p & top-k sampling sampler is disabled because FlashInfer>=v0.2.3 is not backward compatible. Falling back to the PyTorch-native implementation of top-p & top-k sampling.
Loading safetensors checkpoint shards:   0% Completed | 0/8 [00:00<?, ?it/s]
...

INFO 06-14 10:19:39 [loader.py:458] Loading weights took 132.14 seconds
INFO 06-14 10:19:39 [gpu_model_runner.py:1291] Model loading took 27.5185 GiB and 132.297254 seconds
INFO 06-14 10:19:49 [backends.py:416] Using cache directory: /home/user/.cache/vllm/torch_compile_cache/88c77cf337/rank_0_0 for vLLM's torch.compile
INFO 06-14 10:19:49 [backends.py:426] Dynamo bytecode transform time: 9.16 s
INFO 06-14 10:19:54 [backends.py:132] Cache the graph of shape None for later use
INFO 06-14 10:20:27 [backends.py:144] Compiling a graph for general shape takes 38.03 s
INFO 06-14 10:21:02 [monitor.py:33] torch.compile takes 47.19 s in total
INFO 06-14 10:21:03 [kv_cache_utils.py:634] GPU KV cache size: 85,632 tokens
INFO 06-14 10:21:03 [kv_cache_utils.py:637] Maximum concurrency for 40,960 tokens per request: 2.09x
INFO 06-14 10:21:29 [gpu_model_runner.py:1626] Graph capturing finished in 26 secs, took 0.68 GiB
INFO 06-14 10:21:29 [core.py:163] init engine (profile, create kv cache, warmup model) took 109.58 seconds
INFO 06-14 10:21:29 [core_client.py:435] Core engine process 0 ready.
WARNING 06-14 10:21:29 [config.py:1177] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
INFO 06-14 10:21:29 [serving_chat.py:118] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
INFO 06-14 10:21:29 [serving_completion.py:61] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
INFO 06-14 10:21:29 [api_server.py:1081] Starting vLLM API server on http://0.0.0.0:8000
INFO 06-14 10:21:29 [launcher.py:26] Available routes are:
INFO 06-14 10:21:29 [launcher.py:34] Route: /openapi.json, Methods: GET, HEAD
INFO 06-14 10:21:29 [launcher.py:34] Route: /docs, Methods: GET, HEAD
INFO 06-14 10:21:29 [launcher.py:34] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 06-14 10:21:29 [launcher.py:34] Route: /redoc, Methods: GET, HEAD
INFO 06-14 10:21:29 [launcher.py:34] Route: /health, Methods: GET
INFO 06-14 10:21:29 [launcher.py:34] Route: /load, Methods: GET
INFO 06-14 10:21:29 [launcher.py:34] Route: /ping, Methods: GET, POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /tokenize, Methods: POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /detokenize, Methods: POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /v1/models, Methods: GET
INFO 06-14 10:21:29 [launcher.py:34] Route: /version, Methods: GET
INFO 06-14 10:21:29 [launcher.py:34] Route: /v1/chat/completions, Methods: POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /v1/completions, Methods: POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /v1/embeddings, Methods: POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /pooling, Methods: POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /score, Methods: POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /v1/score, Methods: POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /v1/audio/transcriptions, Methods: POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /rerank, Methods: POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /v1/rerank, Methods: POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /v2/rerank, Methods: POST
INFO 06-14 10:21:29 [launcher.py:34] Route: /invocations, Methods: POST
INFO:     Started server process [10139]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO 06-14 10:21:39 [loggers.py:87] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

1. 基础信息

• 模型名称：Qwen3-14B（140亿参数量的Qwen3版本）
• 运行框架：vLLM 0.8.4（高性能LLM推理框架）
• 硬件环境：CUDA平台（NVIDIA GPU）
• 启动命令：vllm serve Qwen3-14B

2. 关键启动过程

• 使用的采样参数（默认）：
  • 生成温度（Temperature）：0.6
  • Top-k：20
  • Top-p：0.95

3. 系统配置

• 监听端口：8000
• 启用功能：
  • Chat（聊天）
  • Completion（文本生成）
  • Embedding（嵌入）
  • 多种推理模式（generate, classify, score, reward）
• 调用接口示例：
  • 本地调用：http://localhost:8000/v1/chat/completions
  • 其他支持接口包括：
    • /v1/completions（文本生成）
    • /v1/embeddings（文本嵌入）
    • /v1/rerank（重排序）

Chatbox 配置