DeepSeek-r1:671b满血版在K8S+SGLang架构下的多节点GPU私有化实践

应用前景#

随着DeepSeek-r1千亿级大模型在代码生成、数学推理等复杂任务中的突破性表现，企业级私有化部署需求呈现指数级增长。当前市场中，Ollama凭借轻量化架构和跨平台兼容性（支持NVIDIA/AMD全系GPU及主流大模型格式），为开发者提供了开箱即用的本地调试方案。但其单节点架构与朴素的调度策略，在面对生产级高并发推理场景时，吞吐量相较vLLM、SGLang等专用推理框架存在30%以上的性能鸿沟。

解决方案#

本文将以DeepSeek-r1-671b满血版为基准模型，深入解析基于Kubernetes+SGLang的云原生推理加速架构。通过融合LeaderWorkerSet控制器实现分布式任务编排、Volcano批量调度系统强化GPU资源抢占式分配，构建具备以下特性的企业级部署方案：

性能跃迁：基于SGLang的RadixAttention核心技术，实现KV Cache复用率提升60%+
弹性拓扑：支持Multi-Node Multi-GPU的动态扩缩容策略（H100/A100异构集群兼容）
生产就绪：集成Prometheus+Grafana的实时推理监控体系，TP99延迟可控在200ms内

选型SGLang推理引擎的理由#

SGLang vs Ollama 关键能力对比矩阵#

能力维度	SGLang (生产级引擎)	Ollama (开发级工具)
架构设计	✅ 分布式推理架构（多机多卡协同）	❌ 单节点运行（仅限本地GPU）
性能表现	🔥 吞吐量提升300%+ （RadixAttention优化）	⏳ 适合低并发场景（朴素调度策略）
生产就绪性	📊 内置Prometheus监控+熔断降级机制	❌ 无监控/高可用保障
扩展能力	⚡ 动态扩缩容+异构集群管理（K8s/Volcano集成）	❌ 固定资源配置（无集群支持）
企业特性	🔒 商业SLA支持+定制化OP开发	❌ 仅社区版维护
适用场景	千亿级模型生产环境部署（电商/金融等高并发场景）	个人开发者本地调试（小模型快速验证）

为什么选择SGLang？#

性能碾压
- Ollama单卡QPS≤20，SGLang分布式集群QPS≥200（10倍提升）
- 在32k长文本场景，SGLang推理延迟稳定在300ms内，Ollama频繁触发OOM
成本优势
- 通过KV Cache复用，集群资源利用率达85%+（Ollama仅40%-50%）
- 支持FP8量化压缩，相同吞吐量下硬件成本降低60%
风险控制
- Ollama无熔断/降级机制，突发流量易导致服务雪崩
- SGLang内置分级流量管控，保障核心业务SLA不中断

决策建议：

选SGLang：当您需要支撑线上生产流量、处理百亿级以上参数模型、实现资源集约化利用
选Ollama：仅用于个人学习研究、小模型快速验证、无SLA要求的本地测试

通过架构级优化与生产增强设计，SGLang在性能、稳定性、扩展性等维度实现对Ollama的代际差距级超越。

环境准备#

本次部署的为满血版 DeepSeek-r1:671b

硬件配置

服务器	数量（台）	CPU（核）	内存（TB）	系统版本
NVIDIA A800 80GB	2	128	2	Ubuntu 22.04.5 LTS

软件平台

软件名称版本备注
Kubernetes v1.30.6 容器编排引擎
GPU Operator v24.9.1 自动化管理配置GPU驱动程序
Volcano v1.9.0 调度引擎
NVIDIA Driver 550.127.05 GPU驱动
NVIDIA-Fabric Manager 550.127.05 NVSwitch互联
CUDA 12.4 Cuda
MLNX_OFED 24.10-0.7.0.0 IB驱动
NCCL 2.21.5 GPU多卡通信
SGLang v0.4.3.post2-cu124 LLM推理引擎
LeaderWorkerSet v0.5.1 PodGroup Deploy API
open-webui v0.5.14 AI聊天互动工具

模型准备#
方式一：通过HuggingFace 下载
仓库地址：https://huggingface.co/deepseek-ai/DeepSeek-R1

软件名称	版本	备注
Kubernetes	v1.30.6	容器编排引擎
GPU Operator	v24.9.1	自动化管理配置GPU驱动程序
Volcano	v1.9.0	调度引擎
NVIDIA Driver	550.127.05	GPU驱动
NVIDIA-Fabric Manager	550.127.05	NVSwitch互联
CUDA	12.4	Cuda
MLNX_OFED	24.10-0.7.0.0	IB驱动
NCCL	2.21.5	GPU多卡通信
SGLang	v0.4.3.post2-cu124	LLM推理引擎
LeaderWorkerSet	v0.5.1	PodGroup Deploy API
open-webui	v0.5.14	AI聊天互动工具

方式二：通过 ModelScope 下载（国内推荐用这个）
仓库地址：https://modelscope.cn/models/deepseek-ai/DeepSeek-R1/files

1
1、安装ModelScope
2
pip3 install modelscope
3

4
2、下载完整模型repo
5
mkdir /mnt/catcat_data/model/DeepSeek-R1 -p
6
nohup modelscope download --model deepseek-ai/DeepSeek-R1 --local_dir /mnt/catcat_data/model/DeepSeek-R1 &

实测满血模型在Linux为642G.

部署#

部署LWS API#

Github项目地址：https://github.com/kubernetes-sigs/lws

使用 LWS API 的主要优势包括：

简化分布式推理的部署 ：通过 LWS API，提供了一个声明式的 API，用户只需定义 Leader 和 Worker 的配置，Kubernetes 控制器会自动处理其生命周期管理。用户可以更轻松地部署复杂的分布式推理工作负载，而无需手动管理 Leader 和 Worker 的依赖关系和副本数量;
无缝水平扩容 ：上文中提到分布式推理的服务需要多个POD 共同提供服务，在进行扩容时也需要以多个Pod 一组为原子单位进行扩展， LWS 可以与 k8s HPA 无缝对接，将 LWS 作为HPA 扩容的Target，实现推理服务整组扩容;
拓扑感知调度 ：在分布式推理中，不同 Pod 需要进行大量数据交互。为了减少通信延时 LWS API 结合了拓扑感知调度，保证能够保证 Leader 和 Worker Pod 能够调度到 RDMA 网络中拓扑距离尽可能接近的节点上。

1
安装 LWS API 的 CRD
2
VERSION=v0.5.1
3
kubectl apply --server-side -f https://github.com/kubernetes-sigs/lws">https://github.com/kubernetes-sigs/lws/releases/download/$VERSION/manifests.yaml
4

5
检查LWS 资源
6
kubectl get pods -n lws-system
7
kubectl get svc -n lws-system
8
kubectl api-resources |grep -i lws

部署DeepSeek-R1#

1
apiVersion: leaderworkerset.x-k8s.io/v1
2
kind: LeaderWorkerSet
3
metadata:
4
  name: sglang
5
  labels:
6
    app: sglang
7
spec:
8
  replicas: 1
9
  startupPolicy: LeaderCreated
10
  rolloutStrategy:
11
    type: RollingUpdate
12
    rollingUpdateConfiguration:
13
      maxSurge: 0
14
      maxUnavailable: 2
15
  leaderWorkerTemplate:
16
    size: 2
17
    restartPolicy: RecreateGroupOnPodRestart
18
    leaderTemplate:
19
      metadata:
20
        labels:
21
          role: leader
22
      spec:
23
        containers:
24
          - name: sglang-head
25
            image: lmsysorg/sglang:v0.4.3.post2-cu124
26
            imagePullPolicy: IfNotPresent
27
            workingDir: /sgl-workspace
28
            command: ["sh", "-c"]
29
            args:
30
            - >
31
              cd /sgl-workspace && python3 -m sglang.launch_server
32
              --model-path /mnt/catcat_data/model/DeepSeek-R1
33
              --served-model-name deepseek-r1
34
              --tp 16
35
              --dist-init-addr $LWS_LEADER_ADDRESS:20000
36
              --nnodes $LWS_GROUP_SIZE
37
              --node-rank 0
38
              --trust-remote-code
39
              --context-length 131072
40
              --enable-metrics
41
              --host 0.0.0.0
42
              --port 8000
43
            env:
44
              - name: GLOO_SOCKET_IFNAME
45
                value: eth0
46
              - name: NCCL_IB_HCA
47
                value: "mlx5_0,mlx5_1,mlx5_4,mlx5_5"
48
              - name: NCCL_P2P_LEVEL
49
                value: "NVL"
50
              - name: NCCL_IB_GID_INDEX
51
                value: "0"
52
              - name: NCCL_IB_CUDA_SUPPORT
53
                value: "1"
54
              - name: NCCL_IB_DISABLE
55
                value: "0"
56
              - name: NCCL_SOCKET_IFNAME
57
                value: "eth0"
58
              - name: NCCL_DEBUG
59
                value: "INFO"
60
              - name: NCCL_NET_GDR_LEVEL
61
                value: "2"
62
              - name: POD_NAME
63
                valueFrom:
64
                  fieldRef:
65
                    fieldPath: metadata.name
66
              - name: SGLANG_USE_MODELSCOPE
67
                value: "true"
68
            ports:
69
            - containerPort: 8000
70
              name: http
71
              protocol: TCP
72
            - containerPort: 20000
73
              name: distributed
74
              protocol: TCP
75
            resources:
76
              limits:
77
                cpu: "128"
78
                memory: "1Ti"
79
                nvidia.com/gpu: "8"
80
                rdma/ib: "4"
81
              requests:
82
                cpu: "128"
83
                memory: "1Ti"
84
                nvidia.com/gpu: "8"
85
                rdma/ib: "4"
86
            securityContext:
87
              capabilities:
88
                add:
89
                - IPC_LOCK
90
                - SYS_PTRACE
91
            volumeMounts:
92
              - mountPath: /mnt/catcat_data/model
93
                name: model-volume
94
              - mountPath: /dev/shm
95
                name: shm-volume
96
              - name: localtime
97
                mountPath: /etc/localtime
98
                readOnly: true
99
            readinessProbe:
100
              tcpSocket:
101
                port: 8000
102
              initialDelaySeconds: 120
103
              periodSeconds: 30
104
        volumes:
105
          - name: model-volume
106
            hostPath:
107
              path: /mnt/catcat_data/model
108
          - name: shm-volume
109
            emptyDir:
110
              sizeLimit: 512Gi
111
              medium: Memory
112
          - name: localtime
113
            hostPath:
114
              path: /etc/localtime
115
              type: File
116
        schedulerName: volcano
117
    workerTemplate:
118
      metadata:
119
        name: sglang-worker
120
      spec:
121
        containers:
122
          - name: sglang-worker
123
            image: lmsysorg/sglang:v0.4.3.post2-cu124
124
            imagePullPolicy: IfNotPresent
125
            workingDir: /sgl-workspace
126
            command: ["sh", "-c"]
127
            args:
128
            - >
129
              cd /sgl-workspace && python3 -m sglang.launch_server
130
              --model-path /mnt/catcat_data/model/DeepSeek-R1
131
              --served-model-name deepseek-r1
132
              --tp 16
133
              --dist-init-addr $LWS_LEADER_ADDRESS:20000
134
              --nnodes $LWS_GROUP_SIZE
135
              --node-rank $LWS_WORKER_INDEX
136
              --trust-remote-code
137
              --context-length 131072
138
              --enable-metrics
139
              --host 0.0.0.0
140
              --port 8000
141
            env:
142
              - name: GLOO_SOCKET_IFNAME
143
                value: eth0
144
              - name: NCCL_IB_HCA
145
                value: "mlx5_0,mlx5_1,mlx5_4,mlx5_5"
146
              - name: NCCL_P2P_LEVEL
147
                value: "NVL"
148
              - name: NCCL_IB_GID_INDEX
149
                value: "0"
150
              - name: NCCL_IB_CUDA_SUPPORT
151
                value: "1"
152
              - name: NCCL_IB_DISABLE
153
                value: "0"
154
              - name: NCCL_SOCKET_IFNAME
155
                value: "eth0"
156
              - name: NCCL_DEBUG
157
                value: "INFO"
158
              - name: NCCL_NET_GDR_LEVEL
159
                value: "2"
160
              - name: SGLANG_USE_MODELSCOPE
161
                value: "true"
162
              - name: LWS_WORKER_INDEX
163
                valueFrom:
164
                  fieldRef:
165
                    fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
166
            ports:
167
            - containerPort: 8000
168
              name: http
169
              protocol: TCP
170
            - containerPort: 20000
171
              name: distributed
172
              protocol: TCP
173
            resources:
174
              limits:
175
                cpu: "128"
176
                memory: "1Ti"
177
                nvidia.com/gpu: "8"
178
                rdma/ib: "4"
179
              requests:
180
                cpu: "128"
181
                memory: "1Ti"
182
                nvidia.com/gpu: "8"
183
                rdma/ib: "4"
184
            securityContext:
185
              capabilities:
186
                add:
187
                - IPC_LOCK
188
                - SYS_PTRACE
189
            volumeMounts:
190
              - mountPath: /mnt/catcat_data/model
191
                name: model-volume
192
              - mountPath: /dev/shm
193
                name: shm-volume
194
              - name: localtime
195
                mountPath: /etc/localtime
196
                readOnly: true
197
        volumes:
198
          - name: model-volume
199
            hostPath:
200
              path: /mnt/catcat_data/model
201
          - name: shm-volume
202
            emptyDir:
203
              sizeLimit: 512Gi
204
              medium: Memory
205
          - name: localtime
206
            hostPath:
207
              path: /etc/localtime
208
              type: File
209
        schedulerName: volcano

1
kubectl apply -f deepseek-r1-lws-sglang.yaml
2

3
kubectl get lws -n deepseek
4
NAME     AGE
5
sglang   1h
6

7
kubectl get pods -n deepseek |grep sglang
8
sglang-0                                 1/1     Running   0          2h
9
sglang-0-1                               1/1     Running   0         2h

1
##查看日志
2
~# kubectl logs -n deepseek sglang-0
3
[2025-02-16 12:25:49] server_args=ServerArgs(model_path='deepseek-ai/DeepSeek-R1', tokenizer_path='deepseek-ai/DeepSeek-R1', tokenizer_mode='auto', load_format='auto', trust_remote_code=True, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, quantization=None, context_length=None, device='cuda', served_model_name='deepseek-ai/DeepSeek-R1', chat_template=None, is_embedding=False, revision=None, skip_tokenizer_init=False, host='0.0.0.0', port=30000, mem_fraction_static=0.81, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=4096, max_prefill_tokens=16384, schedule_policy='lpm', schedule_conservativeness=0.3, cpu_offload_gb=0, prefill_only_one_req=False, tp_size=8, stream_interval=1, stream_output=False, random_seed=773491082, constrained_json_whitespace_pattern=None, watchdog_timeout=300, download_dir=None, base_gpu_id=0, log_level='info', log_level_http=None, log_requests=False, show_time_cost=False, enable_metrics=False, decode_log_interval=40, api_key=None, file_storage_pth='sglang_storage', enable_cache_report=False, dp_size=8, load_balance_method='round_robin', ep_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend='flashinfer', sampling_backend='flashinfer', grammar_backend='outlines', speculative_draft_model_path=None, speculative_algorithm=None, speculative_num_steps=5, speculative_num_draft_tokens=64, speculative_eagle_topk=8, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, disable_jump_forward=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_nccl_nvls=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, disable_mla=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=True, enable_ep_moe=False, enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=160, cuda_graph_bs=None, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, return_hidden_states=False, enable_custom_logit_processor=False, tool_call_parser=None, enable_hierarchical_cache=False, enable_flashinfer_mla=False)
4
Downloading Model to directory: /root/.cache/modelscope/hub/deepseek-ai/DeepSeek-R1
5
Downloading Model to directory: /root/.cache/modelscope/hub/deepseek-ai/DeepSeek-R1
6
INFO 02-16 12:25:53 __init__.py:190] Automatically detected platform cuda.
7
INFO 02-16 12:25:53 __init__.py:190] Automatically detected platform cuda.
8
INFO 02-16 12:26:01 __init__.py:190] Automatically detected platform cuda.
9
INFO 02-16 12:26:01 __init__.py:190] Automatically detected platform cuda.
10
INFO 02-16 12:26:01 __init__.py:190] Automatically detected platform cuda.
11
INFO 02-16 12:26:01 __init__.py:190] Automatically detected platform cuda.
12
INFO 02-16 12:26:01 __init__.py:190] Automatically detected platform cuda.
13
INFO 02-16 12:26:01 __init__.py:190] Automatically detected platform cuda.
14
INFO 02-16 12:26:01 __init__.py:190] Automatically detected platform cuda.
15
INFO 02-16 12:26:01 __init__.py:190] Automatically detected platform cuda.

查看显存占用#

服务访问#

编写SVC

1
apiVersion: v1
2
kind: Service
3
metadata:
4
  name: sglang-api-svc
5
  labels:
6
    app: sglang
7
spec:
8
  selector:
9
      leaderworkerset.sigs.k8s.io/name: sglang
10
      role: leader
11
  ports:
12
    - protocol: TCP
13
      port: 8000
14
      targetPort: http
15
      name: http
16
  type: NodePort

部署SVC

kubectl apply -f deepseek-r1-svc.yaml -n deepseek

Curl 测试部署#

1
curl -X POST http://ip:port/v1/chat/completions -H "Content-Type: application/json" -d '{
2
    "model": "/model",
3
    "messages": [
4
        {
5
            "role": "user",
6
            "content": "你是什么模型"
7
        }
8
    ],
9
    "stream": false,
10
    "temperature": 0.8
11
}'

部署OpenwebUI#

这里给出yaml，不在过多赘述了

1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: open-webui-data-pvc
5
spec:
6
  accessModes:
7
    - ReadWriteOnce
8
  resources:
9
    requests:
10
      storage: 100Gi
11
  storageClassName: nfs-client
12

13
---
14
apiVersion: apps/v1
15
kind: Deployment
16
metadata:
17
  name: open-webui-deployment
18
spec:
19
  replicas: 1
20
  selector:
21
    matchLabels:
22
      app: open-webui
23
  template:
24
    metadata:
25
      labels:
26
        app: open-webui
27
    spec:
28
      containers:
29
      - name: open-webui
30
        image: ghcr.sakiko.de/open-webui/open-webui:main
31
        imagePullPolicy: Always
32
        ports:
33
        - containerPort: 8080
34
        env:
35
        - name: OPENAI_API_BASE_URL
36
          value: "http://ip:port/v1"   # 替换为SGLang API
37
        - name: ENABLE_OLLAMA_API # 禁用 Ollama API，只保留 OpenAI API
38
          value: "False"
39
        volumeMounts:
40
        - name: open-webui-data
41
          mountPath: /app/backend/data
42
      volumes:
43
      - name: open-webui-data
44
        persistentVolumeClaim:
45
          claimName: open-webui-data-pvc
46

47
---
48
apiVersion: v1
49
kind: Service
50
metadata:
51
  name: open-webui-service
52
spec:
53
  type: ClusterIP
54
  ports:
55
    - port: 3000
56
      targetPort: 8080
57
  selector:
58
    app: open-webui