Running deepseek-r1 617B with Ollama on Ubuntu 22.04 + 8×A800

Discovered that my machine happens to meet the requirements for deepseek-r1 617B. Since it was sitting idle anyway, I decided to give it a test run.

System & Hardware Overview#

Processor : 2*Intel(R) Xeon(R) Platinum 8362 CPU @ 2.80GHz
Num of Core : 128 Cor_e
Memory : 1024 GB
Storage : 1.5T NVMe
GPU : 8*A800
NVIDIA-SMI 550.127.05
Driver Version: 550.127.05
CUDA Version: 12.4

Download Ollama#

Download from: https://ollama.com/

Install Ollama#

Directly reuse the official installation script:

1
curl -fsSL https://ollama.com/install.sh | sh

Configure the model download path#

1
mkdir -p /root/ollama/ollama_models

Then add it to Ollama.

If OLLAMA_MODELS is not configured at the beginning, the default path is /usr/share/ollama/.ollama/models.

1
vim .bashrc
2
export OLLAMA_MODELS=/root/ollama/ollama_models

Start the Ollama service#

Run Ollama#

1
ollama server

Modify Ollama configuration#

By default, Ollama only listens on localhost:11434, so it is only accessible from localhost.

1
vim /etc/systemd/system/ollama.service
2
# Add the following under [Service]
3
Environment="OLLAMA_HOST=0.0.0.0"
4

5
cat /etc/systemd/system/ollama.service
6
[Unit]
7
Description=Ollama Service
8
After=network-online.target
9

10
[Service]
11
ExecStart=/usr/local/bin/ollama serve
12
User=ollama
13
Group=ollama
14
Restart=always
15
RestartSec=3
16
Environment="PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin"
17
Environment="OLLAMA_HOST=0.0.0.0"
18

19
[Install]
20
WantedBy=default.target

Restart Ollama#

1
systemctl daemon-reload
2

3
systemctl restart ollama
4

5
# Stop the service
6
systemctl stop ollama
7
# Start the service
8
systemctl start ollama

Run the model#

1
ollama run deepseek-r1:671b

Configure Docker + nvidia-docker2#

Install Docker#

1
export DOWNLOAD_URL="https://mirrors.tuna.tsinghua.edu.cn/docker-ce"
2
curl -fsSL https://raw.githubusercontent.com/docker/docker-install/master/install.sh | sh

Install GPU-Docker components#

1
# Install gpu-docker
2

3
apt-get install -y nvidia-docker2
4
nvidia-ctk runtime configure --runtime=docker
5

6
# This will modify the daemon.json file and add the container runtime

Configure Docker parameters#

1
root@catcat:~# cat /etc/docker/daemon.json
2
{
3
    "data-root": "/root/docker_data",
4
    "experimental": true,
5
    "log-driver": "json-file",
6
    "log-opts": {
7
        "max-file": "3",
8
        "max-size": "20m"
9
    },
10
    "registry-mirrors": [
11
        "https://docker.1ms.run"
12
    ],
13
    "runtimes": {
14
        "nvidia": {
15
            "args": [],
16
            "path": "nvidia-container-runtime"
17
        }
18
    }
19
}

Test#

1
docker run --rm -it --gpus all ubuntu:22.04 /bin/bash

1
root@catcat:~# docker run --rm -it --gpus all ubuntu:22.04 /bin/bash
2
Unable to find image 'ubuntu:22.04' locally
3
22.04: Pulling from library/ubuntu
4
6414378b6477: Pull complete
5
Digest: sha256:0e5e4a57c2499249aafc3b40fcd541e9a456aab7296681a3994d631587203f97
6
Status: Downloaded newer image for ubuntu:22.04
7
root@e36b1bb454b6:/# nvidia-smi
8
Wed Jan 22 02:03:29 2025
9
+-----------------------------------------------------------------------------------------+
10
| NVIDIA-SMI 550.127.05             Driver Version: 550.127.05     CUDA Version: 12.4     |
11
|-----------------------------------------+------------------------+----------------------+
12
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
13
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
14
|                                         |                        |               MIG M. |
15
|=========================================+========================+======================|
16
|   0  NVIDIA A800-SXM4-80GB          Off |   00000000:23:00.0 Off |                    0 |
17
| N/A   29C    P0             56W /  400W |       4MiB /  81920MiB |      0%      Default |
18
|                                         |                        |             Disabled |
19
+-----------------------------------------+------------------------+----------------------+
20
|   1  NVIDIA A800-SXM4-80GB          Off |   00000000:24:00.0 Off |                    0 |
21
| N/A   29C    P0             56W /  400W |       4MiB /  81920MiB |      0%      Default |
22
|                                         |                        |             Disabled |
23
+-----------------------------------------+------------------------+----------------------+
24
|   2  NVIDIA A800-SXM4-80GB          Off |   00000000:43:00.0 Off |                    0 |
25
| N/A   28C    P0             57W /  400W |       4MiB /  81920MiB |      0%      Default |
26
|                                         |                        |             Disabled |
27
+-----------------------------------------+------------------------+----------------------+
28
|   3  NVIDIA A800-SXM4-80GB          Off |   00000000:44:00.0 Off |                    0 |
29
| N/A   28C    P0             58W /  400W |       4MiB /  81920MiB |      0%      Default |
30
|                                         |                        |             Disabled |
31
+-----------------------------------------+------------------------+----------------------+
32
|   4  NVIDIA A800-SXM4-80GB          Off |   00000000:83:00.0 Off |                    0 |
33
| N/A   28C    P0             57W /  400W |       4MiB /  81920MiB |      0%      Default |
34
|                                         |                        |             Disabled |
35
+-----------------------------------------+------------------------+----------------------+
36
|   5  NVIDIA A800-SXM4-80GB          Off |   00000000:84:00.0 Off |                    0 |
37
| N/A   29C    P0             60W /  400W |       4MiB /  81920MiB |      0%      Default |
38
|                                         |                        |             Disabled |
39
+-----------------------------------------+------------------------+----------------------+
40
|   6  NVIDIA A800-SXM4-80GB          Off |   00000000:C3:00.0 Off |                    0 |
41
| N/A   29C    P0             59W /  400W |       4MiB /  81920MiB |      0%      Default |
42
|                                         |                        |             Disabled |
43
+-----------------------------------------+------------------------+----------------------+
44
|   7  NVIDIA A800-SXM4-80GB          Off |   00000000:C4:00.0 Off |                    0 |
45
| N/A   29C    P0             60W /  400W |       4MiB /  81920MiB |      0%      Default |
46
|                                         |                        |             Disabled |
47
+-----------------------------------------+------------------------+----------------------+
48

49
+-----------------------------------------------------------------------------------------+
50
| Processes:                                                                              |
51
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
52
|        ID   ID                                                               Usage      |
53
|=========================================================================================|
54
|  No running processes found                                                             |
55
+-----------------------------------------------------------------------------------------+

Deploy Open WebUI #

1
version: '3.8'
2

3
services:
4
  open-webui:
5
    image: ghcr.sakiko.de/open-webui/open-webui:main
6
    container_name: open-webui
7
    restart: always
8
    ports:
9
      - "3000:8080"
10
    volumes:
11
      - open-webui:/app/backend/data
12
    extra_hosts:
13
      - "host.docker.internal:host-gateway"
14

15
volumes:
16
  open-webui: