本文适用于 HGX A800 / Tesla 系列 NVIDIA 数据中心 GPU。

1. 官方资源链接#

2. 系统基础配置#

2.1 更新软件源#

1
bash <(curl -sSL https://linuxmirrors.cn/main.sh)

2.2 设置主机名#

1
hostnamectl set-hostname ubuntu-2204

2.3 禁用 Nouveau#

1
sudo bash -c "echo blacklist nouveau >> /etc/modprobe.d/blacklist.conf"
2
sudo bash -c "echo options nouveau modeset=0 >> /etc/modprobe.d/blacklist.conf"
3

4
sudo update-initramfs -u
5
reboot

验证（无输出即成功）：

1
lsmod | grep nouveau

3. 图形环境与依赖#

（可选）

1
sudo apt update
2
sudo apt install lightdm
3
sudo apt install gcc make dkms

查看可用驱动：

1
apt install ubuntu-drivers-common
2
ubuntu-drivers devices

4. 安装 NVIDIA 驱动#

以 HGX A800 + 550 驱动为例：

1
wget https://cn.download.nvidia.cn/tesla/550.54.15/NVIDIA-Linux-x86_64-550.54.15.run
2
chmod +x NVIDIA-Linux-x86_64-550.54.15.run
3

4
sudo ./NVIDIA-Linux-x86_64-550.54.15.run \
5
  -no-x-check -no-nouveau-check -no-opengl-files

验证：

1
nvidia-smi

5. 卸载 NVIDIA 驱动#

1
sudo apt-get --purge remove nvidia*
2
sudo apt autoremove
3
sudo apt-get --purge remove "*nvidia*"
4
nvidia-uninstall
5
dpkg -l | grep nvidia
6
reboot

6. 安装 CUDA Toolkit#

示例（CUDA 12.4）：

1
wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda_12.4.0_550.54.14_linux.run
2

3
sudo sh cuda_12.4.0_550.54.14_linux.run

安装时选择：

接受协议
取消安装驱动
安装 CUDA

7. 配置 CUDA 环境变量#

1
echo 'export PATH=$PATH:/usr/local/cuda-12.4/bin' >> ~/.bashrc
2
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-12.4/lib64' >> ~/.bashrc
3
source ~/.bashrc

验证：

1
nvcc -V

8. 卸载 CUDA#

1
cd /usr/local/cuda-12.4/bin
2
sudo ./uninstall_cuda_12.4.pl
3
sudo rm -rf /usr/local/cuda-12.4

9. 安装 IB 驱动 (MLNX OFED)#

示例 Ubuntu 22.04：

1
wget https://content.mellanox.com/ofed/MLNX_OFED-24.04-0.6.6.0/MLNX_OFED_LINUX-24.04-0.6.6.0-ubuntu22.04-x86_64.tgz
2
tar -zxvf MLNX_OFED_LINUX-24.04-0.6.6.0-ubuntu22.04-x86_64.tgz
3
cd MLNX_OFED_LINUX-24.04-0.6.6.0-ubuntu22.04-x86_64
4

5
sudo ./mlnxofedinstall
6
sudo /etc/init.d/openibd restart

验证：

1
ibdev2netdev
2
ibstatus

如无安装：

1
apt install infiniband-diags

10. 关键模块检查#

1
modprobe nvidia_peermem
2
apt-get install nvidia-modprobe
3
modprobe nvidia_peermem
4

5
lsmod | grep nvidia

正常输出示例：

1
nvidia_peermem
2
nvidia_drm
3
nvidia_modeset
4
nvidia

11. 安装 OpenMPI#

1
apt install -y gcc g++ make hwloc hwloc-nox libevent-dev
2

3
wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.5.tar.gz
4
tar zxf openmpi-4.1.5.tar.gz
5
cd openmpi-4.1.5
6

7
./configure --prefix=/usr/local/openmpi
8
make -j$(nproc)
9
make install

环境变量：

1
echo 'export PATH=$PATH:/usr/local/openmpi/bin' >> ~/.bashrc
2
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/openmpi/lib' >> ~/.bashrc
3
source ~/.bashrc

验证：

1
mpiexec --version

12. 配置 IB 网卡 (Netplan)#

示例（多 IB 卡）：

1
network:
2
  version: 2
3
  ethernets:
4
    eth0:
5
      dhcp4: true
6
      match:
7
        macaddress: 52:54:00:13:3f:6d
8
      set-name: eth0
9
    ib0:
10
      dhcp4: no
11
      addresses: [10.1.12.1/24]
12
    ib1:
13
      dhcp4: no
14
      addresses: [10.1.12.2/24]
15
    ib2:
16
      dhcp4: no
17
      addresses: [10.1.12.3/24]
18
    ib3:
19
      dhcp4: no
20
      addresses: [10.1.12.4/24]

应用：

1
netplan apply

13. 安装 NCCL#

13.1 NVIDIA Repository Keyring#

Ubuntu 22.04：

1
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb
2
sudo dpkg -i cuda-keyring_1.0-1_all.deb
3
sudo apt update

13.2 安装特定版本 NCCL#

1
sudo apt install libnccl2=2.20.5-1+cuda12.4 \
2
                 libnccl-dev=2.20.5-1+cuda12.4

13.3 环境变量#

1
vim /etc/profile

加入：

1
NCCL_IB_HCA=mlx5_0:1,mlx5_1:1,mlx5_6:1,mlx5_7:1
2
NCCL_IB_DISABLE=0
3
NCCL_SOCKET_IFNAME=eth0
4
NCCL_IB_GID_INDEX=3
5
NCCL_NET_GDR_LEVEL=2
6
NCCL_DEBUG=INFO

启用：

1
source /etc/profile

验证：

1
ldconfig -v | grep "libnccl.so"

14. 安装 NCCL Tests#

1
git clone https://github.com/NVIDIA/nccl-tests.git
2
cd nccl-tests
3
make MPI=1 MPI_HOME=/usr/local/openmpi

15. 安装 FabricManager#

Ubuntu 22.04 示例#

1
wget https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2204/x86_64/nvidia-fabricmanager-dev-550_550.54.15-1_amd64.deb
2
wget https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2204/x86_64/nvidia-fabricmanager-550_550.54.15-1_amd64.deb
3

4
sudo dpkg -i nvidia-fabricmanager-550_550.54.15-1_amd64.deb
5
sudo dpkg -i nvidia-fabricmanager-dev-550_550.54.15-1_amd64.deb
6

7
systemctl start nvidia-fabricmanager
8
systemctl enable nvidia-fabricmanager
9
systemctl status nvidia-fabricmanager

16. 单机 NCCL 测试#

1
./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8

17. 多机测试#

17.1 SSH 免密#

1
ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
2
chmod 600 ~/.ssh/id_rsa
3
scp ~/.ssh/id_rsa.pub [email protected]:~/.ssh/authorized_keys

17.2 NCCL + MPI 多机测试示例#

1
mpirun --oversubscribe --allow-run-as-root \
2
  -mca plm_rsh_args "-p 22 -q -o StrictHostKeyChecking=no" \
3
  -n 16 -N 8 \
4
  -H node1,node2 \
5
  -bind-to socket -map-by slot \
6
  -mca pml ob1 -mca btl ^openib \
7
  -mca orte_base_help_aggregate 0 \
8
  -mca btl_tcp_if_include eth0 \
9
  -mca coll_hcoll_enable 0 \
10
  -x NCCL_DEBUG=INFO \
11
  -x NCCL_SOCKET_IFNAME=eth0 \
12
  -x NCCL_IB_DISABLE=0 \
13
  -x NCCL_NET_GDR_LEVEL=2 \
14
  -x NCCL_IB_HCA=mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1 \
15
  -x NCCL_IB_GID_INDEX=3 \
16
  ~/nccl-tests/build/all_reduce_perf \
17
  -b 256M -e 8G -f 2 -g 1 -c 1 -n 100

18. 常用组件说明#

NVIDIA 组件#

GPU Driver：驱动 GPU 工作
CUDA Toolkit：GPU 通用计算工具链
cuDNN：深度学习 GPU 加速库

OpenMPI#

用于多机通信的 MPI 实现。

NCCL#

NVIDIA 集合通信库，多 GPU / 多机高效通信。

NCCL Tests#

NCCL 性能与正确性测试工具。

FabricManager#

管理 NVSwitch / NVLink 拓扑，HGX 服务器必需。

Ubuntu NVIDIA GPU + CUDA + NCCL + OFED + OpenMPI 高性能计算环境完整安装指南