GPU in Docker

  docker 在资源上的分配与隔离限于 CPU/内存/存储。但是对于 GPU 并没有去原生的支持。做 GPU 的厂商除了 nvidia 外,还有 AMD，各家的产品标准都不统一，驱动各异。所以 GPU in docker 还得依靠厂商的支持。

nvidia-docker1.0

在 nvidia 官网有这么一张图：

在单台节点上,docker 使用 GPU 的架构如下：

  在 node 节点上装好 nvidia GPU 对应的驱动后，该节点 共享 宿主机的 GPU 和 GPU 驱动，以达到在容器里面使用 GPU 的效果。nvidia 推出了自家的解决方案 nvidia-docker1.0。

nvidia-docker1.0 使用方法

  nvidia-docker1.0 的用法上，有两种：

1. 用 nvidia-docker 来替换 docker 命令。
   举个例子，原来我们如果要用一个 tensflow 的镜像，命令是这样的：

docker run -itd -p 8888:8888 tensorflow/tensorflow

需要使用 GPU 参与运算的话

nvidia-docker run -itd -p 8888:8888  tensorflow/tensorflow:nightly-devel-gpu-py3 

2. 如果需要非入侵方式

docker run -ti `curl -s http://localhost:3476/v1.0/docker/cli` -p 8890:8888 tensorflow/tensorflow:nightly-devel-gpu-py3

其中 curl -s http://localhost:3476/v1.0/docker/cli 返回的结果：

--volume-driver=nvidia-docker
--volume=nvidia_driver_375.39:/usr/local/nvidia:ro
--device=/dev/nvidiactl
--device=/dev/nvidia-uvm
--device=/dev/nvidia0

大概清楚了 nvidia-docker1.0 干了些什么事了：nvidia-docker/src/nvidia-docker/main.go

if (command == "create" || command == "run") && opt != "" {
		vols, err := VolumesNeeded(opt)
		assert(err)
		if vols != nil {
			var nargs []string
			var err error

			if Host != nil {
				nargs, err = GenerateRemoteArgs(opt, vols)
			} else {
				assert(nvidia.LoadUVM())
				assert(nvidia.Init())
				nargs, err = GenerateLocalArgs(opt, vols)
				nvidia.Shutdown()
			}
			assert(err)
			args = append(args[:off], append(nargs, args[off:]...)...)
		}
	}

而细看--volume-driver=nvidia-docker 创建的 nvidia_driver_375.39，其实是一个本地目录，位置在

root@node3:/var/lib/nvidia-docker/volumes/nvidia_driver/375.39# pwd
/var/lib/nvidia-docker/volumes/nvidia_driver/375.39
root@node3:/var/lib/nvidia-docker/volumes/nvidia_driver/375.39# 

所以在调用 API 做业务处理调用的时候，我们只需要在使用 GPU 的容器上多挂载设备和目录

dataVolumes.add("/var/lib/nvidia-docker/volumes/nvidia_driver/"+host.getNvidiaVersion()+":"+"/usr/local/nvidia:ro");
JSONArray devices = new JSONArray();
devices.add("/dev/nvidiactl:/dev/nvidiactl");
devices.add("/dev/nvidia-uvm:/dev/nvidia-uvm");
for(int i = 0;i < host.getGpuNum();i++){
  devices.add("/dev/nvidia"+i+":"+"/dev/nvidia"+i);
}
re.put("devices", devices);

nvidia-docker2.0

  nvidia-docker2.0 在 2017 年 11 月 14 号 master 切换到 2.0；针对 2.0 的版本 nvidia 是这么说的：

Warning: This project is based on an alpha release (libnvidia-container). It is already more stable than 1.0 but we need help testing it.

安装方式：
nvidia-docker2安装后会覆盖掉/etc/docker/daemon.json。实际上测试在 centos7.0 上安装它会提示我是否覆盖。

• Ubuntu

# If you have nvidia-docker 1.0 installed: we need to remove it and all existing GPU containers
docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 docker ps -q -a -f volume={} | xargs -r docker rm -f
sudo apt-get purge -y nvidia-docker

# Add the package repositories
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \
  sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/nvidia-docker.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update

# Install nvidia-docker2 and reload the Docker daemon configuration
sudo apt-get install -y nvidia-docker2
sudo pkill -SIGHUP dockerd

# Test nvidia-smi with the latest official CUDA image
docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi

• CentOS/RHEL 7 x86_64

# If you have nvidia-docker 1.0 installed: we need to remove it and all existing GPU containers
docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 docker ps -q -a -f volume={} | xargs -r docker rm -f
sudo yum remove nvidia-docker

# Add the package repositories
curl -s -L https://nvidia.github.io/nvidia-docker/centos7/x86_64/nvidia-docker.repo | \
  sudo tee /etc/yum.repos.d/nvidia-docker.repo

# Install nvidia-docker2 and reload the Docker daemon configuration
sudo yum install -y nvidia-docker2
sudo pkill -SIGHUP dockerd

# Test nvidia-smi with the latest official CUDA image
docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi

按照上面的安装方法会导致升级当前的 docker 版本，如果你不想升级当前 docker 版本，官方也给了解决方案：

#You must pin the versions of both nvidia-docker2 and nvidia-container-runtime when installing, for instance:

sudo apt-get install -y nvidia-docker2=2.0.1+docker1.12.6-1 nvidia-container-runtime=1.1.0+docker1.12.6-1

>Use `apt-cache madison nvidia-docker2 nvidia-container-runtime` or `yum search --showduplicates nvidia-docker2 nvidia-container-runtime` to list the available versions.

#### nvidia-docker2.0 使用方法

* 添加指定 --runtime
```shell
# Test nvidia-smi with the latest official CUDA image
docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi

• 设置 default runtime

[root@k8s-master ~]# cat /etc/docker/daemon.json 
{
"insecure-registries": [
	"harbor.hoc.ccshu.net"
],
 "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
  },
"default-runtime": "nvidia"
}
[root@k8s-master ~]# systemctl restart docker

[root@k8s-node1 ~]# docker run -itd --name cuda harbor.hoc.ccshu.net/dev/cuda:9.0-devel-ubuntu16.04
6f0c71d7f1dc50370edeeca046e4da6e2c0def877ae57f8f2263e69291bccf3d
[root@k8s-node1 ~]# docker exec -it cuda /bin/bash
root@6f0c71d7f1dc:/# nvidia-smi
Mon Nov 27 07:54:20 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.73                 Driver Version: 384.73                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla M10           On   | 00000000:00:04.0 Off |                  N/A |
| N/A   47C    P8     8W /  53W |      1MiB /  8127MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@6f0c71d7f1dc:/# 

关于 nvidia-docker2.0 的 GPU 隔离，下次更新。