Kubernetes CSI插件中持久卷的挂载和映射机制

笔者在学习k8s的CSI插件开发的过程中，对于卷的挂载不是很理解，特别是挂载卷这个动作为什么是在一个DaemonSet Pod上操作完成的，这个文章做个总结梳理。

注：gcp-filestore-csi-driver这个CSI插件是gcp（谷歌云）为它自家的Filestore产品提供的一个csi插件。

k8s csi 插件是将卷挂载在Node上的？那是怎么映射到Pod的容器的呢？

通过阅读gcp-filestore-csi-driver和其它的csi插件的源码可以确定，挂载卷是由csi插件的node组件的NodePublishVolume接口实现的，csi插件是将持久卷挂载到Node上的！

当Pod被调度到某个Node上时，Kubelet会负责在该Node上创建和运行Pod。在这个过程中，如果Pod需要挂载持久卷，Kubelet会调用CSI插件的Node组件的NodePublishVolume接口来执行挂载操作。Kubelet在启动容器时，会将Node上的挂载点作为容器卷挂载到容器。（k8s的持久卷!=容器卷，k8s的持久卷是一个抽象的存储资源，与底层存储提供商的具体实现细节解耦。）

举个例子。

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: nfs-storage
  resources:
    requests:
      storage: 1Gi
      
apiVersion: v1
kind: Pod
metadata:
  name: nfs-pod
spec:
  containers:
  - name: my-container
    image: nginx
    volumeMounts:
    - name: nfs-volume
      mountPath: /data
  volumes:
  - name: nfs-volume
    persistentVolumeClaim:
      claimName: nfs-pvc

在这个示例中，我们创建了一个名为 ‘nfs-pod’ 的Pod，其中包含一个名为 ‘my-container’ 的容器。我们在容器中使用了一个名为 ‘nfs-volume’ 的卷，并将其挂载到容器的 ‘/data’ 目录下。在 ‘volumes’ 部分，我们将 ‘nfs-volume’ 卷定义为一个PVC（PersistentVolumeClaim），并指定了 ‘nfs-pvc’ 作为PVC的名称。

从这个例子可以看出，我们并没有指定挂载在Node上的路径，那么csi插件是怎么决定挂载在Node上的路径的？

当我们创建一个PVC对象时，Kubernetes会根据PVC的定义和StorageClass的配置来动态调用CSI插件创建一个持久卷（PersistentVolume）。只有在声明使用了这个PVC的Pod部署的时候，kubernetes才会将持久卷挂载到Pod将要部署的Node上的一个路径。具体挂载在Node上的路径是由Kubernetes决定的。

持久卷的详细挂载过程，以及StagingTargetPath和TargetPath的理解

我们通过debug gcp-filestore-csi-driver 这个csi插件来理解。

首先是卷的创建阶段，创建阶段由k8s监听到PVC资源的创建然后调用csi插件的controller组件的CreateVolume接口。

controller组件输出的日志如下：

## CreateVolume
I0520 09:45:57.873175       1 controller.go:191] CreateVolume called with request name:"pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa" capacity_range:<required_bytes:1000000000000 > volume_capabilities:<mount:<mount_flags:"rw" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > parameters:<key:"csi.storage.k8s.io/pv/name" value:"pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa" > parameters:<key:"csi.storage.k8s.io/pvc/name" value:"test-demo-pvc" > parameters:<key:"csi.storage.k8s.io/pvc/namespace" value:"default" > parameters:<key:"network" value:"default" > parameters:<key:"tier" value:"BASIC_HDD" > accessibility_requirements:<requisite:<segments:<key:"topology.gke.io/zone" value:"us-central1-c" > > preferred:<segments:<key:"topology.gke.io/zone" value:"us-central1-c" > > > 
W0520 09:45:57.873460       1 controller.go:513] required bytes 0.9094947017729282TiB is less than minimum instance size capacity 1TiB for tier BASIC_HDD, but no upper bound was specified. Rounding up capacity request to 1TiB for tier BASIC_HDD.
E0520 09:45:57.957996       1 file.go:321] Failed to get instance projects/wujiuye-410808/locations/us-central1-c/instances/pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa
I0520 09:45:57.958278       1 file.go:286] Creating instance "pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa": location us-central1-c, tier "BASIC_HDD", capacity 1024, network "default", ipRange "", connectMode "DIRECT_PEERING", KmsKeyName "", labels map[kubernetes_io_created-for_pv_name:pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa kubernetes_io_created-for_pvc_name:test-demo-pvc kubernetes_io_created-for_pvc_namespace:default storage_gke_io_created-by:filestore_csi_storage_gke_io] backup source ""
I0520 09:46:02.471478       1 file.go:303] For instance pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa, waiting for create instance op projects/wujiuye-410808/locations/us-central1-c/operations/operation-1716198357999-618df917e1eb3-c8e3a6ea-e858bd38 to complete
I0520 09:49:27.527838       1 file.go:326] GetInstance call fetched instance &{CapacityGb:1024 CapacityStepSizeGb:1 CreateTime:2024-05-20T09:46:02.447035850Z Description: DirectoryServices:<nil> Etag: FileShares:[0xc00030c000] KmsKeyName: Labels:map[kubernetes_io_created-for_pv_name:pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa kubernetes_io_created-for_pvc_name:test-demo-pvc kubernetes_io_created-for_pvc_namespace:default storage_gke_io_created-by:filestore_csi_storage_gke_io] MaxCapacityGb:65434 MaxShareCount:1 MultiShareEnabled:false Name:projects/wujiuye-410808/locations/us-central1-c/instances/pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa Networks:[0xc00044bef0] Protocol: SatisfiesPzi:true SatisfiesPzs:false State:READY StatusMessage: SuspensionReasons:[] Tier:BASIC_HDD ServerResponse:{HTTPStatusCode:200 Header:map[Cache-Control:[private] Content-Type:[application/json; charset=UTF-8] Date:[Mon, 20 May 2024 09:49:27 GMT] Server:[ESF] Vary:[Origin X-Origin Referer] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]]} ForceSendFields:[] NullFields:[]}
I0520 09:49:27.528197       1 controller.go:316] CreateVolume succeeded: volume:<capacity_bytes:1099511627776 volume_id:"modeInstance/us-central1-c/pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa/vol1" volume_context:<key:"ip" value:"1.2.3.4" > volume_context:<key:"volume" value:"vol1" > >

这一步是根据pvc资源创建持久卷，这个案例是创建一个gcp的Filestore(nfs)实例，等待创建成功后，得到nfs服务器的ip和共享目录，并将ip（<key:"ip" value:"1.2.3.4" >）和共享目录（<key:"volume" value:"vol1" >）写入VolumeContext响应给k8s。

该方法调用结束后，k8s会自动创建出一个PV资源，并且会自动将PV与PVC绑定。

其次，如果有声明使用这个PVC资源的Pod部署，在Pod由调度器选择出调度到哪个Node上之后，在k8s调用Node上的kubelet部署Pod后，kubelet调用csi插件的node组件（kubelet和csi的node组件在同一Node上）的NodeStageVolume和NodePublishVolume接口，完成卷的挂载。

node的日志如下：

## NodeStageVolume
I0520 09:49:38.298597       1 utils.go:55] GRPC call: /csi.v1.Node/NodeStageVolume
I0520 09:49:38.298868       1 utils.go:56] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/filestore.csi.storage.gke.io/270b9be2ea0df2b0c8b6f6871b7725c553ad5e5f5d3a43e496a85b671461c861/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["rw"]}},"access_mode":{"mode":5}},"volume_context":{"ip":"1.2.3.4","storage.kubernetes.io/csiProvisionerIdentity":"1716198288619-8405-filestore.csi.storage.gke.io","volume":"vol1"},"volume_id":"modeInstance/us-central1-c/pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa/vol1"}
I0520 09:49:38.299239       1 mount_linux.go:243] Detected OS without systemd
I0520 09:49:38.299279       1 mount_linux.go:218] Mounting cmd (mount) with arguments (-t nfs -o rw 10.26.141.122:/vol1 /var/lib/kubelet/plugins/kubernetes.io/csi/filestore.csi.storage.gke.io/270b9be2ea0df2b0c8b6f6871b7725c553ad5e5f5d3a43e496a85b671461c861/globalmount)
I0520 09:49:41.754258       1 node.go:344] NodeStageVolume succeeded on volume modeInstance/us-central1-c/pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa/vol1 to path /var/lib/kubelet/plugins/kubernetes.io/csi/filestore.csi.storage.gke.io/270b9be2ea0df2b0c8b6f6871b7725c553ad5e5f5d3a43e496a85b671461c861/globalmount

## NodePublishVolume
I0520 09:49:41.762556       1 utils.go:55] GRPC call: /csi.v1.Node/NodePublishVolume
I0520 09:49:41.762710       1 utils.go:56] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/filestore.csi.storage.gke.io/270b9be2ea0df2b0c8b6f6871b7725c553ad5e5f5d3a43e496a85b671461c861/globalmount","target_path":"/var/lib/kubelet/pods/82da039b-0f63-4923-aa1d-230e14a2855e/volumes/kubernetes.io~csi/pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["rw"]}},"access_mode":{"mode":5}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"go-web-demo-756986cc45-q6scd","csi.storage.k8s.io/pod.namespace":"default","csi.storage.k8s.io/pod.uid":"82da039b-0f63-4923-aa1d-230e14a2855e","csi.storage.k8s.io/serviceAccount.name":"default","ip":"1.2.3.4","storage.kubernetes.io/csiProvisionerIdentity":"1716198288619-8405-filestore.csi.storage.gke.io","volume":"vol1"},"volume_id":"modeInstance/us-central1-c/pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa/vol1"}
I0520 09:49:41.762812       1 mount_linux.go:218] Mounting cmd (mount) with arguments (-t nfs -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/filestore.csi.storage.gke.io/270b9be2ea0df2b0c8b6f6871b7725c553ad5e5f5d3a43e496a85b671461c861/globalmount /var/lib/kubelet/pods/82da039b-0f63-4923-aa1d-230e14a2855e/volumes/kubernetes.io~csi/pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa/mount)
I0520 09:49:41.766774       1 mount_linux.go:218] Mounting cmd (mount) with arguments (-t nfs -o bind,remount,rw /var/lib/kubelet/plugins/kubernetes.io/csi/filestore.csi.storage.gke.io/270b9be2ea0df2b0c8b6f6871b7725c553ad5e5f5d3a43e496a85b671461c861/globalmount /var/lib/kubelet/pods/82da039b-0f63-4923-aa1d-230e14a2855e/volumes/kubernetes.io~csi/pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa/mount)
I0520 09:49:41.769093       1 node.go:168] Successfully mounted /var/lib/kubelet/pods/82da039b-0f63-4923-aa1d-230e14a2855e/volumes/kubernetes.io~csi/pvc-dcba2328-caad-4dc6-bc70-bd7c5abf4faa/mount

从日记可以看出，StagingTargetPath这个路径是k8s生成的，在调用NodeStageVolume接口时，csi插件将NFS文件系统的共享目录挂载到了StagingTargetPath这个目录下，插件通过VolumeContext获取CreateVolume写入的NFS服务的ip和共享路径。

接着调用NodePublishVolume方法，StagingTargetPath目录同调用NodeStageVolume时传的目录，TargetPath也是k8s生成的一个目录。StagingTargetPath目录挂载了NFS持久卷。NodePublishVolume方法是将StagingTargetPath又挂载到TargetPath。

挂载动作在linux系统上是通过调用mount命令完成的。

那么为什么gcp-filestore-csi-driver 将持久卷挂载到StagingTargetPath，之后又挂载到TargetPath，而不是直接挂载到TargetPath呢？StagingTargetPath是一个全局目录，是可以给当前节点上多个使用相同PVC的Pod一同使用的。

注：是否会调用NodeStageVolume方法，其实是由NodeGetCapabilities接口的返回值实现的。如果csi驱动未实现NodeGetCapabilities接口，那么NodeStageVolume接口就不会被调用，只会调用NodePublishVolume方法。具体我们在下一篇介绍如何实现一个csi驱动时再介绍。