• 0. 说明
    • 涉及镜像
  • 1. 部署RBAC
    • 1.1. csi-attacher-rbac.yaml
    • 1.2. csi-provisioner-rbac.yaml
    • 1.3. csi-nodeplugin-rbac.yaml
  • 2. 部署CSI sidecar containers
    • 2.1. csi-cephfsplugin-provisioner.yaml
    • 2.2. csi-cephfsplugin-attacher.yaml
  • 3. 部署CSI-CephFS-driver(plugin)
    • 3.1. csi-cephfsplugin.yaml
  • 4. 确认部署结果

    0. 说明

    要求Kubernetes的版本在1.11及以上，k8s集群必须允许特权Pod（privileged pods），即apiserver和kubelet需要设置--allow-privileged为true。节点的Docker daemon需要允许挂载共享卷。

    涉及镜像

    • quay.io/k8scsi/csi-provisioner:v0.3.0
    • quay.io/k8scsi/csi-attacher:v0.3.0
    • quay.io/k8scsi/driver-registrar:v0.3.0
    • quay.io/cephcsi/cephfsplugin:v0.3.0

    1. 部署RBAC

    部署service accounts, cluster roles 和 cluster role bindings，这些可供RBD和CephFS CSI plugins共同使用，他们拥有相同的权限。

    1. $ kubectl create -f csi-attacher-rbac.yaml
    2. $ kubectl create -f csi-provisioner-rbac.yaml
    3. $ kubectl create -f csi-nodeplugin-rbac.yaml

    1.1. csi-attacher-rbac.yaml

    1. apiVersion: v1
    2. kind: ServiceAccount
    3. metadata:
    4.   name: csi-attacher
    6. ---
    7. kind: ClusterRole
    8. apiVersion: rbac.authorization.k8s.io/v1
    9. metadata:
    10.   name: external-attacher-runner
    11. rules:
    12.   - apiGroups: [""]
    13.     resources: ["events"]
    14.     verbs: ["get", "list", "watch", "update"]
    15.   - apiGroups: [""]
    16.     resources: ["persistentvolumes"]
    17.     verbs: ["get", "list", "watch", "update"]
    18.   - apiGroups: [""]
    19.     resources: ["nodes"]
    20.     verbs: ["get", "list", "watch"]
    21.   - apiGroups: ["storage.k8s.io"]
    22.     resources: ["volumeattachments"]
    23.     verbs: ["get", "list", "watch", "update"]
    25. ---
    26. kind: ClusterRoleBinding
    27. apiVersion: rbac.authorization.k8s.io/v1
    28. metadata:
    29.   name: csi-attacher-role
    30. subjects:
    31.   - kind: ServiceAccount
    32.     name: csi-attacher
    33.     namespace: default
    34. roleRef:
    35.   kind: ClusterRole
    36.   name: external-attacher-runner
    37.   apiGroup: rbac.authorization.k8s.io

    1.2. csi-provisioner-rbac.yaml

    1. apiVersion: v1
    2. kind: ServiceAccount
    3. metadata:
    4.   name: csi-provisioner
    6. ---
    7. kind: ClusterRole
    8. apiVersion: rbac.authorization.k8s.io/v1
    9. metadata:
    10.   name: external-provisioner-runner
    11. rules:
    12.   - apiGroups: [""]
    13.     resources: ["secrets"]
    14.     verbs: ["get", "list"]
    15.   - apiGroups: [""]
    16.     resources: ["persistentvolumes"]
    17.     verbs: ["get", "list", "watch", "create", "delete"]
    18.   - apiGroups: [""]
    19.     resources: ["persistentvolumeclaims"]
    20.     verbs: ["get", "list", "watch", "update"]
    21.   - apiGroups: ["storage.k8s.io"]
    22.     resources: ["storageclasses"]
    23.     verbs: ["get", "list", "watch"]
    24.   - apiGroups: [""]
    25.     resources: ["events"]
    26.     verbs: ["list", "watch", "create", "update", "patch"]
    28. ---
    29. kind: ClusterRoleBinding
    30. apiVersion: rbac.authorization.k8s.io/v1
    31. metadata:
    32.   name: csi-provisioner-role
    33. subjects:
    34.   - kind: ServiceAccount
    35.     name: csi-provisioner
    36.     namespace: default
    37. roleRef:
    38.   kind: ClusterRole
    39.   name: external-provisioner-runner
    40.   apiGroup: rbac.authorization.k8s.io

    1.3. csi-nodeplugin-rbac.yaml

    1. apiVersion: v1
    2. kind: ServiceAccount
    3. metadata:
    4.   name: csi-nodeplugin
    6. ---
    7. kind: ClusterRole
    8. apiVersion: rbac.authorization.k8s.io/v1
    9. metadata:
    10.   name: csi-nodeplugin
    11. rules:
    12.   - apiGroups: [""]
    13.     resources: ["nodes"]
    14.     verbs: ["get", "list", "update"]
    15.   - apiGroups: [""]
    16.     resources: ["namespaces"]
    17.     verbs: ["get", "list"]
    18.   - apiGroups: [""]
    19.     resources: ["persistentvolumes"]
    20.     verbs: ["get", "list", "watch", "update"]
    21.   - apiGroups: ["storage.k8s.io"]
    22.     resources: ["volumeattachments"]
    23.     verbs: ["get", "list", "watch", "update"]
    25. ---
    26. kind: ClusterRoleBinding
    27. apiVersion: rbac.authorization.k8s.io/v1
    28. metadata:
    29.   name: csi-nodeplugin
    30. subjects:
    31.   - kind: ServiceAccount
    32.     name: csi-nodeplugin
    33.     namespace: default
    34. roleRef:
    35.   kind: ClusterRole
    36.   name: csi-nodeplugin
    37.   apiGroup: rbac.authorization.k8s.io

    2. 部署CSI sidecar containers

    通过StatefulSet的方式部署external-attacher和external-provisioner供CSI CephFS使用。

    1. $ kubectl create -f csi-cephfsplugin-attacher.yaml
    2. $ kubectl create -f csi-cephfsplugin-provisioner.yaml

    2.1. csi-cephfsplugin-provisioner.yaml

    1. kind: Service
    2. apiVersion: v1
    3. metadata:
    4.   name: csi-cephfsplugin-provisioner
    5.   labels:
    6.     app: csi-cephfsplugin-provisioner
    7. spec:
    8.   selector:
    9.     app: csi-cephfsplugin-provisioner
    10.   ports:
    11.     - name: dummy
    12.       port: 12345
    14. ---
    15. kind: StatefulSet
    16. apiVersion: apps/v1beta1
    17. metadata:
    18.   name: csi-cephfsplugin-provisioner
    19. spec:
    20.   serviceName: "csi-cephfsplugin-provisioner"
    21.   replicas: 1
    22.   template:
    23.     metadata:
    24.       labels:
    25.         app: csi-cephfsplugin-provisioner
    26.     spec:
    27.       serviceAccount: csi-provisioner
    28.       containers:
    29.         - name: csi-provisioner
    30.           image: quay.io/k8scsi/csi-provisioner:v0.3.0
    31.           args:
    32.             - "--provisioner=csi-cephfsplugin"
    33.             - "--csi-address=$(ADDRESS)"
    34.             - "--v=5"
    35.           env:
    36.             - name: ADDRESS
    37.               value: /var/lib/kubelet/plugins/csi-cephfsplugin/csi.sock
    38.           imagePullPolicy: "IfNotPresent"
    39.           volumeMounts:
    40.             - name: socket-dir
    41.               mountPath: /var/lib/kubelet/plugins/csi-cephfsplugin
    42.       volumes:
    43.         - name: socket-dir
    44.           hostPath:
    45.             path: /var/lib/kubelet/plugins/csi-cephfsplugin
    46.             type: DirectoryOrCreate

    2.2. csi-cephfsplugin-attacher.yaml

    1. kind: Service
    2. apiVersion: v1
    3. metadata:
    4.   name: csi-cephfsplugin-attacher
    5.   labels:
    6.     app: csi-cephfsplugin-attacher
    7. spec:
    8.   selector:
    9.     app: csi-cephfsplugin-attacher
    10.   ports:
    11.     - name: dummy
    12.       port: 12345
    14. ---
    15. kind: StatefulSet
    16. apiVersion: apps/v1beta1
    17. metadata:
    18.   name: csi-cephfsplugin-attacher
    19. spec:
    20.   serviceName: "csi-cephfsplugin-attacher"
    21.   replicas: 1
    22.   template:
    23.     metadata:
    24.       labels:
    25.         app: csi-cephfsplugin-attacher
    26.     spec:
    27.       serviceAccount: csi-attacher
    28.       containers:
    29.         - name: csi-cephfsplugin-attacher
    30.           image: quay.io/k8scsi/csi-attacher:v0.3.0
    31.           args:
    32.             - "--v=5"
    33.             - "--csi-address=$(ADDRESS)"
    34.           env:
    35.             - name: ADDRESS
    36.               value: /var/lib/kubelet/plugins/csi-cephfsplugin/csi.sock
    37.           imagePullPolicy: "IfNotPresent"
    38.           volumeMounts:
    39.             - name: socket-dir
    40.               mountPath: /var/lib/kubelet/plugins/csi-cephfsplugin
    41.       volumes:
    42.         - name: socket-dir
    43.           hostPath:
    44.             path: /var/lib/kubelet/plugins/csi-cephfsplugin
    45.             type: DirectoryOrCreate

    3. 部署CSI-CephFS-driver(plugin)

    csi-cephfs-plugin 的作用类似nfs-client，部署在所有node节点上，执行ceph的挂载等相关任务。

    通过DaemonSet的方式部署，其中包括两个容器：CSI driver-registrar 和 CSI CephFS driver。

    1. $ kubectl create -f csi-cephfsplugin.yaml

    3.1. csi-cephfsplugin.yaml

    1. kind: DaemonSet
    2. apiVersion: apps/v1beta2
    3. metadata:
    4.   name: csi-cephfsplugin
    5. spec:
    6.   selector:
    7.     matchLabels:
    8.       app: csi-cephfsplugin
    9.   template:
    10.     metadata:
    11.       labels:
    12.         app: csi-cephfsplugin
    13.     spec:
    14.       serviceAccount: csi-nodeplugin
    15.       hostNetwork: true
    16.       # to use e.g. Rook orchestrated cluster, and mons' FQDN is
    17.       # resolved through k8s service, set dns policy to cluster first
    18.       dnsPolicy: ClusterFirstWithHostNet      
    19.       containers:
    20.         - name: driver-registrar
    21.           image: quay.io/k8scsi/driver-registrar:v0.3.0
    22.           args:
    23.             - "--v=5"
    24.             - "--csi-address=$(ADDRESS)"
    25.             - "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)"
    26.           env:
    27.             - name: ADDRESS
    28.               value: /var/lib/kubelet/plugins/csi-cephfsplugin/csi.sock
    29.             - name: DRIVER_REG_SOCK_PATH
    30.               value: /var/lib/kubelet/plugins/csi-cephfsplugin/csi.sock
    31.             - name: KUBE_NODE_NAME
    32.               valueFrom:
    33.                 fieldRef:
    34.                   fieldPath: spec.nodeName
    35.           volumeMounts:
    36.             - name: socket-dir
    37.               mountPath: /var/lib/kubelet/plugins/csi-cephfsplugin
    38.             - name: registration-dir
    39.               mountPath: /registration
    40.         - name: csi-cephfsplugin
    41.           securityContext:
    42.             privileged: true
    43.             capabilities:
    44.               add: ["SYS_ADMIN"]
    45.             allowPrivilegeEscalation: true
    46.           image: quay.io/cephcsi/cephfsplugin:v0.3.0
    47.           args :
    48.             - "--nodeid=$(NODE_ID)"
    49.             - "--endpoint=$(CSI_ENDPOINT)"
    50.             - "--v=5"
    51.             - "--drivername=csi-cephfsplugin"
    52.           env:
    53.             - name: NODE_ID
    54.               valueFrom:
    55.                 fieldRef:
    56.                   fieldPath: spec.nodeName
    57.             - name: CSI_ENDPOINT
    58.               value: unix://var/lib/kubelet/plugins/csi-cephfsplugin/csi.sock
    59.           imagePullPolicy: "IfNotPresent"
    60.           volumeMounts:
    61.             - name: plugin-dir
    62.               mountPath: /var/lib/kubelet/plugins/csi-cephfsplugin
    63.             - name: pods-mount-dir
    64.               mountPath: /var/lib/kubelet/pods
    65.               mountPropagation: "Bidirectional"
    66.             - mountPath: /sys
    67.               name: host-sys
    68.             - name: lib-modules
    69.               mountPath: /lib/modules
    70.               readOnly: true
    71.             - name: host-dev
    72.               mountPath: /dev
    73.       volumes:
    74.         - name: plugin-dir
    75.           hostPath:
    76.             path: /var/lib/kubelet/plugins/csi-cephfsplugin
    77.             type: DirectoryOrCreate
    78.         - name: registration-dir
    79.           hostPath:
    80.             path: /var/lib/kubelet/plugins/
    81.             type: Directory
    82.         - name: pods-mount-dir
    83.           hostPath:
    84.             path: /var/lib/kubelet/pods
    85.             type: Directory
    86.         - name: socket-dir
    87.           hostPath:
    88.             path: /var/lib/kubelet/plugins/csi-cephfsplugin
    89.             type: DirectoryOrCreate
    90.         - name: host-sys
    91.           hostPath:
    92.             path: /sys
    93.         - name: lib-modules
    94.           hostPath:
    95.             path: /lib/modules
    96.         - name: host-dev
    97.           hostPath:
    98.             path: /dev

    4. 确认部署结果

    1. $ kubectl get all
    2. NAME                                 READY     STATUS    RESTARTS   AGE
    3. pod/csi-cephfsplugin-attacher-0      1/1       Running   0          26s
    4. pod/csi-cephfsplugin-provisioner-0   1/1       Running   0          25s
    5. pod/csi-cephfsplugin-rljcv           2/2       Running   0          24s
    7. NAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
    8. service/csi-cephfsplugin-attacher      ClusterIP   10.104.116.218   <none>        12345/TCP   27s
    9. service/csi-cephfsplugin-provisioner   ClusterIP   10.101.78.75     <none>        12345/TCP   26s
    11. ...

    参考文档：

    • https://github.com/ceph/ceph-csi
    • https://github.com/ceph/ceph-csi/blob/master/docs/deploy-cephfs.md
    • https://github.com/ceph/ceph-csi/tree/master/deploy/cephfs/kubernetes