K8S 部署 Redis 主从

Redis

应该是我们比较常用的了吧，不管是缓存，还是消息队列，或者是快速查询的短存储，都经常用到。这篇主要是进行在 K8S 上进行 Redis 的主从部署。不讲太多了，大部分都是已经写好的配置文件，直接奉上。

Redis PV

首先创建 Redis－ＰV，打算创建三个节点的主从，配置如下
redis-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-redis-sentinel-0
  namespace: database
spec:
  capacity:
    storage: 4Gi
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: "redis-sentinel-storage-class"
  nfs:
    # real share directory
    path: /data/nfs/redis-cluster/0
    # nfs real ip
    server: nfs ip

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-redis-sentinel-1
  namespace: database
spec:
  capacity:
    storage: 4Gi
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: "redis-sentinel-storage-class"
  nfs:
    # real share directory
    path: /data/nfs/redis-cluster/1
    # nfs real ip
    server: nfs ip

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-redis-sentinel-2
  namespace: database
spec:
  capacity:
    storage: 4Gi
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: "redis-sentinel-storage-class"
  nfs:
    # real share directory
    path: /data/nfs/redis-cluster/2
    # nfs real ip
    server: nfs ip
​

依旧是用了 NFS

kubectl apply -f redis-pv.yaml
​

Redis ConfigMap

这里部署 Redis　Config，直接上文件
redis-config.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: redis-sentinel-config
  namespace: database
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
    redis-master.conf: |
      port 6379
      tcp-backlog 511
      timeout 0
      tcp-keepalive 0
      loglevel notice
      databases 16
      save 900 1
      save 300 10
      save 60 10000
      stop-writes-on-bgsave-error yes
      rdbcompression yes
      rdbchecksum yes
      dbfilename dump.rdb
      dir /data/
      slave-serve-stale-data yes
      repl-diskless-sync no
      repl-diskless-sync-delay 5
      repl-disable-tcp-nodelay no
      slave-priority 100
      appendonly no
      appendfilename "appendonly.aof"
      appendfsync everysec
      no-appendfsync-on-rewrite no
      auto-aof-rewrite-percentage 100
      auto-aof-rewrite-min-size 64mb
      aof-load-truncated yes
      lua-time-limit 5000
      slowlog-log-slower-than 10000
      slowlog-max-len 128
      latency-monitor-threshold 0
      notify-keyspace-events ""
      hash-max-ziplist-entries 512
      hash-max-ziplist-value 64
      list-max-ziplist-entries 512
      list-max-ziplist-value 64
      set-max-intset-entries 512
      zset-max-ziplist-entries 128
      zset-max-ziplist-value 64
      hll-sparse-max-bytes 3000
      activerehashing yes
      client-output-buffer-limit normal 0 0 0
      client-output-buffer-limit slave 256mb 64mb 60
      client-output-buffer-limit pubsub 64mb 16mb 60
      hz 10
      aof-rewrite-incremental-fsync yes
    redis-slave.conf: |
      port 6379
      slaveof redis-sentinel-master-ss-0.redis-sentinel-master-ss.database.svc.cluster.local 6379
      tcp-backlog 511
      timeout 0
      tcp-keepalive 0
      loglevel notice
      databases 16
      save 900 1
      save 300 10
      save 60 10000
      stop-writes-on-bgsave-error yes
      rdbcompression yes
      rdbchecksum yes
      dbfilename dump.rdb
      dir /data/
      slave-serve-stale-data yes
      slave-read-only yes
      repl-diskless-sync no
      repl-diskless-sync-delay 5
      repl-disable-tcp-nodelay no
      slave-priority 100
      appendonly no
      appendfilename "appendonly.aof"
      appendfsync everysec
      no-appendfsync-on-rewrite no
      auto-aof-rewrite-percentage 100
      auto-aof-rewrite-min-size 64mb
      aof-load-truncated yes
      lua-time-limit 5000
      slowlog-log-slower-than 10000
      slowlog-max-len 128
      latency-monitor-threshold 0
      notify-keyspace-events ""
      hash-max-ziplist-entries 512
      hash-max-ziplist-value 64
      list-max-ziplist-entries 512
      list-max-ziplist-value 64
      set-max-intset-entries 512
      zset-max-ziplist-entries 128
      zset-max-ziplist-value 64
      hll-sparse-max-bytes 3000
      activerehashing yes
      client-output-buffer-limit normal 0 0 0
      client-output-buffer-limit slave 256mb 64mb 60
      client-output-buffer-limit pubsub 64mb 16mb 60
      hz 10
      aof-rewrite-incremental-fsync yes
    redis-sentinel.conf: |
      port 26379
      dir /data
      sentinel monitor mymaster redis-sentinel-master-ss-0.redis-sentinel-master-ss.database.svc.cluster.local 6379 2
      sentinel down-after-milliseconds mymaster 30000
      sentinel parallel-syncs mymaster 1
      sentinel failover-timeout mymaster 180000
​

Redis 的相关配置。

kubectl apply -f redis-config.yaml
​

Redis Service

这里我们来创建 Redis 的服务，因为是要主从服务，创建方式分为 Master,Slave
redis-master-service.yaml

kind: Service
apiVersion: v1
metadata:
  labels:
    app: redis-sentinel-master-ss
  name: redis-sentinel-master-ss
  namespace: database
spec:
  clusterIP: None
  ports:
  - name: redis
    port: 6379
    targetPort: 6379
  selector:
    app: redis-sentinel-master-ss
​

redis-slave-service.yaml

kind: Service
apiVersion: v1
metadata:
  labels:
    app: redis-sentinel-slave-ss
  name: redis-sentinel-slave-ss
  namespace: database
spec:
  clusterIP: None
  ports:
  - name: redis
    port: 6379
    targetPort: 6379
  selector:
    app: redis-sentinel-slave-ss
​

创建执行

kubectl apply -f redis-master-service.yaml -f redis-slave-service.yaml
​

Redis RBAC

熟悉 Redis 的持久化方式的就清楚 rbac 了，不多说了，直接上配置
redis-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: redis-sentinel
  namespace: database
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: redis-sentinel
  namespace: database
rules:
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: redis-sentinel
  namespace: database
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: redis-sentinel
subjects:
- kind: ServiceAccount
  name: redis-sentinel
  namespace: database
​

执行

kubectl apply -f redis-rbac.yaml
​

Redis StatefulSet

关键主从主体了。
redis-sfs.yaml

kind: StatefulSet
apiVersion: apps/v1
metadata:
  labels:
    app: redis-sentinel-master-ss
  name: redis-sentinel-master-ss
  namespace: database
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-sentinel-master-ss
  serviceName: redis-sentinel-master-ss
  template:
    metadata:
      labels:
        app: redis-sentinel-master-ss
    spec:
      containers:
      - args:
        - -c
        - cp /mnt/redis-master.conf /data/ ; redis-server /data/redis-master.conf
        command:
        - sh
        image: redis
        imagePullPolicy: IfNotPresent
        name: redis-master
        ports:
        - containerPort: 6379
          name: masterport
          protocol: TCP
        volumeMounts:
        - mountPath: /mnt/
          name: config-volume
          readOnly: false
        - mountPath: /data/
          name: redis-sentinel-master-storage
          readOnly: false
      serviceAccountName: redis-sentinel
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          items:
          - key: redis-master.conf
            path: redis-master.conf
          name: redis-sentinel-config 
        name: config-volume
  volumeClaimTemplates:
  - metadata:
      name: redis-sentinel-master-storage
    spec:
      accessModes:
      - ReadWriteMany
      storageClassName: "redis-sentinel-storage-class"
      resources:
        requests:
          storage: 4Gi
​

redis-slave-sfs.yaml

kind: StatefulSet
apiVersion: apps/v1
metadata:
  labels:
    app: redis-sentinel-slave-ss
  name: redis-sentinel-slave-ss
  namespace: database
spec:
  replicas: 2
  selector:
    matchLabels:
      app: redis-sentinel-slave-ss
  serviceName: redis-sentinel-slave-ss
  template:
    metadata:
      labels:
        app: redis-sentinel-slave-ss
    spec:
      containers:
      - args:
        - -c
        - cp /mnt/redis-slave.conf /data/ ; redis-server /data/redis-slave.conf
        command:
        - sh
        image: redis
        imagePullPolicy: IfNotPresent
        name: redis-slave
        ports:
        - containerPort: 6379
          name: slaveport
          protocol: TCP
        volumeMounts:
        - mountPath: /mnt/
          name: config-volume
          readOnly: false
        - mountPath: /data/
          name: redis-sentinel-slave-storage
          readOnly: false
      serviceAccountName: redis-sentinel
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          items:
          - key: redis-slave.conf
            path: redis-slave.conf
          name: redis-sentinel-config 
        name: config-volume
  volumeClaimTemplates:
  - metadata:
      name: redis-sentinel-slave-storage
    spec:
      accessModes:
      - ReadWriteMany
      storageClassName: "redis-sentinel-storage-class"
      resources:
        requests:
          storage: 4Gi

​

执行

kubectl apply -f redis-master-sfs.yaml -f redis-slave-sfs.yaml
​

验证

验证一下主从是否成功

kubectl exec -ti redis-sentinel-slave-ss-1 -n database -- redis-cli -h redis-sentinel-master-ss-0.redis-sentinel-master-ss.database.svc.cluster.local info replication
​

验证信息如下

# Replication
role:master
connected_slaves:2
slave0:ip=179.20.2.29,port=6379,state=online,offset=179994198,lag=1
slave1:ip=179.20.1.16,port=6379,state=online,offset=179994501,lag=0
master_replid:0485ebc3da868283b253f584a02775cca2e140a6
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:179994501
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:178945926
repl_backlog_histlen:1048576

​

查看 nfs 是否有信息文件，完成创建