Running App On k8s By Helm

把app部署到k8s运行,几个关键点:

把app部署到k8s时,考虑最小化改动程序代码,这里把原来app运行时用到的config.yaml的内容设置到一个环境变量如这里的talkingbot_yaml中, app启动代码优先读这个环境变量,如果不为空则取改变量值作为替换为原来config.yaml的内容,如果为空则说明按照原有的文件路径的方式取config.yaml

部署app的演进历程 #

running one app:raw #

running one app:docker #

running one app:k8s #

running one app:k8s-helm #

what is helm?

show me the code 实操: 以部署stateless app的部署为例子 #

Dokcerfile #

FROM alpine

RUN apk add --no-cache ca-certificates su-exec

VOLUME ["/data/talkingbot"]
COPY ./TalkingBot /bin/TalkingBot
COPY ./deployment/dumb-init /bin/dumb-init
RUN chmod +x /bin/dumb-init
EXPOSE  8086
ENTRYPOINT ["/bin/dumb-init", "/bin/TalkingBot","-c","/data/talkingbot/config.yaml"]

假设开始的app为TalkingBot,http监听端口为8086,这里使用dumb-init来防止一些daemon process issue

生成镜像

docker build -t talkingbot:1.0 .
#docker run --rm --name TalkingBot -p 8086:8086 -v /data/talkingbot:/data/talkingbot talkingbot:1.0

k8s Deployment all in one yaml file #

部署app用的k8s配置k8s-all-in-one.yaml, 文件里面包含了k8s运行app所用到的
deployment,service,configmap,pc,pvc

apiVersion: apps/v1
kind: Deployment
metadata:
  name: talkingbot
  namespace: kube-public
spec:
  replicas: 2
  selector:
    matchLabels:
      app: talkingbot
  template:
    metadata:
      labels:
        app: talkingbot
    spec:
      containers:
      - name: talkingbot
        image: 'talkingbot:1.0'
        env:
        - name: talkingbot_yaml
          valueFrom:
            configMapKeyRef:
              name: talkingbot-config
              key: configmapkey-talkingbot.yaml
        ports:
        - containerPort: 8086
        volumeMounts:
        - name: talkingbot-volume
          mountPath: /data/talkingbot
      volumes:
      - name: talkingbot-volume
        emptyDir: {}
        # hostPath:
          # directory location on physical host
          # path: /data/go/src/talkingbot/config
---
apiVersion: v1
kind: Service
metadata:
  name: talkingbot-svc
  namespace: kube-public
  labels:
    app: talkingbot-svc
spec:
  type: NodePort
  ports:
  - port: 8086 #port of service running inner pod, must exist
    name: talkingbot-port
    nodePort: 30081 # request by external load balance, must exist when `type: NodePort`
    # targetPort: 8170 #port of service running inner pod?, can be ignored ; By default the targetPort will be set to the same value as the port
  # externalIPs:
  # - 192.168.1.125
  selector:
    app: talkingbot

---
apiVersion: v1
data:
  configmapkey-talkingbot.yaml: |
    appName: 'TalkingBot' #应用名称
    serverPort: 8086    #端口
    devMode: True   #开发调试模式开关
    flowDir: '/data/talkingbot' #已载入流程存放路径,if empty ,it will use ./
    audioDir: "/data/talkingbot/audioDir/"
    httpTimeoutSecond: 1
    delayResponceBeginSecond: 1 #开始延迟回复,秒
    httpRateLimitMax: 100 #请求model时最大迸发数,防止model负载过高503,默认100/s
    redisDeployKind: '1' #1:单例(默认), 2:哨兵主从, 3:官方redis-cluster
    redisSentinelMasterName: 'master' #哨兵模式下 master key name, 默认master
    redisMain:  'my-redis-master.default.svc.cluster.local:6379'
    redisPassword: 'LEt3bEzn4g'
kind: ConfigMap
metadata:
  name: talkingbot-config
  namespace: kube-public

操作记录

jialin in DESKTOP-F6Q50AM in TalkingBot on  k8s [$?] via 🐹 v1.18
❯ kubectl apply -f  k8s-all-in-one.yaml
deployment.apps/talkingbot created
service/talkingbot-svc created
configmap/talkingbot-config created

jialin in DESKTOP-F6Q50AM in TalkingBot on  k8s [$?] via 🐹 v1.18
❯ kubectl get pod -n kube-public
NAME                          READY   STATUS              RESTARTS     AGE
talkingbot-57b9948868-5tzxr   0/1     ContainerCreating   0            2s
talkingbot-57b9948868-lrw77   0/1     ContainerCreating   0            2s

jialin in DESKTOP-F6Q50AM in TalkingBot on  k8s [$?] via 🐹 v1.18
❯ kubectl get pod -n kube-public
NAME                          READY   STATUS    RESTARTS     AGE
talkingbot-57b9948868-5tzxr   1/1     Running   0            12s
talkingbot-57b9948868-lrw77   1/1     Running   0            12s

k8s Deployment by helm #

helm 把上面的 # 做成了 模板配置化的形式,把很可能需要变更的配置项放到了values.yaml中,部署时在values.yaml 进行配置即可
helm的结构如下

#tree helm-talkingbot

helm-talkingbot
├── charts
├── Chart.yaml
├── templates
│   ├── configmap.yaml
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── NOTES.txt
│   ├── pvc.yaml
│   ├── serviceaccount.yaml
│   ├── service.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml

文件太多,打包放这里k8s-helm-talkingbot.tar.gz, helm部署配置在目录helm-talkingbot

操作记录

helm install my-talkingbot ./helm-talkingbot --set volumePermissions.enabled=true &&kubectl apply -f mytalkingbotpv.yaml

NAME: my-talkingbot
LAST DEPLOYED: Sat May 14 10:17:54 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
talkingbot; can be accessed via port 8086 on the following DNS name from within your cluster:

my-talkingbot-helm-talkingbot.default.svc.cluster.local
persistentvolume/mytalkingbotpv1 created

jialin in DESKTOP-F6Q50AM in TalkingBot/deployment on  k8s [$✘!?]
❯ kubectl get pod -n default
NAME                                             READY   STATUS              RESTARTS      AGE
dev212-dd6b559dc-qx7j6                           1/1     Running             2 (14d ago)   25d
my-minio-7785f7576f-fzgxr                        1/1     Running             0             8d
my-redis-master-0                                1/1     Running             1 (14d ago)   18d
my-redis-replicas-0                              1/1     Running             0             8d
my-redis-replicas-1                              1/1     Running             0             8d
my-redis-replicas-2                              1/1     Running             0             8d
my-talkingbot-helm-talkingbot-67d7c8fc95-8k8j6   0/1     ContainerCreating   0             11s

jialin in DESKTOP-F6Q50AM in TalkingBot/deployment on  k8s [$✘!?]
❯ kubectl get pod -n default
NAME                                             READY   STATUS    RESTARTS      AGE
dev212-dd6b559dc-qx7j6                           1/1     Running   2 (14d ago)   25d
my-minio-7785f7576f-fzgxr                        1/1     Running   0             8d
my-redis-master-0                                1/1     Running   1 (14d ago)   18d
my-redis-replicas-0                              1/1     Running   0             8d
my-redis-replicas-1                              1/1     Running   0             8d
my-redis-replicas-2                              1/1     Running   0             8d
my-talkingbot-helm-talkingbot-67d7c8fc95-8k8j6   1/1     Running   0             15s

k8s集群环境内可以通过这个my-talkingbot-helm-talkingbot.default.svc.cluster.local:8086地址访问talkingbot的服务

参考资料:
why helma and what is helm?
k8s serivce
k8s architecture
k8s load balancer
twelve-factor methodology
12 Factor App meets Kubernetes: Benefits for cloud-native apps

2022-05-14