Kubernetes 1.9集群使用traefik发布服务

共计 6007 个字符，预计需要花费 16 分钟才能阅读完成。

在前文中介绍了在 kubernetes 1.5.2 集群环境中使用 traefik 进行服务发布。Traefik 采用 daemonset 方式部署，连接 api-server 走的是 http 协议，也未配置 rbac。本文将介绍在 k8s 1.9 版本中使用 deployment 方式部署 traefik 来进行服务发布。

在开始之前，需要先了解一下什么是 RBAC。RBAC（基于角色的访问控制）使用 rbac.authorization.k8s.io  API 组来实现权限控制，RBAC 允许管理员通过 Kubernetes API 动态的配置权限策略。在 1.6 版本中 RBAC 还处于 Beat 阶段，如果想要开启 RBAC 授权模式需要在 apiserver 组件中指定 –authorization-mode=RBAC 选项。

在 RBAC API 的四个重要概念：
Role：是一系列的权限的集合，例如一个角色可以包含读取 Pod 的权限和列出 Pod 的权限
ClusterRole：跟 Role 类似，但是可以在集群中到处使用（Role 是 namespace 一级的）
RoloBinding：把角色映射到用户，从而让这些用户继承角色在 namespace 中的权限。
ClusterRoleBinding：让用户继承 ClusterRole 在整个集群中的权限。

简单点说 RBAC 实现了在 k8s 集群中对 api-server 的鉴权，更多的 RBAC 知识点请查阅官方文档：https://kubernetes.io/docs/admin/authorization/rbac/

一、给集群的节点打上 label
因为选择 deployment 方式部署，所以要给集群的节点打上 label，后续选择 nodeSelector 指定 traefik=proxy，副本数和集群节点数一致的时候，所有的节点上都会运行一个 pod

# kubectl get nodes –show-labels
# kubectl label  node vm1 traefik=proxy
# kubectl label  node vm2 traefik=proxy
# kubectl get nodes –show-labels

二、准备 yaml 文件
1、rbac 文件

# cat traefik-rbac.yaml
—
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  – apiGroups:
      – “”
    resources:
      – services
      – endpoints
      – secrets
    verbs:
      – get
      – list
      – watch
  – apiGroups:
      – extensions
    resources:
      – ingresses
    verbs:
      – get
      – list
      – watch
—
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
– kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

在启用 rbac 的环境下，如果鉴权未配置清楚，则 traefik pod 会报错如下

E0226 00:15:27.729832 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: services is forbidden: User “system:serviceaccount:kube-system:default” cannot list services at the cluster scope
E0226 00:15:29.013298 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: endpoints is forbidden: User “system:serviceaccount:kube-system:default” cannot list endpoints at the cluster scope
E0226 00:15:29.213354 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Secret: secrets is forbidden: User “system:serviceaccount:kube-system:default” cannot list secrets at the cluster scope
E0226 00:15:29.698574 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User “system:serviceaccount:kube-system:default” cannot list ingresses.extensions at the cluster scope
E0226 00:15:30.411837 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: services is forbidden: User “system:serviceaccount:kube-system:default” cannot list services at the cluster scope
E0226 00:15:31.912887 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: endpoints is forbidden: User “system:serviceaccount:kube-system:default” cannot list endpoints at the cluster scope

2、traefik 的 deployment 文件

# cat traefik-deployment.yaml 
—
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
—
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 2
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      hostNetwork: true
      nodeSelector:
        traefik: proxy
      terminationGracePeriodSeconds: 60
      containers:
      – image: traefik
        name: traefik-ingress-lb
        ports:
        – name: web
          containerPort: 80
          hostPort: 80
        – name: admin
          containerPort: 8081
        args:
        – –web
        – –web.address=:8081
        – –kubernetes

3、traefik 的 service 文件

# cat traefik-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  – port: 80
targetPort: 8081

4、通过 yaml 文件创建 clusterrole、clusterrolebinding、deployment、serviceaccount、service

# ls
# kubectl create -f traefik-rbac.yaml
# kubectl create -f traefik-deployment.yaml
# kubectl create -f traefik-service.yaml

# kubectl get pod -n kube-system
# kubectl get svc -n kube-system
# kubectl get svc

可以看到集群中 default namespace 中存在一个 frontend 服务。kube-system namespace 中存在 nginx-test、traefik-web-ui、kubernetes-dashboard 三个服务。我们后续将创建 4 个 ingress

通过 web-ui 可以看到在两个节点上各运行了一个 pod

三、通过 yaml 文件创建 ingress

 # cat ui.yaml
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  – port: 80
    targetPort: 8081
—
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  – host: traefik-ui
    http:
      paths:
      – backend:
          serviceName: traefik-web-ui
          servicePort: 80# cat webui-ing.yaml                 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-ingress
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  – host: k8s.webui
    http:
      paths:
      – backend:
          serviceName: kubernetes-dashboard
          servicePort: 443# cat Redis-ing.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  – host: k8s.frontend
    http:
      paths:
      – backend:
          serviceName: frontend
          servicePort: 80# cat nginx-ing.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-nginx-ingress
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  – host: test.fjhb.cn
    http:
      paths:
      – backend:
          serviceName: nginx-test
          servicePort: 80# kubectl create -f ui.yaml
# kubectl create -f webui-ing.yaml
# kubectl create -f redis-ing.yaml
# kubectl get ingress
# kubectl get ingress -n kube-system

三、验证
1、通过访问 traefik service 对应的 nodeport 端口，4 个 ingress 配置都加载到了

2、修改测试机 hosts 文件，将 4 个域名的解析分配到两台节点上

3、浏览器访问测试

这里出现 500 错误的原因是，后端的 kubernetes-dashboard 配置的是 https 协议



可以在 health 页面看到 http 状态码的统计信息