Kubernetes Ingress實戰

本節內容：

  • 服務發現與負載均衡
  • Ingress實戰

一、服務發現與負載均衡

在前面的安裝部署kubernetes集羣中已經簡單用示例來演示了Pod和Service，Kubernetes通過Service資源在Kubernetes集羣內針對容器實現了服務發現和負載均衡。而Service就是kubernetes服務發現與負載均衡中的一種。

目前，kubernetes中的負載均衡大致可以分為以下幾種機制，每種機制都有其特定的應用場景：

  • Service：直接用Service提供cluster內部的負載均衡，並藉助cloud provider提供的LB提供外部訪問
  • Ingress Controller：還是用Service提供cluster內部的負載均衡，但是通過自定義LB提供外部訪問
  • Service Load Balancer：把load balancer直接跑在容器中，實現Bare Metal的Service Load Balancer
  • Custom Load Balancer：自定義負載均衡，並替代kube-proxy，一般在物理部署Kubernetes時使用，方便接入公司已有的外部服務

1. Service

Service是對一組提供相同功能的Pods的抽象，併為它們提供一個統一的入口。藉助Service，應用可以方便的實現服務發現與負載均衡，並實現應用的零宕機升級。Service通過標籤來選取服務後端，一般配合Replication Controller或者Deployment來保證後端容器的正常運行。這些匹配標籤的Pod IP和端口列表組成endpoints,由kube-proxy負責將服務IP負載均衡到這些endpoints上。

Service有四種類型:

  1. ClusterIP:默認類型,自動分配一個僅cluster內部可以訪問的虛擬IP
  2. NodePort:在ClusterIP基礎上為Service在每台機器上綁定一個端口,這樣就可以 通過 <NodeIP>:NodePort 來訪問該服務
  3. LoadBalancer:在NodePort的基礎上,藉助cloud provider創建一個外部的負載均 衡器,並將請求轉發到 <NodeIP>:NodePort
  4. ExternalName:將服務通過DNS CNAME記錄方式轉發到指定的域名(通 過 spec.externlName 設定)。需要kube-dns版本在1.7以上。

另外，也可以將已有的服務以Service的形式加入到Kubernetes集羣中來，只需要在創建 Service的時候不指定Label selector，而是在Service創建好後手動為其添加endpoint。

Service雖然解決了服務發現和負載均衡的問題，但它在使用上還是有一些限制，比如

  • 只支持4層負載均衡，沒有7層功能
  • 對外訪問的時候，NodePort類型需要在外部搭建額外的負載均衡，而LoadBalancer要求kubernetes必須跑在支持的cloud provider上面。

2. Ingress和Ingress Controller簡介

（1）Ingress

Ingress就是為了解決這些限制而引入的新資源，主要用來將服務暴露到cluster外面，並且可以自定義服務的訪問策略。比如想要通過負載均衡器實現不同子域名到不同服務的訪問：

foo.bar.com --|                 |-> foo.bar.com s1:80
              | 178.91.123.132  |
bar.foo.com --|                 |-> bar.foo.com s2:80

可以這樣來定義Ingress：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - backend:
          serviceName: s1
          servicePort: 80
  - host: bar.foo.com
    http:
      paths:
      - backend:
          serviceName: s2
          servicePort: 80

【注意】：Ingress本身並不會自動創建負載均衡器，cluster中需要運行一個ingress controller來根據Ingress的定義來管理負載均衡器。

目前社區提供了nginx和gce的參考實現

簡單的説，ingress就是從kubernetes集羣外訪問集羣的入口，將用户的URL請求轉發到不同的service上。Ingress相當於nginx、apache等負載均衡方向代理服務器，其中還包括規則定義，即URL的路由信息，路由信息得的刷新由Ingress controller來提供。

（2）Ingress Controller

Ingress Controller 實質上可以理解為是個監視器，Ingress Controller 通過不斷地跟 kubernetes API 打交道，實時的感知後端 service、pod 等變化，比如添加和減少 pod，service 增加與減少等;當得到這些變化信息後，Ingress Controller 再結合下文的 Ingress 生成配置，然後更新反向代理負載均衡器，並刷新其配置，達到服務發現的作用。

二、Ingress實戰

在使用Ingress resource之前，有必要先了解下面幾件事情。Ingress是beta版本的resource，在kubernetes1.1之前還沒有。你需要一個Ingress Controller來實現Ingress，單純的創建一個Ingress沒有任何意義。

目前社區提供了nginx和gce的參考實現。當然還有其他實現，開源的 NGINX 和 NGINX Plus 開發了相應的 Ingress controller。

  • GCE/GKE會在master節點上部署一個ingress controller。你可以在一個pod中部署任意個自定義的ingress controller。你必須正確地annotate每個ingress，比如 運行多個ingress controller 和 關閉glbc
  • 在非GCE/GKE的環境中，你需要在pod中部署一個controller。

（1）使用 NGINX 和 NGINX Plus 的 Ingress Controller 進行 Kubernetes 的負載均衡

https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/complete-example

前置需求

Make sure you have access to the Ingress controller image:

  • For NGINX Ingress controller, use the image nginxdemos/nginx-ingress from DockerHub .
  • For NGINX Plus Ingress controller, build your own image and push it to your private Docker registry by following the instructions from here .

The installation manifests are located in theinstallfolder. In the steps below we assume that you will be running the commands from that folder.

1. Create a Namespace, a SA and the Default Secret.

  1. Create a namespace and a service account for the Ingress controller:

    kubectl apply -f common/ns-and-sa.yaml

  2. Create a secret with a TLS certificate and a key for the default server in NGINX:

    $ kubectl apply -f common/default-server-secret.yaml

    Note: The default server returns the Not Found page with the 404 status code for all requests for domains for which there are no Ingress rules defined. For testing purposes we include a self-signed certificate and key that we generated. However, we recommend that you use your own certificate and key.

  3. Optional. Create a config map for customizing NGINX configuration (read more about customizationhere):

    $ kubectl apply -f common/nginx-config.yaml

2. Configure RBAC

If RBAC is enabled in your cluster, create a cluster role and bind it to the service account, created in Step 1:

$ kubectl apply -f rbac/rbac.yaml

Note: To perform this step you must be a cluster admin.

3. Deploy the Ingress Controller

We include two options for deploying the Ingress controller:

  • Deployment . Use a Deployment if you plan to dynamically change the number of Ingress controller replicas.
  • DaemonSet . Use a DaemonSet for deploying the Ingress controller on every node or a subset of nodes.
3.1 Create a Deployment

For NGINX, run:

$ kubectl apply -f deployment/nginx-ingress.yaml

For NGINX Plus, run:

$ kubectl apply -f deployment/nginx-plus-ingress.yaml

Note: Update thenginx-plus-ingress.yamlwith the container image that you have built.

Kubernetes will create one Ingress controller pod.

3.2 Create a DaemonSet

For NGINX, run:

$ kubectl apply -f daemon-set/nginx-ingress.yaml

For NGINX Plus, run:

$ kubectl apply -f daemon-set/nginx-plus-ingress.yaml

Note: Update thenginx-plus-ingress.yamlwith the container image that you have built.

Kubernetes will create an Ingress controller pod on every node of the cluster. Readthis docto learn how to run the Ingress controller on a subset of nodes, instead of every node of the cluster.

3.3 Check that the Ingress Controller is Running

Run the following command to make sure that the Ingress controller pods are running:

$ kubectl get pods --namespace=nginx-ingress

4. Get Access to the Ingress Controller

If you created a daemonset, ports 80 and 443 of the Ingress controller container are mapped to the same ports of the node where the container is running. To access the Ingress controller, use those ports and an IP address of any node of the cluster where the Ingress controller is running.

If you created a deployment, below are two options for accessing the Ingress controller pods.

4.1 Service with the Type NodePort

Create a service with the typeNodePort:

$ kubectl create -f service/nodeport.yaml

Kubernetes will allocate two ports on every node of the cluster. To access the Ingress controller, use an IP address of any node of the cluster along with two allocated ports. Read more about the type NodePorthere.

4.2 Service with the Type LoadBalancer

Create a service with the typeLoadBalancer. Kubernetes will allocate and configure a cloud load balancer for load balancing the Ingress controller pods.

Create a service using a manifest for your cloud provider:

  • For GCP or Azure, run:
    $ kubectl apply -f service/loadbalancer.yaml

  • For AWS, run:

    $ kubectl apply -f service/loadbalancer-aws-elb.yaml

    Kubernetes will allocate a Classic Load Balancer (ELB) in TCP mode with the PROXY protocol enabled to pass the client's information (the IP address and the port). NGINX must be configured to use the PROXY protocol:

    • Add the following keys to the config map file nginx-config.yaml from the Step 1 :
      proxy-protocol: "True"
real-ip-header: "proxy_protocol"
set-real-ip-from: "0.0.0.0/0"
    • Update the config map:
      kubectl apply -f common/nginx-config.yaml

    Note
    : For AWS, additional options regarding an allocated load balancer are available, such as the type of a load balancer and SSL termination. Read
    this doc
    to learn more.

Use the public IP of the load balancer to access the Ingress controller. To get the public IP:

  • For GCP or Azure, run:
    $ kubectl get svc nginx-ingress --namespace=nginx-ingress

  • In case of AWS ELB, the public IP is not reported by kubectl, as the IP addresses of the ELB are not static and you should not rely on them, but rely on the ELB DNS name instead. However, you can use them for testing purposes. To get the DNS name of the ELB, run:

    $ kubectl describe svc nginx-ingress --namespace=nginx-ingress

    You can resolve the DNS name into an IP address using
    nslookup
    :

    $ nslookup <dns-name>

Read more about the type LoadBalancerhere.

（2）配置需要測試的service

部署測試服務cafe service

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: coffee
spec:
  replicas: 2
  selector:
    matchLabels:
      app: coffee
  template:
    metadata:
      labels:
        app: coffee
    spec:
      containers:
      - name: coffee
        image: nginxdemos/hello:plain-text
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: coffee-svc
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: coffee
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: tea
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tea 
  template:
    metadata:
      labels:
        app: tea 
    spec:
      containers:
      - name: tea 
        image: nginxdemos/hello:plain-text
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: tea-svc
  labels:
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: tea

（3）創建Ingress

假設這兩個服務要暴露到集羣外部。要創建一個ingress：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
spec:
  tls:
  - hosts:
    - cafe.example.com
    secretName: cafe-secret
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80

創建ingress：

[root@node1 nginx_ingress]# kubectl get ing
NAME              HOSTS                                  ADDRESS   PORTS     AGE
test              n17.my.com,n18.my.com                            80        52s

（3）測試Application

  1. To access the application, curl the coffee and the tea services. We'll usecurl's --insecure option to turn off certificate verification of our self-signed certificate and the --resolve option to set the Host header of a request withcafe.example.com

    To get coffee:再refresh一次會發現load balance到另一個container

    $ curl --resolve cafe.example.com:443:$IC_IP https://cafe.example.com/coffee --insecure
Server address: 10.12.0.18:80
Server name: coffee-7586895968-r26zn
$ curl --resolve cafe.example.com:443:$IC_IP https://cafe.example.com/coffee --insecure
Server address: 10.12.1.18:80
Server name: coffee-7586895968-r26zn

...

  2. If your rather prefer tea:

    $ curl --resolve cafe.example.com:443:$IC_IP https://cafe.example.com/tea --insecure
Server address: 10.12.0.19:80
Server name: tea-7cd44fcb4d-xfw2x
...

    Note: If you're using a NodePort service to expose the Ingress controller, replace port 443 in the commands above with the node port that corresponds to port 443.

results matching ""

    No results matching ""