Specifying multiple HTTP match uri in Istio Canary deployment via Flagger

[mfarrokhnia]

I am gonna use automatic Canary deployment so I tried to follow the process via Flagger.
Here was my VirtualService file for routing:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: {{ .Values.project }}
  namespace: {{ .Values.service.namespace }}
spec:
  hosts:
    - {{ .Values.subdomain }}
  gateways:
    - mygateway.istio-system.svc.cluster.local
  http:
    {{- range $key, $value := .Values.routing.http }}
    - name: {{ $key }}
{{ toYaml $value | indent 6 }}
    {{- end }}

Which the routing part looks like this:

http:
    r1:
      match:
        - uri:
            prefix: /myservice/monitor
      route:
        - destination:
            host: myservice
            port:
              number: 9090
    r2:
      match:
        - uri:
            prefix: /myservice
      route:
        - destination:
            host: myservice
            port:
              number: 8080
      corsPolicy:
        allowCredentials: false
        allowHeaders:
        - X-Tenant-Identifier
        - Content-Type
        - Authorization
        allowMethods:
        - GET
        - POST
        - PATCH
        allowOrigin:
        - "*"
        maxAge: 24h    `

However as I found the Flagger overwites the virtualservice, I have removed this file and modified the canary.yaml file based on my requirements but I get yaml error:

{{- if .Values.canary.enabled }}
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
  name: {{ .Values.project }}
  namespace: {{ .Values.service.namespace }}
  labels:
    app: {{ .Values.project }}
    chart: {{ template "myservice-chart.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name:  {{ .Values.project }}
  progressDeadlineSeconds: 60
  autoscalerRef:
    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    name:  {{ .Values.project }}    
  service:
    port: 8080
    portDiscovery: true
    {{- if .Values.canary.istioIngress.enabled }}
    gateways:
    -  {{ .Values.canary.istioIngress.gateway }}
    hosts:
    - {{ .Values.canary.istioIngress.host }}
    {{- end }}
    trafficPolicy:
      tls:
        # use ISTIO_MUTUAL when mTLS is enabled
        mode: DISABLE
    # HTTP match conditions (optional)
    match:
      - uri:
          prefix: /myservice
    # cross-origin resource sharing policy (optional)
      corsPolicy:
        allowOrigin:
          - "*"
        allowMethods:
          - GET
          - POST
          - PATCH
        allowCredentials: false
        allowHeaders:
          - X-Tenant-Identifier
          - Content-Type
          - Authorization
        maxAge: 24h
      - uri:
          prefix: /myservice/monitor
  canaryAnalysis:
    interval: {{ .Values.canary.analysis.interval }}
    threshold: {{ .Values.canary.analysis.threshold }}
    maxWeight: {{ .Values.canary.analysis.maxWeight }}
    stepWeight: {{ .Values.canary.analysis.stepWeight }}
    metrics:
    - name: request-success-rate
      threshold: {{ .Values.canary.thresholds.successRate }}
      interval: 1m
    - name: request-duration
      threshold: {{ .Values.canary.thresholds.latency }}
      interval: 1m
    webhooks:
      {{- if .Values.canary.loadtest.enabled }}
      - name: load-test-get
        url: {{ .Values.canary.loadtest.url }}
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 5 -c 2 http://myservice.default:8080"
      - name: load-test-post
        url: {{ .Values.canary.loadtest.url }}
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 5 -c 2 -m POST -d '{\"test\": true}' http://myservice.default:8080/echo"
      {{- end }}  
{{- end }}

Can anyone help with this issue?

[stefanprodan]

Flagger doesn't have support for multiple matches, the service.match only accepts URIs matches for all ports. Instead of using matches for Prometheus scraping why not add annotations on the deployment?

  prometheus.io/path: "/myservice/monitor"
  prometheus.io/port: "9090"
  prometheus.io/scrape: "true"

Here is an example: https://github.com/weaveworks/flagger/blob/master/kustomize/podinfo/deployment.yaml#L20-L22

[mfarrokhnia]

@stefanprodan Thanks for reply. Then how can I connect to prometheus? which url would it be?
what about this case:

match:
     - uri:
           prefix: /myservice
     - uri:
           prefix: /api/myservice
     rewrite:
       uri: /myservice    
     route:
       - destination:
           host: myservice
           port:
             number: 8080
corsPolicy:
     allowOrigin:
       - "*" 
     allowMethods:
       - POST
       - PATCH
     allowCredentials: false
     allowHeaders:
       - X-Tenant-Identifier
       - Content-Type
       - authorization
     maxAge: "24h"        

can I have two prefixes?

[stefanprodan]

Then how can I connect to prometheus?

Prometheus is pull based, it discoverers the pods based on annotations and scapes the app not the other way around. You don't want to load balance the Prometheus scraping since you'll get inconsistent metrics.

My advice is to add the Prometheus annotations inside the deployments and remove the matchs for port 9090.

By the way, your Canary yaml is invalid, there is no corsPolicy under match. Please see the docs how a Canary definition looks.

[mfarrokhnia]

Then how can I connect to prometheus?

Prometheus is pull based, it discoverers the pods based on annotations and scapes the app not the other way around. You don't want to load balance the Prometheus scraping since you'll get inconsistent metrics.

My advice is to add the Prometheus annotations inside the deployments and remove the matchs for port 9090.

By the way, your Canary yaml is invalid, there is no corsPolicy under match. Please see the docs how a Canary definition looks.

yeah it has extra space in corspolicy, I'll fix that. Is it possible to use several prefixes in Canary?

[stefanprodan]

Is it possible to use several prefixes in Canary?

Yes that should work

[mfarrokhnia]

Is it possible to use several prefixes in Canary?

Yes that should work

It seems it is not working with this way that I tried, it just deploys the last prefix:

match:
      - uri:
          prefix: /api/myservice
          prefix: /myservice

[stefanprodan]

That's not a valid Istio match. Use:

    match:
      - uri:
          prefix: /api/myservice
      - uri:
          prefix: /myservice

[mfarrokhnia]

That's not a valid Istio match. Use:

    match:
      - uri:
          prefix: /api/myservice
      - uri:
          prefix: /myservice

thanks that worked. But the service gives 503 error service unavailable with external DNS, that means the routing is not working fine but it's working fine by port-forwarding. You mentioned that Istio uses Port Discovery and it's not needed to specify ports for virtualservice that's made by Canary, how can I make sure it is doing the right routing?

[stefanprodan]

You can look at the ClusterIPs, virtual service and destination rules generated by Flagger to make sure it did the right thing. Are you getting 503 for at the gateway level?

[mfarrokhnia]

@stefanprodan Yes, it is the gateway issue as port-forwarding localy workes fine. I have the ports in service.yaml

spec:
  clusterIP: 10.0.140.144
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  - name: debug
    port: 8000
    protocol: TCP
    targetPort: 8000
  - name: monitoring
    port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app: myservice-primary

and virtualservice looks like below:

spec:
  gateways:
  - mygateway.istio-system.svc.cluster.local
  hosts:
  - api.mydomain.come
  - myservice
  http:
  - corsPolicy:
      allowHeaders:
      - X-Tenant-Identifier
      - Content-Type
      - Authorization
      allowMethods:
      - GET
      - POST
      - PATCH
      allowOrigin:
      - '*'
      maxAge: 24h
    match:
    - uri:
        prefix: /api/myservice
    - uri:
        prefix: /myservice
    route:
    - destination:
        host: myservice-primary
      weight: 100
    - destination:
        host: myservice-canary
      weight: 0

and destination rule for myservice-primary is like this:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  creationTimestamp: "2020-02-11T11:41:41Z"
  generation: 1
  name: myservice-primary
  namespace: default
  ownerReferences:
  - apiVersion: flagger.app/v1alpha3
    blockOwnerDeletion: true
    controller: true
    kind: Canary
    name: myservice
    uid: 69ca19aa-4cc3-11ea-979f-76b1ead32f2a
  resourceVersion: "1043445"
  selfLink: /apis/networking.istio.io/v1alpha3/namespaces/default/destinationrules/myservice-primary
  uid: 786a3162-4cc3-11ea-979f-76b1ead32f2a
spec:
  host: myservice-primary
  trafficPolicy:
    tls:
      mode: DISABLE

I can not see any port under virtualservice that might be because of it I guess but you said istio does port discovery itself. Do you have any idea what it is missing?

[stefanprodan]

The Istio port discovery works inside the mesh, you could test it by running a pod inside the mesh and curl to 8000, 9090. I think there is no need for these ports to be exposed, Prometheus connect directly to the pods based on the annotations. Set portDiscovery: false and your service should work with the public gateway.

[mfarrokhnia]

The Istio port discovery works inside the mesh, you could test it by running a pod inside the mesh and curl to 8000, 9090. I think there is no need for these ports to be exposed, Prometheus connect directly to the pods based on the annotations. Set portDiscovery: false and your service should work with the public gateway.

That worked by deactivating portDiscovery. But why was that a problem? Should I never activate the portDiscovery?
and one more thing do I need to use annotation for debug port 8000 as well?

[stefanprodan]

That worked by deactivating portDiscovery. But why was that a problem? Should I never activate the portDiscovery?

If your app is exposed only inside the mesh then you can use portDiscovery. The Istio gateway can't map port 90/443 to more then one port but the internal one called mesh can.

Do I need to use annotation for debug port 8000 as well?

No, the annotations are only for Prometheus. You should be able to connect to 8000 with kubectl port-forward deploy/app 8000:8000.

[mfarrokhnia]

That worked by deactivating portDiscovery. But why was that a problem? Should I never activate the portDiscovery?

If your app is exposed only inside the mesh then you can use portDiscovery. The Istio gateway can't map port 90/443 to more then one port but the internal one called mesh can.

Do I need to use annotation for debug port 8000 as well?

No, the annotations are only for Prometheus. You should be able to connect to 8000 with kubectl port-forward deploy/app 8000:8000.

Thanks a lot for your help :)