Image docker.m.daocloud.io/istio/pilot:1.17.1 does not exist but istiod 1.17.1 is the only supported chart version in dce-community repo

[DonaldKellett]

Image docker.m.daocloud.io/istio/pilot:1.17.1 does not exist but istiod 1.17.1 is the only supported chart version in dce-community repo (https://release.daocloud.io/chartrepo/community)

Chart details

• Repository: dce-community (https://release.daocloud.io/chartrepo/community)
• Chart name: istiod
• Chart version: 1.17.1

Steps to reproduce

Install DCE5 Community with dce5-installer, then:

helm repo add dce-community https://release.daocloud.io/chartrepo/community
helm repo update
helm -n istio-system upgrade \
    --reuse-values \
    istiod \
    dce-community/istiod \
    --version 1.17.1

Wait for the upgrade to fail, then run:

kubectl -n istio-system get po -l istio=pilot

Sample output:

NAME                      READY   STATUS             RESTARTS   AGE
istiod-84c44d956d-r94wr   1/1     Running            0          23h
istiod-b999b56b9-wtzwt    0/1     ImagePullBackOff   0          9m56s

Now check the events of the Pod with ImagePullBackOff error:

kubectl -n istio-system describe po istiod-b999b56b9-wtzwt | grep -A10 Events:

Output:

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  11m                    default-scheduler  Successfully assigned istio-system/istiod-b999b56b9-wtzwt to dcetest01
  Normal   Pulling    4m43s (x4 over 11m)    kubelet            Pulling image "docker.m.daocloud.io/istio/pilot:1.17.1"
  Warning  Failed     3m23s (x4 over 9m33s)  kubelet            Error: ErrImagePull
  Normal   BackOff    2m53s (x7 over 9m32s)  kubelet            Back-off pulling image "docker.m.daocloud.io/istio/pilot:1.17.1"
  Warning  Failed     2m53s (x7 over 9m32s)  kubelet            Error: ImagePullBackOff
  Warning  Failed     41s (x5 over 9m33s)    kubelet            Failed to pull image "docker.m.daocloud.io/istio/pilot:1.17.1": rpc error: code = Canceled desc = failed to pull and unpack image "docker.m.daocloud.io/istio/pilot:1.17.1": context canceled

Chart values

Not modified from initial DCE5 Community installation using dce5-installer:

global:
  hub: docker.m.daocloud.io/istio
  imagePullPolicy: IfNotPresent
  meshID: global-service
  multiCluster:
    clusterName: kpanda-global-cluster
meshConfig:
  defaultConfig:
    extraStatTags:
    - destination_mesh_id
    - source_mesh_id
    gatewayTopology:
      numTrustedProxies: 2
    proxyMetadata:
      ISTIO_META_DNS_AUTO_ALLOCATE: "true"
      ISTIO_META_DNS_CAPTURE: "true"
pilot:
  autoscaleMin: 1
  replicaCount: 1
  resources:
    requests:
      cpu: 256m
      memory: 256Mi
telemetry:
  v2:
    prometheus:
      configOverride:
        gateway:
          debug: "false"
          disable_host_header_fallback: true
          metrics:
          - dimensions:
              destination_cluster: node.metadata['CLUSTER_ID']
              destination_mesh_id: node.metadata['MESH_ID']
              source_cluster: downstream_peer.cluster_id
              source_mesh_id: downstream_peer.mesh_id
          stat_prefix: istio
        inboundSidecar:
          debug: "false"
          disable_host_header_fallback: true
          metrics:
          - dimensions:
              destination_cluster: node.metadata['CLUSTER_ID']
              destination_mesh_id: node.metadata['MESH_ID']
              source_cluster: downstream_peer.cluster_id
              source_mesh_id: downstream_peer.mesh_id
          stat_prefix: istio
        outboundSidecar:
          debug: "false"
          metrics:
          - dimensions:
              destination_cluster: node.metadata['CLUSTER_ID']
              destination_mesh_id: node.metadata['MESH_ID']
              source_cluster: downstream_peer.cluster_id
              source_mesh_id: downstream_peer.mesh_id
          stat_prefix: istio

Alternatives considered

Falling back to version 1.16.x of istiod does not work with DaoCloud Helm repository mirror since the only existing version is 1.17.1 at the time of writing.

Some output snippets taken from Flux v2 shown below:

NAME                                            REVISION        SUSPENDED       READY   MESSAGE

helmchart/istio-system-istio-base               1.17.1          False           True    pulled 'base' chart with version '1.17.1'
helmchart/istio-system-istio-ingressgateway     1.17.1          False           True    pulled 'gateway' chart with version '1.17.1'
helmchart/istio-system-istiod                   1.17.1          False           False   invalid chart reference: failed to get chart version for remote reference: no 'istiod' chart with version matching '>=1.16.1 <1.17.0' found

NAME                                            REVISION        SUSPENDED       READY   MESSAGE

helmchart/istio-system-istio-base               1.17.1          False           True    pulled 'base' chart with version '1.17.1'
helmchart/istio-system-istio-ingressgateway     1.17.1          False           True    pulled 'gateway' chart with version '1.17.1'
helmchart/istio-system-istiod                   1.17.1          False           False   invalid chart reference: failed to get chart version for remote reference: no 'istiod' chart with version matching '>=1.17.0 <1.17.1' found

[kebe7jun]

I checked and found that docker.m.daocloud.io/istio/pilot:1.17.1 exists and I see that your istiod instance has started successfully, maybe it's an environment issue?

[DonaldKellett]

The successful istiod instance is from the original dce5-installer installation which is at version 1.16.1 (but 1.16.1 cannot be found in dce-community repo).

[DonaldKellett]

Just did a docker pull docker.m.daocloud.io/istio/pilot:1.17.1 locally and the image does indeed exist, just stuck in "Pulling fs layer" indefinitely

Perhaps this issue only affects HK and overseas regions?