GKE Subsetting recalculation only appears to occur when nodes are added, not removed

[glasser]

I use GCE ingresses via hosted GKE (not running my own ingress-gce).

We have a large-ish cluster with GKE subsetting and a lot of internal network pass-through load balancers.

We wanted to move all our pods to a new nodepool so we created the nodepool, drained the old nodes gradually until they didn't have any pods (other than daemonsets), and then scaled the original node pool down to 0 nodes.

This broke many of our load balancers because it removed all the GCE_VM_IP endpoints from their zonal NEGs.

After some experimentation, it appears that when you remove nodes from GKE, GCE_VM_IP endpoints can be removed from the zonal NEG groups associated with internal network pass-through load balancers (with externalTrafficPolicy: Cluster) and the controller won't actively add endpoints for newer nodes.

Adding just one more node to the cluster after this seems sufficient to trigger recalculation and get those NEGs back to 25. But if you don't do that, it seems like you can just lose endpoints from your NEGs and eventually break them!

[swetharepakula]

Thanks @glasser for reporting!

This should be fixed with #2622.

We are currently qualifying the new release and hope to have it fixed in GKE in the next couple weeks.

[swetharepakula]

We also have made release nodes with possible mitigations for the time being: https://cloud.google.com/kubernetes-engine/docs/release-notes#August_14_2024

The easiest way to avoid downtime is to switch to using xTP=Local

[glasser]

Thanks, this sounds exactly like our issue. Not quite up to date enough on reading our release notes I guess!

Switching over our production cluster to Local today seems like a bit less of a scary thing than just scaling down bit by bit and trigger some sort of scale up on another node pool after each time. (Though I don't know a better way to trigger scale-up than to schedule something that doesn't fit... I don't want to just scale the pool normally because there's currently a fair amount of imbalance across zones.)

[glasser]

(Ah OK, we can just scale up a random other node pool.)

This will be called out in the release notes when it is fixed?

[gauravkghildiyal]

Hi @glasser. Not sure if you already noticed, but we recently sent out the release updates for this fix: https://cloud.google.com/kubernetes-engine/docs/release-notes#September_10_2024

[glasser]

@gauravkghildiyal Thanks! And the release notes are clear about what we need to do (control plane upgrade). Trying it in our dev cluster now!