Support Agones on GKE Autopilot

[zmerlynn]

Is your feature request related to a problem? Please describe.

Agones has a number of friction points on GKE Autopilot:

• Arbitrary host port selection is not allowed (denied by autogke-no-host-port)
• Use of the cluster-autoscaler.kubernetes.io/safe-to-evict=false to minimize disruption (denied by autogke-node-affinity-selector-limitation):
  • On game server pods, Agones defaults to cluster-autoscaler.kubernetes.io/safe-to-evict=false annotation - this can be mitigated with Allow cluster autoscaler to scale down game server pods #2747
  • The Agones controller also defaults to `cluster-autoscaler.kubernetes.io/safe-to-evict=false', presumably because it is not HA and would like to minimize disruption, e.g. Split agones controller into leader elected pods #1296

Note that Autopilot issues were previously brought up in #2249 - the state of the world has changed considerably since then (mutating webhooks are now allowed), but there are still friction points.

Describe the solution you'd like

I propose the following:

• We've recently added a feature to allow for Autopilot host port assignment using a special annotation. In order to allow this to be used with Agones, I propose:
  • A small cloud-provider library to allow for per-provider abstractions
  • In said cloud provider abstraction, we add a game server pod interceptor that can add the appropriate annotation
  • We also add a validator that the portPolicy is Dynamic (we may be able to support Passthrough later, but let's start here)
  • (This section is complete with Sync Pod host ports back to GameServer in GCP #2782 and Split port allocators, implement Autopilot port allocation/policies #2789)
• The controllers/webhooks/allocator can be made HA, at which point we won't need to annotate safe-to-evict=false (Design: Make it possible to evict agones-controller by separating into controller/extensions #2797).
• So much testing.
• Add docs!

[roberthbailey]

One question on

On game server pods, Agones defaults to cluster-autoscaler.kubernetes.io/safe-to-evict=false annotation - this can be mitigated with #2747

The current mitigation will require that any fleet spec being moved from non-autopilot to autopilot will need to be updated to add extra metadata. It would be nice in the longer term if we could better support upgrades (across all platforms) and switch the default behavior to allow a safe eviction but give folks the chance to prevent it if that doesn't work for their game server.

[zmerlynn]

One question on

On game server pods, Agones defaults to cluster-autoscaler.kubernetes.io/safe-to-evict=false annotation - this can be mitigated with #2747

The current mitigation will require that any fleet spec being moved from non-autopilot to autopilot will need to be updated to add extra metadata. It would be nice in the longer term if we could better support upgrades (across all platforms) and switch the default behavior to allow a safe eviction but give folks the chance to prevent it if that doesn't work for their game server.

I debated this, and it again comes to explicit intent vs transparently working: One possible option is that we just set this annotation to be true on Autopilot, but that doesn't seem necessarily correct either.