High Availability Code Blind

High Availability for Code Blind Controller

The agones-controller responsibility is split up into agones-controller, which enacts the Code Blind control loop, and agones-extensions, which acts as a service endpoint for webhooks and the allocation extension API. Splitting these responsibilities allows the agones-extensions pod to be horizontally scaled, making the Code Blind control plane highly available and more resiliant to disruption.

Multiple agones-controller pods enabled, with a primary controller selected via leader election. Having multiple agones-controller minimizes downtime of the service from pod disruptions such as deployment updates, autoscaler evictions, and crashes.

Extension Pod Configrations

The agones-extensions binary has a similar helm configuration to agones-controller, see here. If you previously overrode agones.controller.* settings, you may need to override the same agones.extensions.* setting.

To change controller.numWorkers to 200 from 100 values and through the use of helm --set, add the follow to the helm command:

 ...
 --set agones.controller.numWorkers=200
 ...

An important configuration to note is the PodDisruptionBudget fields, agones.extensions.pdb.minAvailable and agones.extensions.pdb.maxUnavailable. Currently, the agones.extensions.pdb.minAvailable field is set to 1.

Deployment Considerations

Leader election will automatically be enabled and agones.controller.replicas is > 1. agones.controller.replicas defaults to 2.

The default configuration now deploys 2 agones-controller pods and 2 agones-extensions pods, replacing the previous single agones-controller pod setup. For example:

NAME                                 READY   STATUS    RESTARTS   AGE
agones-allocator-78c6b8c79-h9nqc     1/1     Running   0          23h
agones-allocator-78c6b8c79-l2bzp     1/1     Running   0          23h
agones-allocator-78c6b8c79-rw75j     1/1     Running   0          23h
agones-controller-fbf944f4-vs9xx     1/1     Running   0          23h
agones-controller-fbf944f4-sjk3t     1/1     Running   0          23h
agones-extensions-5648fc7dcf-hm6lk   1/1     Running   0          23h
agones-extensions-5648fc7dcf-qbc6h   1/1     Running   0          23h
agones-ping-5b9647874-2rrl6          1/1     Running   0          27h
agones-ping-5b9647874-rksgg          1/1     Running   0          27h

The number of replicas for agones-extensions can be set using helm variable agones.extensions.replicas, but the default is 2.

We expect the aggregate memory consumption of the pods will be slightly higher than the previous singleton pod, but as the responsibilities are now split across the pods, the aggregate CPU consumption should also be similar.

Feature Design

Please see HA Code Blind.