Streamlining Configuration Updates for Canary Deployments

We currently have one fairly large issue with our canary deployment server. Whenever we want to modify its configuration, we need to restart it, so there will always be some downtime when we adjust our load balancing style, balance ratio, etc. As we're building this to allow for the seamless deploy of new versions, that is a pretty big issue - but one that we can definitely fix!

To do that, we're going to allow our server to read and write configuration from another server, and to update itself on the fly. Normally one might use something like etcd or consul to store our configuration. But - good news - we've already written a key-value server! So we will use that.

But first, let's talk tradeoffs - our system is about to get a lot more complex. Not only do we need to have an entire other system running, but we need to determine how we set that system up, and how it is able to adjust our canary deployment server.

This is, however, necessary complexity. As it stands, our canary deployment server can't really do what it's intended to do. So let's get to it!

Configuration change method#

We can start by deciding how we want to have our configuration updated. We need a server that will store our configuration and allow it to be updated, with an update rate to allow us to get updates regularly. Since our total amount of configuration is relatively small (we only care about the load balancing method, ratio, sticky sessions, config server and rate) we can simply make that into a serializable struct, and allow it to be read and written on our configuration server.

JSON is a nice simple encoding with great standard library support in the encoding/json package, and we can attach it directly to the struct used for configuration. To support that, we need to do a quick refactor. In the load_balancer file, we can pull our load balancer config into its own struct, give it the proper tags and attach it to the load balancer, adding our config server and update rate at the same time.

// LoadBalancerConfig holds our load balancer's configuration.

Quick note, because we have pulled a number of fields that used to be in LoadBalancer into LoadBalancerConfig, our (lb *LoadBalancer) NextServer and (lb *LoadBalancer) ProxyRequest functions need to be updated to call into the new Config struct. Specifically in NextServer, lb.style will become lb.Config.Style and lb.ratio will become lb.Config.Ratio. For ProxyRequest, lb.stickySessions will become lb.Config.StickySessions.

Now that we have the configuration, we need to write the update methods.

Updating configuration#

To finalize our configuration setup, we need four main pieces:

• writing configuration

• reading configuration

• updating configuration while running

• setting our update rate to regularly do the above

Writing configuration

We could technically skip this, but then setting up our config server might fall to some strange bash script. We already know what configuration we care about, so we can easily use our server to set up its own config. Given a fully setup LoadBalancerConfig, we can marshal it and send a Post request to our key-value server, doing some simple error checking along the way.