Updating our server and collar without downtime

Two kinds of software in Fudini need updating without anyone noticing: the server behind the app, and the firmware on the collar around a dog's neck. In October we fixed both. A deploy used to take the API down for up to 30 seconds. Now it swaps the old server for the new one with no gap. The collar used to need a USB cable for every update. Now it updates over Bluetooth from the app. This post covers how each one works, and why the collar can't do what the server does yet.

The server: blue and green

Every push to our main branch builds the backend, copies it to our server in Ireland and restarts it. Until 23 October "restarts" meant stopping the old process and starting the new one. For 10 to 30 seconds, every request from the app and every page on fudini.co got an error.

The usual fix is called a blue/green deploy. The running version is "blue". You start the new version, "green", next to it. You check that green is healthy, send traffic to it and only then stop blue. If green never becomes healthy, you throw it away and blue keeps serving as if nothing happened.

We don't run a load balancer or a second machine, so we built it from what we already had: Docker Compose and Caddy, our reverse proxy.

• Database changes first. Migrations run before anything else, while the old server is still serving. If one fails, the deploy stops there and nothing changes. Every migration only adds tables or columns, so the old server keeps working against the new schema.
• Start green next to blue. We tell Compose to run two copies of the server instead of one. Both answer on the same internal name, and Caddy sends requests to whichever one is up.
• Wait for healthy. The new copy has two minutes to pass its health check. If it fails or crashes, we print its last 50 log lines, remove it and stop the deploy. Blue never notices.
• Drain blue. Once green is healthy, blue gets a stop signal and up to 15 seconds to finish the requests it is already handling, such as a photo upload or a collar sync, before it exits.
• Retry the gap. For a moment Docker's internal DNS can still point to the container that is shutting down. Caddy retries a refused connection every quarter of a second for up to 30 seconds, so the request lands on green instead of failing.

The whole deploy takes about three minutes from push to live, and the swap itself takes about 15 seconds.

The collar: updates over Bluetooth

Our prototype collar is a small Bluetooth board with a motion sensor and a battery, sealed in a clip-on case. Before firmware 0.9.2 the only way to update it was to open the case, plug in a USB cable and copy a file. That's fine on a desk and impossible once the collar is on a dog and the owner is a customer.

Now an update works like this:

• We publish a version. A new firmware build is uploaded from our admin panel with a version number and a short changelog. The file is stored on S3 and served through our API.
• The app notices. When the phone connects, it reads the collar's firmware version and asks the server whether a newer one exists. If it does, the collar screen offers the update and shows what changed.
• The collar gets ready. The app asks the collar to restart into its update mode. Before it does, the collar checks its battery and refuses if the charge is too low to finish safely, unless it is on a charger.
• The phone sends the image. The app streams the new firmware over Bluetooth, in small pieces, and the collar writes each one to flash. Then it checks the whole image and restarts into the new version.

On the bench an update takes just under three minutes on battery. The activity the collar recorded but hasn't synced yet sits in a separate flash chip and survives every update. The next sync picks it up as usual.

The bug that only showed up on phones

The first attempts from the app failed after a few seconds with "operation failed". Our bench script worked every time. The difference was pacing. The phone libraries send a burst of packets before waiting for the collar to confirm them, and their default burst was larger than the collar's update code can buffer while it writes to flash. We made the phone wait for a confirmation after every few packets, on both iOS and Android.

Why the collar isn't blue/green yet

This is the honest part. A blue/green update on a device means keeping two copies of the firmware: the running one and the new one. You write the new copy next to the old, check it and only then switch over. If anything goes wrong, the device still has the old version to boot.

Our prototype board doesn't work that way. Its update code erases the old firmware first and then writes the new one in its place. In the middle of an update there is no working firmware on the collar at all, only the small update program that came with the board.

So we tested the ways an update can go wrong:

• The phone walks out of range. On the bench we cut the connection at 30%. The collar stayed in update mode, and our bench tool reconnected, asked how much had already been written and continued from there. The update finished and checked out. The app doesn't resume a dropped update yet; that is next on our list.
• The owner gives up. An abandoned update leaves the collar waiting in update mode. It is not lost, and no recorded activity is gone: the update can be sent again and finished. But until it finishes, the collar records nothing.
• The battery is low. That's why the collar refuses to start an update on a low battery. Running out halfway is the case we can't recover from over the air.

That is good enough for our own prototype collars. It isn't good enough for collars we sell. A customer shouldn't have to know that an update must be finished.

What production collars will do

For the production collar we're moving to two firmware slots. The new version is written to the spare slot while the old one keeps running and recording. The collar switches only after the new image is complete and verified. If the new version fails to start properly, the collar goes back to the old one by itself. That is the device version of blue and green.

The same change adds signed firmware: the collar will only accept images signed with our key, so nobody else can put their own code on it. Together with the per-collar key we added in firmware 0.10.0, which means only the owner's phones can start an update, that closes the update path end to end.