Rolling out a release

Every week there is a new CoreCP version: 0.51.0, 0.52.0, and so on. It is prepared and tested for you; you decide whether it goes to your servers, and you roll it out. This page describes your part of that week, from reading the release dossier to the last wave of servers.

What happens before your turn is not yours to do, but it helps to know it:

• Monday the candidate is made: a test version of everything finished that week, for example 0.51.0~rc.1. It is installed on the test setup the way a real server installs it — from packages — and the full check round runs on it.
• Tuesday, after a day on the test setup, the release itself is made (0.51.0) and a release dossier is waiting for you.
• Wednesday 09:00 you roll out.

On Friday, Saturday and Sunday there is no ordinary rollout. When Monday's check round runs late, the rollout moves to Thursday — never to Friday.

Reading the release dossier (Tuesday)

The dossier is one page per version, in docs/release-decisions/ of the source code, with the date and the version in its name. You are sent its path. Read it in this order; it takes five to ten minutes:

1. Status at the top: when it says "waiting for the owner's decision", the release is ready for your decision.
2. Gate evidence: which candidate was tested, how many checks passed, and the fingerprint of the log. This is the same evidence your panel checks when you roll out.
3. Go/no-go record: nine pieces of evidence, each met, not met or not measured, and a recommendation. A recommendation is not a decision.
4. Rollback: how to go back to the previous version, should you need to.
5. Open points: things somebody still has to finish.

You write your decision on the Decision line of the dossier: GO, GO WITH CONDITIONS, NO-GO or DEFER. Nobody else fills in that line.

Rolling out on the panel (Wednesday)

A rollout always goes in the same order: your package source and the panel itself first, then the servers in waves.

1. Put the release on your own package source

1. Go to Releases → Production source (in Dutch: Releases → Productiebron).
2. Check in the box that the Working key is valid. If it says the key has expired or was revoked, rolling out is not possible; follow the key management runbook.
3. Enter the version under Release set, for example 0.51.0, and a short sentence under Reason, for example "week release 0.51".
4. Click Roll out (in Dutch: Uitrollen). The panel asks for your second factor.
5. The log box shows three steps: fetching, verifying and publishing. While verifying, the panel checks that the set is signed by the build server, that every package is right, and that the gate evidence is the evidence the dossier named. If anything is off, the set is refused with the reason, and the previous version stays on your source.

After done your servers can fetch the new version. Nothing has been installed yet.

2. Update the panel itself

The panel must always be at least as new as the servers it manages. It updates itself on the panel server (see From a terminal below). Under Releases → Rollout plan you then see the new version under The panel, with the outcome of the update.

3. The servers, in waves

1. Stay on Releases → Rollout plan. The plan is already there: the panel first, then wave 1 with the test server, then wave 2, then wave 3.
2. Click Start rollout from plan. The panel refuses while it is not ahead itself — finish step 2 first.
3. Follow the rollout on the releases board. The first wave is the canary: only when it is healthy does the next wave start.
4. A server that was already unhealthy before the rollout is skipped, with the reason. Fix that and start a recovery rollout afterwards.

When the last wave is updated, you are done. Inform your customers through your own company's status page and channels; CoreCP itself only sends notifications in the panel and by mail.

Back to the previous version

If something goes wrong, the dossier's Rollback section says exactly what to do. In short:

• On Releases → Production source, enter the previous version and click Roll out: your source keeps the current and three previous versions.
• The panel puts itself back while its last database step has not run.
• Servers that already took the new version are put back one by one.

A fault that cannot wait until next week goes through a patch or an emergency update: see Patch and emergency update.

From a terminal

On the panel server, as root:

# What the source serves, which key signs, whether a rollout is possible
corecp-panel dist status

# Fetch, verify and publish the release (the same as the Roll out button)
corecp-panel dist publish 0.51.0 --reason "week release 0.51"

# Update the panel itself: backup, swap, check, finish
corecp-panel self-update --to 0.51.0
corecp-panel self-update status

# Back to the previous panel package, while the last database step has not run
corecp-panel self-update --rollback

On a server you see which version runs and which source it trusts:

corectl --version
corectl update trust

See also

• Managing releases — the releases board, the rollout plan and the emergency lane.
• Patch and emergency update — a fix between two week releases.
• How CoreCP versions work — what the numbers and the suffixes mean.