Rolling out a release

Since the coreops switchover, Claude makes a release on your command, through /core-cut; a patch or emergency update runs through /core-hotfix. What follows on this page is what happens underneath, automatically.

A new CoreCP version — 0.51.0, 0.52.0, and so on — comes when you (through Claude, with /core-cut) ask for one. It is prepared and tested for you; you decide whether it goes to your servers, and you roll it out. This page describes your part of that, from reading the release dossier to the last wave of servers.

What happens before your turn is not yours to do, but it helps to know it:

• The candidate is made as soon as you ask for a release: a test version of everything finished at that point, for example 0.51.0~rc.1. It is installed on the test setup the way a real server installs it — from packages — and the full check round runs on it. Every check round works in a copy of the code of its own, with the web packages of exactly that version installed in it and the same keys as the development server, so the checks in the browser and on the AI connection really run.
• After at least two hours on the test setup with no new finding, the release itself is made (0.51.0) and a release dossier is waiting for you.
• After that, you roll out whenever it suits you.

On Friday, Saturday and Sunday there is no ordinary rollout. When a candidate is ready late in the week, the rollout simply waits until after the weekend.

Reading the release dossier

The dossier is one page per version, in docs/release-decisions/ of the source code, with the date and the version in its name. You are sent its path. Read it in this order; it takes five to ten minutes:

1. Status at the top: when it says "waiting for the owner's decision", the release is ready for your decision.
2. Gate evidence: which candidate was tested, how many checks passed, and the fingerprint of the log. This is the same evidence your panel checks when you roll out. When there is also a full source gate line, the package check was limited to the core set in ops/config.json; the rest of that check round was measured from source that week instead of from packages.
3. Go/no-go record: nine pieces of evidence, each met, not met or not measured, and a recommendation. A recommendation is not a decision.
4. Rollback: how to go back to the previous version, should you need to.
5. Open points: things somebody still has to finish.

You write your decision on the Decision line of the dossier: GO, GO WITH CONDITIONS, NO-GO or DEFER. Nobody else fills in that line.

Rolling out on the panel

A rollout always goes in the same order: your package source and the panel itself first, then the servers in waves.

1. Put the release on your own package source

1. Go to Releases → Production source (in Dutch: Releases → Productiebron).
2. Check in the box that the Working key is valid. If it says the key has expired or was revoked, rolling out is not possible; follow the key management runbook.
3. Enter the version under Release set, for example 0.51.0, and a short sentence under Reason, for example "week release 0.51".
4. Click Roll out (in Dutch: Uitrollen). The panel asks for your second factor.
5. The log box shows three steps: fetching, verifying and publishing. While verifying, the panel checks that the set is signed by the build server, that every package is right, and that the gate evidence is the evidence the dossier named. If anything is off, the set is refused with the reason, and the previous version stays on your source.

After done your servers can fetch the new version. Nothing has been installed yet.

2. Update the panel itself

The panel must always be at least as new as the servers it manages. It updates itself on the panel server (see From a terminal below). Under Releases → Rollout plan you then see the new version under The panel, with the outcome of the update.

3. The servers, in waves

1. Stay on Releases → Rollout plan. The plan is already there: the panel first, then wave 1 with the test server, then wave 2, then wave 3.
2. Click Start rollout from plan. The panel refuses while it is not ahead itself — finish step 2 first.
3. Follow the rollout on the releases board. The first wave is the canary: only when it is healthy does the next wave start.
4. A server that was already unhealthy before the rollout is skipped, with the reason. Fix that and start a recovery rollout afterwards.

When the last wave is updated, you are done. Inform your customers through your own company's status page and channels; CoreCP itself only sends notifications in the panel and by mail.

Back to the previous version

If something goes wrong, the dossier's Rollback section says exactly what to do. In short:

• On Releases → Production source, enter the previous version and click Roll out: your source keeps the current and three previous versions.
• The panel puts itself back while its last database step has not run.
• Servers that already took the new version are put back one by one.

A fault that cannot wait until the next cut goes through a patch or an emergency update: see Patch and emergency update.

From a terminal

On the panel server, as root:

# What the source serves, which key signs, whether a rollout is possible
corecp-panel dist status

# Fetch, verify and publish the release (the same as the Roll out button)
corecp-panel dist publish 0.51.0 --reason "week release 0.51"

# Update the panel itself: backup, swap, check, finish
corecp-panel self-update --to 0.51.0
corecp-panel self-update status

# Back to the previous panel package, while the last database step has not run
corecp-panel self-update --rollback

On a server you see which version runs and which source it trusts:

corectl --version
corectl update trust

Rolling out a component (not the same as a release)

A release is CoreCP itself — the panel and the agent, under one version number. Beside it, your servers run components CoreCP rebuilds from the vendor's or Debian's source: ProFTPD, Rspamd. Those are not part of a release and you roll them out separately.

Where: Servers → Components & security, click the component. If a package is waiting that has been built and tested on staging, the button Roll out through the patch route is there. You tick the servers, can try it first, and get one line back per server.

The difference from the button above, in one sentence: with a release you decide which version of CoreCP the fleet gets; here you decide which servers get a rebuilt component. Neither happens by itself.

What the server does is the same either way: it checks the signature and the bytes before anything is unpacked, and installs nothing if something does not match. See Components and security for what that page shows you and how to roll a component back.

See also

• Managing releases — the releases board, the rollout plan and the emergency lane.
• Patch and emergency update — a fix between two week releases.
• How CoreCP versions work — what the numbers and the suffixes mean.
• Components and security — the other server software, and the patch route.

The reboot plan and the rollout plan are two plans

Servers → Maintenance also has a plan with slots and an order, and it is not this one. The rollout plan above is the order a new version reaches the servers in; the reboot plan is the order servers go down in at night to restart.

They look alike and have opposite rules. A rollout always starts with the panel, because the panel may never be older than the servers it drives. A reboot night ends with the panel, and only if you approve that separately — because for the length of that reboot there is no panel.