Your servers' backups
There are two kinds of backup in CoreCP, and they belong to different people. A hosting account's backup is the customer's: they make it, to their own storage, and they put it back themselves (see Your own backups). A server's backup is you
Written for: Administrator
There are two kinds of backup in CoreCP, and they belong to different people. A hosting account's backup is the customer's: they make it, to their own storage, and they put it back themselves (see Your own backups). A server's backup is yours: the whole machine, with /etc, /home, every database and the state CoreCP keeps for itself.
This page is about the second kind. You will find it under Servers → Backups.
A destination first, then servers
Where your fleet's backups go is recorded once, not once per server. Under Servers → Backups → Destinations you record a destination:
- the kind of storage — a rest-server, an sftp server, or S3;
- the address of that storage;
- the path under which every server gets a place of its own;
- the credentials, where the kind needs them.
Every server gets a repository of its own under that path. That is not a detail: restic deduplicates within a repository, and in a shared one every server can reach every other server's snapshots — including to delete them. The password to such a repository is generated on the server itself. The panel keeps a sealed copy of it, locked with the panel's master key, so that a server that is lost can still be restored on another one. For a rest-server destination, the panel also keeps the server's login to that storage. The copy is taken when you attach the server, and again whenever the password on the server changes. The panel never shows it on a screen, and the assistant cannot reach it.
On Servers → Backups, a server whose copy is missing says so. Usually this means the panel could not reach the server yet. Its backups keep running in the meantime, and the panel tries again within minutes.
Still write the repository password down somewhere off the server. The panel's copy is only as good as the panel's master key and its own backup. Without the password, nobody can read the snapshots, us included. You will find it on the server in /etc/corecp/secrets/backup.env.
Try it before you rely on it
Beside each destination is Validate destination. It first asks which server should do the measuring, and that is deliberate: whether storage is reachable is a property of the route between one machine and that storage. A green tick with no machine attached to it says nothing.
The chosen server then walks a checklist — does the storage answer, is there a repository, does anything actually go in — and writes a small probe snapshot which it removes again. That last part is the point: storage that answers politely and cannot be written to (a full disk, a read-only export, an S3 policy that only allows reads) is exactly what this button exists to catch.
If a step fails, the page names which, and says why in restic's own words — for example restic cat: Fatal: unable to open config file: Stat: unexpected HTTP response (401): 401 Unauthorized rather than just exit status 1. Passwords never appear in that line.
A CoreCP backup server is validated from a server that is attached to it. Such a destination has no stored login: the backup server gives every server a login of its own when you press Attach server. Validating from a server that was never attached is therefore refused before anything happens, with "Attach this server first (Attach server); the backup server issues a login of its own to every server." The backup server itself is not in the list of servers to measure from.
In the terminal it is the same measurement:
corectl backup destination check --target sftp \
--endpoint backup@storage.example.net --repo /srv/restic/stck1 --initWhich destination? The panel proposes one
With more than one destination you do not have to remember which has the most room. On the backups page of a server that writes nowhere yet — and in the add-server wizard, once the machine has joined — the panel shows a recommendation, with the measurements it rests on underneath it.
There are five of them, in the order they decide:
- another location. Does the storage stand somewhere other than the server itself? A snapshot beside the machine it protects survives a failed disk, not the building. A destination in the same location is still offered — last, and with a warning beside it.
- what the last check said. A destination whose last validation failed is ranked last and is never the recommendation.
- free space, measured on the storage machine itself.
- how many servers already write there — fewer is better spread.
- the measured distance between this server and that storage.
What the panel has not measured says so. "Free space not measured" counts neither as "plenty of room" nor as "nearly full": it sits exactly between the two. The same goes for a location nobody filled in.
Free space and distance exist only when the destination's address is a machine this panel knows. For storage at a third party the panel cannot know them, and it says so rather than guessing.
The destinations that were not chosen are folded away underneath, each with its reason. One that cannot be used at all — belonging to another server group, with no stored login, or the server itself as its storage — is listed too, in as many words.
Fill in the location of a destination (Servers → Backups → Destinations) and of your servers, in your own words: "fra1", "ams-rack-3". The panel only compares whether two locations are equal; empty means unknown.
When a server runs websites, databases or mailboxes and no destination is recorded for it, the server advisor says so, with a button to that same page.
Attaching a server
Switch the backup role on from the server's configuration panel, or pick a destination at the backup role in the add-server wizard. Attaching does in one go what is otherwise four separate acts: install the role, put the credentials on the machine, write the target into node.yaml and create the repository.
With an sftp destination the server generates a key of its own and prints the public half. Put that in the storage machine's authorized_keys — that is the one act that belongs on the other side.
If you add the server through the wizard the machine does not exist yet, so the choice is recorded and the server's own backup page says afterwards that it still has to be attached, with the button beside it.
The Backups tab is there on a server that takes no backups either. It says what is true and which destination the panel recommends — that server is precisely the one you have to be able to reach. There is no schedule form on it: without storage the server would refuse every change anyway.
Moving to different storage is Move to different storage, above "Where this server writes". Mind what it does and does not do: the snapshots in the old repository stay there — they do not move with it and they are not cleaned up — and the first run to the new storage is a full one, because there is no history there yet to build on.
Schedule, retention and pruning
Servers → server → Backups shows three runs in a row, in the order the night runs them: the server, then the accounts, then the retention pass. They are staggered because the account pass reads what the server pass has just walked.
Each run is picked as how often plus at what time — "every night at 01:00", "every week on Sunday at 04:00". Behind Advanced is the calendar expression itself; that is literally what the server is given, and you only need it for schedules the controls cannot say (twice a night, weekdays only, a list of dates). Under each run is when it fires next, out of the server's own answer.
A schedule the server cannot read is refused, with the server's own sentence under the box it was about. Nothing is saved then — your typo cannot stop a timer. If a timer is stopped anyway (a server from before this version, or a hand-edited file), the page says so in words instead of drawing it green.
You can keep working while the account pass runs. The pass gives the server back after each account. A website, mailbox or password you create in those minutes waits at most for the account being written and then goes through; you no longer see "this node is busy with backup.accounts". An account you remove during the pass is skipped: that does not count as a failed backup. On the server itself, corectl node lock shows who holds the server right now.
Do they all start at once?
Every attach takes the default time, so a fleet converges on one minute by itself — and then ten servers write to the same storage at once. If this server collides with neighbours on the same destination, the screen says so: which machines, on which storage, which slot is the quietest in that night, and how many minutes of room it has around it. One button fills it in; you press save.
What the panel cannot read as a plain clock time is not counted, and it says so. An advice that quietly treated those schedules as free minutes would recommend a time it never checked.
How long is a backup kept?
Retention is four groups — daily, weekly, monthly, and a floor regardless of date. They apply per group: "7 daily" means seven days for each account and for the server layer, not seven snapshots in total. And note what a number in a group actually means: "keep daily 7" keeps the last snapshot of each of the last seven days, not the last seven snapshots. The screen says that under each box, so you do not have to remember it.
Each group has a minus and a plus, and stepping below 1 lands on none — the word, not a number. That is deliberate: "keep none of this group" used to be a minus sign five pixels wide next to a 1, and the two mean opposite things. The floor is the odd one out: its "off" is none as well, and it simply means there is no floor — and setting it to none really removes it, on a pattern and on a single server's own form alike.
You cannot turn all four off. A policy that keeps nothing is refused by the server, with the group it is about named — the backup tool refuses such a policy outright, so the retention pass would fail every week without removing anything, and nothing on the screen would say so.
One rule for the whole fleet
Typing four numbers per server is how two servers drift apart with nothing to say so. Under Servers → Backups → Retention patterns you write the rule down once, give it a name, and point servers at it.
Changing a pattern is two presses, and the first one is the important one. Save records the rule and touches no server. Apply is the second press, and before it happens the panel says what it is about to do:
- how many servers end up with different numbers,
- how many of them will keep less than they keep now — the direction in which snapshots get deleted,
- and how many the panel has never read a retention from, which is neither of the two above and is counted on its own.
If one server is unreachable, the others still get the rule. The report says which one refused and what it said, and one button retries exactly that server. Nothing is rolled back: a server that took the numbers has them, and undoing that because a different server failed would be a second change nobody asked for.
On a server's own page you can see which pattern it follows and whether it has actually taken the numbers. "Recorded, but not on this server yet" is its own answer — it is the honest one for a machine that has been unreachable since you put it on the rule.
Is the cleanup actually running?
The retention pass can fail quietly. The most common way: it takes an exclusive lock on the repository, and a run that was killed while holding one leaves that lock behind — every later pass then fails against it, while the backups themselves keep working and the timers keep looking healthy.
So the server's backup page says whether the last cleanup pass finished, never ran, or failed, and with what. "Nobody has run one yet" is its own answer and never counted as healthy. The pass now clears a lock the backup tool itself considers abandoned and tries again — before its very first step, because such a lock refuses even a read of the repository; a lock a running pass is holding is left exactly where it is.
The note on that page appears whenever there is something to say, and that includes the two findings that are easy to miss: a server that keeps nothing at all, and storage that is running out of room. When there is nothing to say, the page stays quiet.
If your storage refuses deletes (a rest-server started with --append-only), pruning from this server removes nothing. That is not a fault but the design: writing and erasing deliberately do not live on the same machine. The screen says so, and the retention pass then belongs on the storage machine.
On an SSH destination with append-only (ssh-append-only) that is how it is set up: the destination page hands you a setup script that installs the retention timer on the destination itself. The server checks after every run that it is still pruning there. When it is not, the server's advisor shows the row Retention on the destination with what to do: run that setup on the destination as root, or check its timer there.
Restoring
Restore… lets you pick a snapshot. By default it lands in a separate directory and the running system is untouched — which is what you want when you only need to see what was in /etc last week.
To write over the running system you have to type the server's name out. The panel does not fill it in for you: a confirmation the panel could produce is not a confirmation. Run a reconcile on that server afterwards, because the state underneath the services has changed.
Old account backups that stop growing
A hosting account that the panel manages is backed up as one browsable snapshot. You can list the files, mailboxes, aliases or certificates in it without unpacking anything. Accounts that were backed up before this leave a group of older snapshots behind, and that group receives nothing new. Left alone, the cleanup would never remove it, because the cleanup counts from a group's newest snapshot.
The panel therefore records such a closed group on the server's backup page. You can still restore from it until it expires. It expires once the longest period you keep backups for has passed since the group closed. The panel then removes exactly those snapshots, by their ids, and checks afterwards that they are gone. A newer account with the same name keeps all of its own backups. If the removal cannot be confirmed, the group stays in the list with the reason, and the panel tries again later.
An account the panel does not know
An account that was made on the server by hand, and not in the panel, has no id from the panel. It is still backed up every night, but in the older form: one archive per run that you cannot browse without unpacking it. The server's health shows this as one amber row, Account without a panel id, which names those accounts. Amber means it wants a look and nothing is broken: the server stays healthy and keeps taking updates. Accounts the panel does know get their id from the next synchronisation (Servers → the server → Synchronise).
Deleted accounts
Delete an account and it leaves the panel, not the backups. Under Servers → Backups → Deleted accounts it is listed with its restore points, its owner and the date it expires: the longest retention window of its server, counted from the deletion. Change the retention and the date moves with it. A week before, it appears in the daily digest of the realm's owner.
- Restore account: choose the restore point, the target server (its own, or another web server — even with the old server off: the panel then lends the target a temporary read grant on the backup server and hands it the keys for that one run only), dry-run, type the account name. When the task ends the account is back in the panel under its original owner, with its original package (or the one you chose in the wizard). If the username is taken on the target you are told so and nothing happens.
- Hold: a held account does not expire, with a reason and an optional end date. It can be released at any time.
- Purge now: after a second factor and the typed account name the panel removes exactly this account's snapshots, by their ids, and only once a listing confirms they are gone does it count as purged. If that cannot be shown it stays as purge pending and the panel retries later.
- Find left-behind backups: backups of accounts that vanished before this feature are found and recorded as found afterwards. They can be restored at once but are never purged until you confirm the found list. An account still on the server but not in the panel is reported as such and left alone.
The assistant can show the list, never restore or purge.
Who is behind?
The list under Servers → Backups is ordered by how far behind a machine is: one that has never produced a snapshot first, then the stalest. A server with no backup role is on the list too, saying so in as many words — precisely because "it is not in the list" reads as "nothing is wrong".
Behind is the same line the server itself uses: corectl doctor fails a machine whose newest snapshot is older than two days. One missed night is a hiccup, two is a pattern.
How current is the list?
The panel reads the backups of every server itself, every two minutes, beside its other reads across the fleet (at most eight servers at a time). A snapshot or restore test is on the list within two minutes wherever it was started: by the server's timer, with corectl backup node or corectl backup restoretest on the server itself, or from the panel. There is no need to press Refresh to see it.
Under each server's state is when the panel last read it. When a read fails — the server is rebooting, or briefly unreachable — the last row stays with its age, and under it is why the read failed: "read failed …: …". Such a server is never "never"; that word only means the repository was read and was empty. A server the panel has not been able to read at all is "not read yet", with the reason. One server that does not answer does not hold up the others.
A backup server is listed as a backup server
A server that keeps other servers' backups (corectl backup target serve, see below) is listed as a backup server: with the servers whose backups it keeps and the free space on the disk the repositories are on. It needs no destination of its own, so it is not listed as "no destination", not as a warning, and not under Problems or Unprotected.
What reaches the platform overview
The attention list on the platform overview names only what needs attention about backups:
- a server without a snapshot within the two-day line;
- a server whose repository was unreachable on the last read;
- a server that takes snapshots and has never had one restored as a test.
A backup server, a server the panel has not read yet, and a server linked to a backup server from the terminal (without the panel recording the destination) are not on it.
Is it ever tested that they can be read back?
Yes. Once a week every server with the backup role takes a sample out of its newest snapshot on its own, reads it back and throws it away again. The server's backup screen says when that last went well; when it went wrong it turns up in the list of problems, with the server's own sentence beside it. A server that has snapshots and has never had one read back is a promise — this turns it into a fact.
You can also run it yourself with Run the restore test now, and switch it off with corectl backup configure --restoretest off.
A backup server of your own that lets nothing be deleted
The safest destination is a CoreCP backup server: a server of your own (b1 in production) that keeps your other servers' backups and refuses to ever delete one. Whoever takes over one of your servers can write new snapshots with it, but cannot erase old ones or read another server's.
- Set the backup server up once from the terminal (see below). After that it is an ordinary server in the panel.
- Under Servers → Backups → Destinations → Add destination, choose the kind CoreCP backup server and fill in its address, for example
https://b1.example.net:8000. You do not fill in a login: when a server is attached, the backup server gives it a login of its own. - Attach your servers as usual. The panel asks the backup server for a login and a key and puts them on the server; the panel itself keeps neither. This needs full access to both servers.
Did a link stop half-way? The login goes to the server first, then the address together with the backup server's certificate, and only then does the server make contact. If a link still stops (the backup server was briefly unreachable, say), you see it on the server's backup page and in the destination's list of servers: Half-linked — "no login for … on this server — link it again" — or the step that failed. Click Link again: it completes the link from that step. Nothing needs doing on the server itself.
Only servers that have joined the panel. A server whose join is still outstanding, or for which the panel has no address yet, is not in the list under Attach server: the backup server lets each server in from the addresses the panel knows for it, and without one the port would stay shut to it. Over the API you get "Join {server} to the panel first; after that it can write backups here." Nothing is created in that case. The backup server itself is not in the list either: its own copy is only worth something on another machine.
You see every step. Linking runs in six steps: login on the backup server, credential on the server, backup role, configure the target, create the repository, and the prune key. Each step is pending, running, done or failed; a failed step shows its own error, and a step waiting for a system update on the server to finish says so ("Waiting for the server's apt lock"). Reload the page and the list is still there while the linking carries on. Press Attach server again after a failed step and it continues from that step, with the login the backup server already issued.
Did a server's address change (joined again, or through the IP manager)? Its row in the destination's list then says the backup server still lets it in from the old address. Click Update addresses: the backup server lets it in from the new address, with the same login.
The free space on the destination's card is that of the directory the backups live in (/srv/restic, or the path you gave backup target serve --path), naming that path: "92% · 11.0 TB free on b1 (/srv/restic)". If it says "(/srv/restic, on /)", the backups are on the backup server's system disk. "Not measured" means the backup server still runs an agent that does not take this measurement.
"Not measured" means the server did not read its repository this time: there is no login, or the backup server did not answer in time. The reason is shown beside it. The page therefore never hangs, and at most one such read runs per server at a time.
Cleanup happens on the backup server. An attached server can no longer remove anything itself, and Clean up on its page says so. Every night at 05:30 the backup server applies the retention you set on the server's page — a change there is passed on by itself. It counts in time windows ("everything from the last 7 days, one per day") rather than in numbers, so fake snapshots cannot push the real ones out. A policy with only "keep the last N" is refused for that reason.
The Immutable check in the server's health is red for this kind when the backup server would allow a delete after all.
From the terminal
On the backup server:
corectl backup target serve # rest-server, append-only, TLS
corectl backup client add w1 --from 203.0.113.10 # login and key for w1, shown once
corectl backup client set w1 --keep-daily 14 # w1's retention
corectl backup target prune --dry-run # what the cleanup would remove
corectl backup target statusPort 8000 follows the servers automatically. Every serve, client add, client set --from and client remove puts the backup server's firewall right at once: open to exactly the addresses of the attached servers, shut to one you removed — also on a backup server that has no other roles at all. You never run corectl firewall apply for it. corectl backup target status shows "open to" from the firewall that is actually loaded; if that differs from the attached servers it adds a MISMATCH line, so the status never claims an opening that is not there. If the firewall cannot be applied, the command keeps the login, says what failed and leaves the port as it was (never wider); the status shows the failure until corectl firewall apply succeeds.
On the server that writes to it (the login and the key are read from stdin, never as an argument):
corectl role add backup --target rest-append-only --endpoint https://b1.example.net:8000 --repo w1
corectl backup configure --ca-cert /etc/corecp/agent/ca.crt
corectl backup credential set --rest-user w1 --rest-password-stdin
corectl backup init
corectl backup target-key add --password-stdinEvery --…-stdin follows the same rule: an empty input or a broken pipe, or the ordinary option given as well, and the server refuses the command instead of storing an empty password. One value is read per command.
The backup server also keeps an eye on the panel and mails when it has not answered for longer than the time you set:
corectl backup target watch --url https://p1.example.net --after 10m \
--mail-to ops@example.net --relay-host mail.example.net --testDoes your mail server want a login? Add --relay-port 587 --relay-security starttls --relay-user <login> --relay-password-file /etc/corecp/secrets/deadman-relay.pass (the file holds only the password, chmod 0600). The backup server logs in with AUTH PLAIN or AUTH LOGIN, whichever your mail server offers, so a mail server that only knows AUTH LOGIN works too. --test sends a test mail straight away; if the mail server and CoreCP have no login method in common, the message names the methods on both sides.
The panel's own database goes to the same server with pgBackRest; how to set that up is in the panel's recovery plan.
See also
- Restoring a backup — the journey from snapshot to verification.
- Your own backups — the backup a customer makes and restores themselves.
- Adding a server — where the destination sits in the wizard.
- Backup destinations: NAS, Linux server or FTPS