Rebalancing accounts across your servers

A server fills up, or you want to take one out of service. Customers have to move — not one service of one customer, but whole accounts, and preferably in one go. Rebalance servers does that: you pick a destination, you pick who travels, you read what would happen first, and only then does anything move.

Every customer travels as a whole: their website, their database, their mail, their DNS and their backup target, with one moment at which they cut over. That is the most important property of this screen. Move those five services separately and there is an afternoon in which a customer's website is on one machine and their database is still on another — which is exactly when things break.

You will find it under Migration → Rebalance servers.

When to use this, and when not

This screen is for machines that are already yours. For the other two cases there are other tools, and they are not going anywhere:

For one exceptionally large account the live migration is still the better tool: it syncs in the background for days and cuts over at the very end. Rebalancing works differently — it takes an archive and holds the account briefly — and for ordinary accounts that is faster and simpler.

Where it lives, and how you get back

Rebalancing runs live under Migration › Rebalance servers. All three screens — the list of runs, the page where you start one, and the report of a single run — carry the way back in the top left: the report and the start page return you to the list, and the list returns you to Migration. On a phone that reads ‹ Migration instead of a trail, so one press takes you back without reaching for the browser's own back button.

Step 1 — where to, and who travels

Press New rebalancing.

• Destination. The machine that will serve these customers. Only machines the panel has really heard from are offered, and it has to carry the services you take with you: a server without the mail service cannot take mailboxes.
• Accounts. Search by name, or use the rule Everything on <server> — which is usually what you want when you are emptying a machine. The rule is expanded to the names themselves straight away, so the report can be compared one-for-one with what you approved.
• Services. All five travel by default. What an account does not have is skipped.
• Copy at once. How many accounts are copied at the same time. Two is a good start: the source machines keep serving their customers meanwhile, and a higher number does not make the copying faster — it makes the source slower at everything else. Cutting over is always one customer at a time, whatever you put here.

Step 2 — look at what would happen first

The start button does not exist yet. That is deliberate: first you press Check N account(s).

That check reads every account's placement and asks the servers for their zones. Nothing changes. What comes back is a line per customer:

• Can go — with the services that will travel.
• Note — for example: a zone answered by a nameserver that is not moving with it, and which will therefore keep getting queries afterwards.
• Cannot go — with the reason. The most common one is the database series: if the customer's plan sells MariaDB 11.8 and the destination serves 10.11, that customer stays. That is not a fault in the rebalancing; it is the rule that keeps their data on the series they paid for.

At the top there is one sentence about addresses: records pointing at a machine in this run become the destination's address, and everything else is left exactly as it is. A subdomain a customer deliberately pointed at a third party keeps pointing where it pointed.

Only once the check has answered does the button Move N account(s) to <server> appear.

Step 3 — the run, and the report

From here on it is a report per customer. Each one walks its own step list:

1. Checking — everything that has to be refused before a byte is copied.
2. Held briefly — the account takes no new writes. This lasts as long as its own archive and transfer take, not as long as the whole run. If you had already suspended the account, it stays suspended and this run does not touch that.
3. Copying — a copy per service on the destination, while the old machine carries on serving.
4. Cutting over — the short moment at which the destination takes over, for all of this customer's services back to back.
5. Released, Checking afterwards, Old machine — that last step deletes nothing; it tells you what is still on the old machine.

One customer that gets stuck does not hold up the rest. Their row carries the reason, the run carries on, and at the end it says there is work left with the tally beside it. If a customer gets stuck before they have cut over, they are handed back to their own machine automatically — a customer left frozen is an outage, and not one you want as the aftermath of your own maintenance.

# Following along from a terminal works too: the report is ordinary JSON.
curl -s -b cookies.txt https://panel.example.net/api/v1/account-migrations \
  | python3 -m json.tool | head -30

Step 4 — hand back, carry on, or stop

Click a customer in the report and you see their step list and what the run ran into along the way.

• Hand back to the old server is offered for as long as the customer has not cut over. Their old machine is still serving them at that moment, so handing them back is something nobody notices. What has already been copied stays on the destination for the next attempt; throwing it away is what would make handing back the dangerous operation.
• After the cutover that button is not there, and the screen says why: moving them back is a new rebalancing, not a hand-back.
• Carry on resumes a run at the first customer that is not finished. What landed is not done again.
• Stop the rebalancing leaves everyone who has cut over where they are, and hands everyone who has not back to their own machine.

What is still yours to do

• Nameservers. If a customer's zones name nameservers that are not moving, those zones keep being served there. The check says so beforehand; changing them is a decision you make.
• Clearing out the old copy. A migration deletes nothing a customer owns: the old machine stops serving — the vhost stops, the zone is withdrawn — but files, databases and mailboxes stay. That is exactly what keeps the way back open. The last step of each customer names what is still there and the command that removes it; do that when you are satisfied, not before.
• Cron and loose ends. Anything that does not hang off one of the five services does not travel. Look it over on the old machine before you take it out of service.