Checking, exporting and watching the audit log

The audit log is this panel's evidence: who did what, when, and whether they were allowed to. This page is about the three things around it that until now you could only reach over SSH:

1. the chain — the control that says whether anything in that log has been edited;
2. the export — the log as a file, filtered exactly as you have it on screen;
3. detection — the decoy credentials (canaries) that give away somebody looking around with stolen keys, and the channel that wakes you about it.

The chain, the canaries and the paging channel are administrator's work: you see them only as Global admin. The export is not — it belongs to the screen and works for anybody who may read the audit log, always within their own scope.

1. The audit chain

Go to Insight → Audit log. Above the table is the Audit chain block.

Every line in the log is hashed onto the previous one as it is written. Edit one line afterwards and its own hash breaks; remove one and the line after it points at something that is no longer there. That is what "verify the chain" checks.

What the three tiles say

Under them is the head hash: the hash of the newest line. That value is written to the build server every minute, on a machine this panel cannot reach. Rewriting the whole chain produces something internally consistent — and changes that value. So compare it with the copy now and then:

# on panel1: what the panel says the head is
ssh root@panel1.corecp.dev 'corecp-panel audit status'

# on the build server: what was delivered from panel1
ssh root@build.corecp.dev 'tail -1 /srv/corecp/audit/panel1.corecp.dev/head.txt'

Verifying

Press Verify. The panel asks first for a fresh confirmation of who you are (your authenticator app or your passkey): a verdict about evidence should come from a session that has just proved whose it is.

The server then recomputes the whole chain and answers in one sentence:

• The chain is intact — no line has been edited or removed, and the head belongs to the newest line. It says how many rows that was and how long it took (on a log of fifty thousand rows, about two seconds).
• The chain does not add up — a table underneath names every line that does not fit: the number, when it was written, which operation it was, and whether it was edited (its own hash no longer matches) or its link is gone (its predecessor is no longer there).

If you get the second answer, do not carry on working on this panel: take a copy of the database, compare the head hash with the build server's, and follow docs/dr-runbook.md.

The same from the command line, for instance in a cron job:

ssh root@panel1.corecp.dev 'corecp-panel audit verify'
# exit 0 = intact, exit 1 = something is wrong (and it names the row)

Who else passes this verdict, and why they agree

The button is not the only thing that asks this question. The monthly restore drill on the build server really restores the backup and checks the chain in that copy — which is the evidence that the log still adds up after a recovery as well.

What matters there is that both ask literally the same question. The verdict "does this row fit the chain" lives in the database itself, next to the hash function it is a verdict about, and the button and the drill both call it. So it cannot happen that the panel says "intact" while the drill rejects the same rows.

That is not theoretical: it is exactly what happened, because the drill had remembered an older version of the question in which rows were compared by sequence number. On a busy panel the order of the numbers is not the order of the chain — two simultaneous operations get their number before they get their place in the chain — and then such a comparison calls healthy rows broken.

1b. Finding one request by its support ID

When the panel refuses somebody something, it shows a support ID under the message — a short code naming exactly that one request (see The support ID on an error message). Somebody reporting a problem is asked to pass it on. This is where you use it.

Go to Insight → Audit log and paste the code into the search box. The search covers the support ID as well as the operation, the person and the reason, so you get the entry for precisely that request: what was attempted, by whom, from which address, and whether it was allowed.

Two things are worth knowing about it.

It stays inside your own scope. Searching a code somebody gave you does not widen what you may read. A reseller who searches an ID belonging to a request in another group finds nothing — the search narrows, it never opens.

A second attempt has a second code. Every request gets its own, so somebody who pressed the button twice has two codes and two entries. Ask for the one they were looking at.

If the entry alone is not enough, the panel machine can collect the audit entries and the log lines of that one request into a single file:

corecp-panel support bundle --config /etc/corecp-panel/panel.yaml \
    --request-id <code> --out /root/support-<code>.tar.gz

What is in that file, and why it is safe to send, is in Sending diagnostics.

1c. Searching and filtering on the screen

Above the table are the two controls that make the log smaller: a search box and a dropdown for the outcome.

• The search box looks at the action, the subject and the requester. Type account.create and you keep only account creations; type a domain name and you keep everything that was done to that domain, by anyone.
• The dropdown is called Filter by outcome and has four positions: all outcomes, succeeded, denied and failed. "Denied" is the interesting one — those are the actions somebody attempted and was not allowed to take.

Search:  account.create        Filter by outcome:  Denied
→ every attempt to create an account that was refused

Two things to know. The filters work inside your own scope — they make the list smaller, never larger, and a reseller with an empty filter still sees only their own group. And the export in the next section follows exactly these two settings, so filtering before you export saves you the work afterwards.

Since round 5 both controls also say out loud what they filter. You will not see a difference — the purpose is already written above them — but somebody driving the panel with a screen reader will: they used to hear only the chosen value ("All outcomes") and now hear what that value belongs to as well.

2. The log as a file

Top right of the audit screen are Export as CSV and Export as JSONL (on a phone, behind the "…" button).

Two things are worth saying here.

What you download is what you have on screen. The search term and the outcome filter travel with the download. Search for account.create and export, and only those lines are in the file.

And it is your scope, not somebody else's. The server applies the same filter it applies to the screen: a reseller gets the lines of their own group and nothing outside it. Editing the link by hand gains nothing.

The file is written oldest-first, so you can walk it or append to it. On very large logs the export stops at fifty thousand rows; those are the newest fifty thousand, and the last line of the file says there were more. For the whole log, use the command line:

# everything from event 0, as JSONL, in chunks of 5000
ssh root@panel1.corecp.dev 'corecp-panel audit export --since 0 --limit 5000' > audit.jsonl

Every export writes an audit line of its own (audit.export) with the number of rows, the format and the filter it was taken through. A copy of the log leaving this machine ought to leave a trace.

3. Canaries and the paging channel

Go to Settings → Platform security. Under the authentication policy are two blocks.

Canaries

In three places sits something that looks real and that nothing legitimate ever uses: a server that is not a server, a bootstrap credential no node was ever issued, and an API token in panel.yaml that is granted nothing. Anybody who tries one found it somewhere — and that is an alarm on the spot.

The block shows, per canary, where it sits, how often it has been touched and when that last was. What it never shows is the value itself: a canary works only while an intruder is the only one who meets it. If the API-token decoy is missing, the screen says so with the command beside it — that decoy has to be in panel.yaml to be findable at all, and no web request may edit that file:

# on panel1, as root: plant whatever is not there yet
corecp-panel canary ensure
corecp-panel canary status

The paging channel, and the test that proves it

The second block is the channel that wakes you: where an alarm goes, how far the classifier has walked the audit log, what is queued, and — the most important and least dramatic of the three — how many messages are sitting undelivered in the spool file. A growing spool means the channel is broken and nobody has been told, because the thing that would tell them is the channel.

So there is a Send a test page button. It sends one message with TEST in the subject to the addresses in panel.yaml, and the screen then says whether it was delivered and where. Here too the panel asks for a fresh confirmation first: the button makes somebody's phone go off, at night as well.

If alerting is switched off (alerting.enabled: false) the button refuses with that reason, rather than queueing something nobody will get.

The same from the command line:

ssh root@panel1.corecp.dev 'corecp-panel alert status'   # channel, queue, spool
ssh root@panel1.corecp.dev 'corecp-panel alert test'     # one test message
ssh root@panel1.corecp.dev 'corecp-panel alert kinds'    # what pages, and why

What is deliberately not here

Break-glass, the master key (KEK), the SSO clients and the certificate authority have no button in the panel, and that stays that way. An emergency lever that runs through the panel does not work when the panel is the problem, and a key that signs the whole fleet's trust should not be reachable from a browser session. What the panel does show of them is the state of play with the command beside it; see Security → Break-glass.