8. Troubleshooting

This guide describes Lawliet 1.5.13.s (the current release). Screens, commands and procedures may change in later versions; the usage guide is updated with each release. Check the version your server runs under Settings > Platform or with ./lawliet version, and see the Releases page for what changed.

Symptom, cause, fix. Start every investigation with the same three commands on the server:

sudo ./lawliet status          # what is unhealthy
sudo ./lawliet doctor          # findings with their fixes
sudo ./lawliet logs backend -n 300

If the fix below does not help, send sudo ./lawliet diag --since 24h and the build id to support (Doctor, diag and support).

8.1 Installation and start#

Symptom	Cause	Fix
The first start takes several minutes; services show `starting`	Normal. The first start loads the images (about 830 MB per CPU type), creates the database and runs every migration. The backend is allowed 3 minutes before its health check counts	Wait. `install` waits up to 7 minutes, `start` up to 5. Then check `sudo ./lawliet status`
`Not everything became healthy within 7 minutes` (install) or `Not all services are healthy yet` (start)	A slow disk or CPU, too little memory, or a real fault	Run `sudo ./lawliet status` again after a minute. If the backend is still not healthy: `sudo ./lawliet logs backend -n 300`, then `sudo ./lawliet doctor`
`Docker is installed but not running (or this user cannot reach it)`	Docker stopped, or the account is not root / not in the `docker` group	`sudo systemctl start docker`; run `./lawliet` with `sudo`
`Docker Compose not found`	The Compose plugin is missing	Install `docker-compose-plugin` (Installation 2.2)
`A Lawliet deployment named 'lawliet' already exists on this host`	Containers of an earlier install exist, running or stopped	Installation 2.6
`80 or 443 is already in use on this host`	Another web server uses the port	Accept the port the installer picked (it is in every URL it prints), or stop the other service and install again
`This bundle has no images for this server (<arch>)`	Wrong CPU type	A bundle for this CPU (`uname -m`)
`Bundled image ... failed its checksum`	Damaged copy	Copy the bundle again; check `sha256sum -c`
`Database schema is not stamped`	The backend could not prepare the database on first boot	`sudo ./lawliet logs backend`; send `sudo ./lawliet diag` to support
`Not installed yet in <dir> - run: ./lawliet install`	You are in the wrong directory (for example the old one after an update) or `.env` is missing	`cd` to the installation directory
`Nothing is running` but the console works	The CLI looks at a different Compose project name	Use the project name `status` names (`COMPOSE_PROJECT_NAME=...`). Do not start a second copy

8.2 Console access#

Symptom	Cause	Fix
The login page loads, pressing Sign in does nothing	Plain-HTTP mode: port 8000 is not reachable from the workstation (the page comes from 3000, but calls the API on 8000)	Open 8000/tcp from operator workstations, or move to HTTPS (`sudo ./lawliet tls on`)
`Invalid host header` (HTTP 403)	The address in the browser (or used by an agent) is not in `TRUSTED_HOSTS`	`sudo ./lawliet doctor --fix` adds this server's own addresses. For another name: 8.9
The browser warns about the certificate	A self-signed certificate, or one that does not name the address typed	Use the name or address the certificate names, or install a CA-issued certificate (TLS 6.4)
The browser insists on `https://` after `tls off`	The browser remembered HSTS for that host name	Use `https://` (443 keeps answering) or clear the browser's HSTS entry for the name. IP addresses are not affected
Red banner Encryption is off: passwords, sessions and agent traffic cross the network unencrypted	TLS is off	`sudo ./lawliet tls on` (TLS 6.8)
Banner Four-eyes is inactive until a second approver exists.	Only one approver exists	Create the second approver (Installation 2.9, step 5)
Only the activation screen opens	No valid licence	Installation 2.10
The platform is paused until a renewed licence is installed.	The licence has expired	Install the renewed key. Data is untouched
A menu entry is missing for a user	The role lacks the permission, or the module is not in the licence	Settings > Users > Adjust, or Settings > Platform, panel License (Modules in this build)

8.3 Certificates#

Symptom	Cause	Fix
Agent install by IP fails: curl error 60, no alternative certificate subject name matches target ipv4 address	The certificate does not name that IP address	Self-signed: `sudo ./lawliet tls repair`. CA-issued: a certificate that names the address, then `sudo ./lawliet tls`
curl error 60 self-signed certificate on an agent host	The host was not given the server's certificate	`sudo ./lawliet tls --export`, copy `lawliet-server.pem` to the host, use the command from `./lawliet token`
Every agent offline right after a certificate change	The agents trust the previous certificate	TLS 6.6
`status` says `certificate does not cover <address>`	A new network address on the server, or an old self-signed certificate	`sudo ./lawliet tls repair` (TLS 6.5)
Console certificate card: Expiring soon or Action needed	Under 30 / under 7 days, expired, or an address not named	Self-signed: `sudo ./lawliet tls repair`. CA-issued: renew with your CA, then `sudo ./lawliet tls`
Card: certificate not valid yet	The server clock is behind	8.5
`tls repair` stops: `N of M agent(s) hold the new certificate` and Nothing was switched	Some agents are older than 2.1.9.s or offline	Run it again later to keep waiting, use `--ack`/`--force`, or `tls cancel` (TLS 6.5)

8.4 Agents#

Symptom	Cause	Fix
Agent installed but never appears	Port 443 (plain HTTP: 8000) blocked from the host; wrong or old enrolment secret (HTTP 401); certificate not trusted; licence seats all used	From the host: `curl -vk https://<server>/health` must return `"status": "healthy"`. Take a fresh command from `./lawliet token` or Environment > Install agent. Check seats in Settings > Platform > License
Agent shown offline	No heartbeat for 5 minutes: the host is off, the agent service stopped, the network path is blocked, or the certificate changed	Check the host and its agent service; `curl -vk https://<server>/health` from the host; after a certificate change see 8.3. Agent details: Agents overview
Agent appears once, then goes offline	Intermittent network path, or the host cannot resolve the server's name	Use the IP address, or fix DNS on the host
Agent online, but commands never run	The host clock differs from the server's by more than 10 minutes behind or 60 seconds ahead; commands are refused	Fix the host's clock (8.5)
`./lawliet token` says No enrolment secret is set	`AGENT_REGISTRATION_SECRET` is empty	`sudo ./lawliet token --rotate`
Agents listed as still using plain HTTP	Enrolled while TLS was off, or before a move from plain HTTP	Reinstall each over HTTPS with the current command; their tokens crossed the network in clear

8.5 Clock and time#

Clocks matter in four places:

Where	Tolerance	Symptom when wrong
Agent commands	10 minutes behind, 60 seconds ahead of the command's time	Agent online, commands never run
Two-factor codes	The current 30-second code only	That code was not accepted. Check the time on your phone
Licence	The licence start time	The licence is not valid yet
Kerberos (Windows sign-in, remote deployment)	5 minutes	Kerberos logons fail for everyone at once

Check and fix the server clock#

timedatectl                                   # look for "System clock synchronized: yes"

What you see when it is right: System clock synchronized: yes and the correct time. Otherwise use one of the two fixes below.

With an NTP server on your network:

sudo sed -i 's/^#\?NTP=.*/NTP=<your NTP server>/' /etc/systemd/timesyncd.conf
sudo systemctl restart systemd-timesyncd
sudo timedatectl set-ntp true
timedatectl timesync-status

With no time source (air-gapped, no NTP): set it by hand from a trusted clock, and check it regularly:

sudo timedatectl set-time 'YYYY-MM-DD HH:MM:SS'

Containers use the host's clock; no restart is needed. Fix agent hosts the same way (Windows: w32tm /resync against your domain's time source).

sudo ./lawliet doctor reports an unsynchronised server clock, and sudo ./lawliet diag lists agent clock offsets (ok under 1 minute, warning, critical from 5 minutes).

8.6 Remote deployment and WinRM#

The console's remote deployment (Environment > Deploy agents remotely) explains each failure in its checklist. The common ones:

Message	Cause	Fix
<host> answered on port 5986, but not with TLS - that is a plain-HTTP WinRM listener (5985 is WinRM's HTTP port), and this credential connects over HTTPS.	The host has only the plain-HTTP WinRM listener that `winrm quickconfig` creates	Recommended: clear the credential's port (5986 is used) and add an HTTPS listener on the host (Windows agents 12.8). Or switch the credential's WinRM connection to HTTP 5985 (insecure: not protected by TLS; a second administrator must approve)
Port 5985 is WinRM's plain-HTTP port, but this credential connects to WinRM over HTTPS. (when saving the credential)	5985 typed as the port of an HTTPS credential	Leave the port empty, or switch the credential to HTTP 5985 as above
Port 5986 is WinRM's HTTPS port, but this credential connects over plain HTTP.	5986 on a credential set to HTTP 5985	Leave the port empty to use 5985, or switch the credential back to HTTPS 5986 (recommended)
<host> answered on port 5986, but its WinRM certificate could not be validated	The listener's certificate is not trusted, or does not name the host as entered	Paste the CA that issued it (your AD CS root, or the host's self-signed certificate) as WinRM server CA on the credential; make sure it names the host as entered
Warning WinRM over plain HTTP (insecure) on every checklist	The credential is set to HTTP 5985	Use only on a trusted network, temporarily; switch back to HTTPS 5986 or roll out by Group Policy
<host> did not answer on port <port> within 15 s	The host is off, or a firewall drops the connection	Allow the port from the Lawliet server in the host's and the network's firewall
The host refused the login for <user> over WinRM.	Wrong password, account locked/disabled/expired, not an administrator, or NTLM refused by the domain	Check the account before retrying: repeated failures lock a domain account
Kerberos: Check that this server reaches the KDCs (domain controllers) on TCP port 88	No path to the domain controllers	Open TCP 88 from the Lawliet server to the KDCs; check Environment > Deploy agents remotely > Kerberos realms
Kerberos clock skew	The server and the domain differ by more than 5 minutes	Synchronise the server with the domain controllers (8.5)

Full procedures: Remote deployment from the console.

Symptom	Cause	Fix
The administrator password is lost		On the server: `sudo ./lawliet passwd` (CLI 3.17)
Another user's password is lost		On the Sign in page press Forgot?, enter the email on the account and press Send link. This needs outbound email (Installation 2.11). Otherwise see Users, roles and approvals
Authenticator lost, recovery codes at hand		On the code screen press Use a recovery code and enter one (each works once)
Authenticator and recovery codes lost (another user)		An administrator presses Reset 2FA for that user under Settings > Users
Authenticator lost on the only administrator account		`sudo ./lawliet passwd` and answer yes to Turn two-factor OFF for this account?; enrol again after signing in
Too many failed 2FA attempts. Try again later.	5 wrong codes	Wait 15 minutes. Check the phone's clock
A critical action stays Sent for approval	No approver has acted	The approver opens Approvals > Waiting for me
An approved request cannot be executed any more	The execution window (default 15 minutes) passed, or the request expired (60 minutes without a decision)	Make the request again
`409 MANAGEMENT_WOULD_LOCK`	The change would leave nobody able to approve user or policy changes	Choose a pair of approvers who can approve each other (Installation 2.9)

8.8 Updates, backups and licence#

Symptom	Where to look
An update refused the bundle	Updates 4.3, If it fails
A delta was refused	Updates 4.7
The new version does not start	`cd <new dir> && sudo ./lawliet rollback` (Updates 4.6)
The console cannot stage updates (cannot write .../updates)	`sudo ./lawliet start`
A backup or restore failed	Backup and restore
After a restore on another server, two-factor sign-in fails	The `.keys` file was missing (Backup 5.4)
A licence key is refused	Installation 2.10, If it fails
The licence stopped working after moving to new hardware	The fingerprint changed: send the new one (`sudo ./lawliet licence`) to your vendor

8.9 Add an address to TRUSTED_HOSTS#

Use when operators or agents reach the server by a name or address the installer did not know (a new DNS alias, a NAT address).


Risk	Low. A typo leaves that address refused, nothing else
Rollback	`.env` is backed up in step 1; copy it back and run `sudo ./lawliet start`

Keep a copy: sudo cp -p .env .env.backup.$(date +%Y%m%d-%H%M%S).
Edit .env and add the name to the comma-separated TRUSTED_HOSTS line, for example TRUSTED_HOSTS=10.0.0.5,lawliet.corp.example,localhost,127.0.0.1,lawliet-alias.corp.example. Host names and addresses only; never *.
If operators will open the console at that name, also add its origin to CORS_ORIGINS, for example https://lawliet-alias.corp.example.
Apply (a restart would not re-read .env):
```
sudo ./lawliet start
```
If agents or browsers will use that name over HTTPS, the certificate must name it too (TLS 6.4 or 6.5). Installing a certificate with ./lawliet tls or tls repair adds its names to TRUSTED_HOSTS by itself.