4. Updates, rollback and delta updates

How to move the server to a new version, how to go back, and how the smaller delta updates work.

4.1 Before you update#

1. Read the release notes that came with the version.

2. Check the current state. Fix anything red first:

   sudo ./lawliet status
   sudo ./lawliet doctor

3. Note the build id from status (you may need it for support).

4. Check disk space in the parent of the installation directory. The update unpacks the new version beside the current one and needs about twice the bundle's size free:

   df -h "$(dirname "$PWD")"

5. Plan a short maintenance window. The console is unavailable for a few minutes.

6. Have a copy of the latest backup off the server (Backup and restore). The update takes its own backup too, and refuses to continue if that fails.

Risk: the update stops the running version and migrates the database to the new schema. Rollback: sudo ./lawliet rollback from the new directory returns to the old version with the database as it was before the update (4.6).

4.2 Get the new bundle onto the server#

As for the installation (Installation, 2.3): download the bundle and its .sha256, check it with sha256sum -c, carry it to the server, and check it again there. Put it anywhere on the server, for example /media/usb or /opt/incoming. Do not unpack it over the current installation.

The update checks the bundle's vendor signature itself, before any of it runs. The checksum check protects the copy; the signature check protects against a bundle that was not made by the vendor.

4.3 Update from a file (the standard route)#

1. Go to the current installation directory:

   cd /opt/lawliet-<current>

2. Run the update with the path to the new bundle:

   sudo ./lawliet update /opt/incoming/lawliet-docker-<new>-<customer>.tar.gz

3. Wait. Do not interrupt it. It prints each step:

   Section	What happens
   Upgrade	Shows From <current version> and Using <file>
   Rebuilding the new version from the delta	Only for a delta (4.7)
   Backing up before anything changes	A full backup into ./backups. A failure stops the update
   Unpacking	Checks space and unpacks beside the current directory. OK New bundle: lawliet-<new>
   Checking the new bundle's signature	OK Signed by the vendor; every file matches its manifest, and nothing else is there
   Continuing with <new>'s own update steps	The new version's own script takes over from here
   Loading the new version's images	While the current version is still running
   Carrying your configuration across	.env, deployment settings, TLS certificate, backups. Records what rollback needs
   Stopping the current version	Containers only. Volumes (your data) stay
   Starting the new version	Database migrations run on the first start
   Verifying	Schema at revision <rev> and Running version <new>
   After the update	The new version's post-update steps (4.5)

4. What you see at the end:

     Upgrade complete.
   
     New directory              /opt/lawliet-<new>
     Old directory              /opt/lawliet-<current>  (keep it: ./lawliet rollback returns to it)
     Backup                     /opt/lawliet-<current>/backups/lawliet-<time>.sql.gz
   
     ! Run future commands from the new directory: cd /opt/lawliet-<new>

5. Change to the new directory and check:

   cd /opt/lawliet-<new>
   sudo ./lawliet status

6. Sign in to the console and check the version in Settings > Platform (panel Deployment).

7. Update anything that points at the old directory: backup jobs (cron), monitoring scripts, runbooks.

The old directory stays. Keep it until you are sure you will not roll back.

An unpacked bundle directory is accepted in place of the .tar.gz. A bundle repacked as .zip is accepted only through the console upload (4.4).

If it fails#

Every refusal before Stopping the current version ends with The running version was not touched. Nothing needs undoing; fix the cause and run the same command again.

4.4 Update through the console#

The console can receive and verify a bundle for you. It never applies it: a container cannot replace the images it runs from. The last step is always one command on the server.

Route A: upload a bundle (air-gapped sites)#

1. Go to Settings > Platform, panel Platform updates, section Install from a file.
2. Press Choose bundle (.tar.gz or .zip) and select the bundle as delivered. (Up to 2 GB. For a larger file, copy it to the server and use 4.3.)
3. Four-eyes: the console shows Waiting for a second administrator to approve <file> (request #N). Keep the page open.
4. The approver opens Approvals > Waiting for me, reads the request and presses Approve.
5. Back in Settings > Platform, press Continue upload. If you had left the page, the approved request is listed there; choose the same file again.
6. The server checks the signature and every file. What you see: <version> is staged and verified, ready to apply. and the command ./lawliet update --staged with a copy button.
7. On the server, in the current installation directory:

   sudo ./lawliet update --staged

   This is the same update as 4.3. It checks the staged file's digest again before using it.

To drop a staged bundle without applying it, press Discard it.

Route B: the update feed#

Only when your vendor gave you an update feed address and UPDATE_FEED_URL is set in .env. Without it the panel says No update feed is configured, so this deployment makes no outbound requests and cannot see new versions.

1. Settings > Platform > Platform updates > Check for updates.
2. Download and verify for the version offered. Four-eyes: Stage a platform update.
3. Apply on the server with sudo ./lawliet update --staged.

If it fails#

4.5 What happens after the update#

The new version runs these steps once, after it has started and been verified. None of them can fail the update; each reports what it could not do and the command to do it.

4.6 Roll back#

Use this when the new version does not work for you. It returns to the version this one was updated from with the database as it was before the update.

Risk: everything written after the update (alerts, events, agents enrolled, users and settings changed) is not in the restored database. It is saved to a separate dump first and is not lost, but going back means working without it. Rollback of the rollback: run the update again later; the data written in between can be loaded only into the newer version (./lawliet restore of the saved dump from the newer directory).

Procedure#

1. Go to the new (current) directory:

   cd /opt/lawliet-<new>

2. Run:

   sudo ./lawliet rollback

3. Read the summary and answer the question:

     From                       <new>  (/opt/lawliet-<new>)
     Back to                    <old>  (/opt/lawliet-<old>)
     Database from              <pre-update dump>  (schema <rev>)
     Current data saved to      /opt/lawliet-<new>/backups/before-rollback-<time>/
   
     ! Anything written since the update (<time>) is not in the restored database:
     ...
     Roll back to <old>? (y/N)

   If settings in .env changed since the update (for example a licence installed), it names them. The old version runs with its own .env, so install such a licence again afterwards with sudo ./lawliet licence <key>.

4. It then: stops the application, saves the current database to backups/before-rollback-<time>/, stops the new version, restores the pre-update backup into the old version's database, starts the old version from its own directory, and verifies health, version and schema.

5. What you see:

     Rolled back to <old>.
   
     Running from               /opt/lawliet-<old>
     Newer data saved in        /opt/lawliet-<new>/backups/before-rollback-<time>
     This directory             /opt/lawliet-<new>  (kept; nothing was deleted)
   
     ! Run future commands from the previous directory: cd /opt/lawliet-<old>

6. Change to the old directory and check: cd /opt/lawliet-<old> && sudo ./lawliet status.

When there is no rollback record#

For an update finished by a version older than 1.5.12.s, or files moved by hand, name the pair yourself:

sudo ./lawliet rollback --to /opt/lawliet-<old> --backup /opt/lawliet-<old>/backups/lawliet-<time>.sql.gz

Use the dump the update took (the newest lawliet-<time>.sql.gz in the old directory's backups/ from just before the update).

If it fails#

Each check below happens before anything changes and ends with Nothing was changed.

Do not try to roll back by running an older bundle's update over a newer database: the older backend refuses a database migrated by a newer version ("Can't locate revision").

4.7 Delta updates#

A delta carries only what changed between the build you have installed and a newer one, for one CPU. It is much smaller than a full bundle. You apply it with the same command:

sudo ./lawliet update /opt/incoming/lawliet-delta-<old>-<build>-to-<new>-<build>-amd64.tar.gz

A delta is made for one exact build: the build id of your installation (shown by ./lawliet status). Before the backup, and while the current version keeps running, the update checks the delta with your installed backend image, with no network:

• the delta's signature, and that this installation is the build it was made for;
• that the files and image layers it reuses are unchanged since install;
• that the rebuilt bundle equals the new version's own signed manifest.

Then it rebuilds the complete new bundle beside the current directory and continues exactly as a full update (4.3).

What you see: the delta checks, each marked OK, then the same output as a full update. Roll back as after a full update (4.6).

Keep the images/ folder of the installation directory: a delta reuses the layers in it.

If it fails#

Every delta refusal ends with The running version was not touched.

4.8 After you are satisfied#

• Keep the old directory at least until the next update. Deleting it removes the ability to roll back.
• Docker keeps the old version's images. They are tagged with the old version and build id (lawliet-backend:<version>-<build id> and so on). Removing them is optional, and only after you no longer need the old directory.
• Copy the new directory's backups/ off the server as usual.