60. Agent health: why a host is offline or slow
This guide describes Lawliet 1.6.3.b (the current release, a beta: earlier releases are beta. The first stable release will be 2.0.0). Screens, commands and procedures may change in later versions; the usage guide is updated with each release. Check the version your server runs under Settings > Platform or with ./lawliet version, and see the Releases page for what changed.
Environment > Health answers one question for every host: why is it offline, late or slow? Each host is listed with its reasons in plain language, the most unhealthy first - for example:
heartbeat gaps of 4-6 min every ~10 min since 14:02: while the main loop was busy (main_loop_age 290 s)
Every time on the page is in your own time zone.
| Task | Permission | Four-eyes |
|---|---|---|
| See the fleet's health and one host's health | View agents | - |
| See the agent's own log lines and Diagnostics errors under Last errors | View agents and Manage settings | - |
The page only reads what the server already holds; it changes nothing on a host.
60.1 Open the page#
- Environment > Health (the tab next to Hosts and Agent versions): the whole fleet.
- Environment > Hosts, click a host, then Health: why offline or slow in its panel: the health page with that host open.
The list refreshes every 30 seconds. The buttons above it filter by state: offline, unhealthy, degraded, healthy (with the count of each).
| State | Meaning |
|---|---|
| offline | No heartbeat for 5 minutes or more (or the host never reported) |
| unhealthy | One serious reason (a crash loop, repeating heartbeat gaps, a stuck main loop, a clock 5 minutes off, tampering) or several smaller ones |
| degraded | Something to look at, nothing serious: a few gaps, a failed command, plain HTTP |
| healthy | Heartbeats on time, nothing lost, nothing failing |
The order is by a score that adds up every reason; the score is shown on each row.
60.2 What the reasons mean#
| Reason | What it means | What to do |
|---|---|---|
| offline: no heartbeat for ... | Nothing has reached the server since the time shown | Check the host is up, the Lawliet agent service runs, and the host reaches the server (proxy, firewall) |
| heartbeat late | Over the gap threshold but under 5 minutes | Usually passes; repeated, it shows as gaps |
| heartbeat gaps of X-Y min every ~N min since T: ... | Gaps that repeat at a regular interval, with the cause the heartbeats themselves report | See the cause after the colon |
| N heartbeat gaps in the last 24 h, longest ... | Gaps that do not repeat | See the cause after the colon |
| while the main loop was busy (cause) / main loop busy (its own reason) | The agent's main loop (log shipping, DLP, policy refresh, watchdog) had not completed a pass for that long. As a cause it is what was seen during the gaps, not proof it held the heartbeats: since agents 2.1.14.s the heartbeat runs on its own thread | Look at what the host is scanning (DLP paths, log volume); collect Diagnostics |
| the agent restarted / the host restarted (cause) | The gap ended with a new agent process, or a new boot | See Restarts in the host panel |
| crash loop: the agent restarted N times within an hour | The agent process keeps stopping. Counted from the start time agents 1.5.16.s and later report | Collect Diagnostics: its log says why it stops |
| data lost: N DLP finding(s) dropped / N command result(s) dropped | The agent's queue was full, or results were never acknowledged. Counted since the agent started | Find why the server was slow or unreachable; re-run the DLP scan |
| N DLP findings waiting to be sent | A backlog of 1000 or more | Usually the server or network is slow |
| the agent uses X % CPU and Y MB, over its ... budget | The agent's own cost, reported with endpoint telemetry (chapter 55) | Lower the telemetry profile, or accept it |
| the upgrade to V failed / was refused / was rolled back / has been 'state' since T | The latest agent upgrade (chapter 4) | Read the detail; retry from Agent versions |
| clock skew | The host clock is a minute or more away from the server's, with what that breaks | Synchronise the host with NTP |
| N command(s) failed or timed out in the last 24 h | The last one is named with its error | Open the host panel's Last errors |
| talks to the server over plain HTTP | The agent token and its data cross the network unencrypted | Move the agent to HTTPS (chapter 6) |
| self-protection reports ... | The agent's files or service were tampered with (chapter 16) | Investigate the host |
A gap is a time between two heartbeats longer than 2.5 heartbeat intervals (at least the interval plus 45 seconds): 150 seconds for the default 60-second interval. Agents 1.5.16.s and later report their interval; for older agents 60 seconds is assumed.
60.3 Procedure: find out why one host is offline or slow#
- Open Environment > Health and click the host (or open it from Environment > Hosts).
- Read Why: the reasons, most serious first.
- Read the sections below it:
| Section | What it shows |
|---|---|
| Heartbeats, last 24 h | One bar for the last 24 hours: green on time, amber a gap, red offline, grey not watched yet (before Lawliet 1.5.16.s started watching this host - no gap is ever claimed for that time). Under it, each gap with its times, the main loop age and whether the agent or the host restarted. The percentage counts watched time only. |
| CPU and memory | The agent's own CPU and memory against its telemetry budget (when endpoint telemetry is on), and the host's CPU and memory from the last heartbeat |
| Queues | Main loop pass age, DLP findings waiting / dropped, command results dropped, FIM events dropped. An agent older than 2.1.14.s does not report its queues: the page says so |
| Last errors | Failed or timed-out commands (24 h); with Manage settings also the errors and warning/error log lines of the last Diagnostics collection (they can name paths and accounts on the host, as the support archive does) |
| Restarts, upgrade, clock, transport | Agent restarts (24 h, and whether that is a crash loop), host restarts, the latest upgrade, the clock offset explained, HTTPS or plain HTTP |
60.4 What is recorded, and for how long#
The server records a heartbeat gap, an agent restart or a host restart only when one happens - a healthy host writes nothing. Records are kept 7 days (at most 500 per host). Removing an agent removes its records; clear-demo removes the sandbox's.
Restarts are counted only for agents that report their start time (1.5.16.s and later); for an older agent the page says "not reported by this agent". A restart is counted whenever the reported start time changes, also after the host clock was set back.
In the sandbox, the health page shows a host with repeating gaps (busy main loop), a host in a crash loop and a host that restarted.