30. Logs: search, dashboards and sources

Lawliet collects logs from its agents, from syslog senders (firewalls, switches, servers) and from HTTP shippers, stores them, and lets you search them with a pipe query language. Detection rules watch the same store and raise alerts (chapter 31).

The Logs area has eight tabs: Search, Dashboards, Alerts, Rules, Sigma, Entities, ATT&CK, Sources. This chapter covers Search, Dashboards and Sources.

Log collection is a licensed module. Without it the Logs pages say "Log collection is not included in this licence"; everything else keeps working.

Permissions:

30.1 The Search tab#

Path: Logs (the Search tab).

• Tiles: Events (24h), Sources receiving, Silent sources, Dropped on ingest.
• Query box, the Time range buttons (15m, 1h, 24h, 7d, 30d), Search and Refresh.
• Severity (any, low+, medium+, high+, critical) and Clear filters.
• From / To for an exact range ("Both ends are needed before this replaces the preset above"); Back to presets returns to the buttons.
• Recent (your last queries), Saved (saved searches) and Try: (starter queries: Failed logins, Successful logins, Privilege escalation, Accounts under attack, Logins over time, Loudest hosts).
• Break down by: a field to count the results by.
• Results with Newer / Older paging.
• Under the query: Make this a rule, Save search, Add to dashboard.

The query language in brief#

The full reference is SIEM query language.

After the filter, add stages with |:

action:logon_failed | stats count() as attempts, dc(src_ip) as sources by user | sort -attempts | head 10
category:authentication | timechart span=1h count() by outcome
* | top 10 host

Stages: stats, where, top, timechart, table, sort, head, dedup.

Procedure: investigate failed logins for one account#

1. Open Logs. Press Try: > Failed logins, or type action:logon_failed.
2. Choose 24h and press Search.
3. Narrow: add AND user:jsmith to the query, then Search.
4. Choose src_ip in Break down by to see which addresses tried.
5. To keep it: Save search, give a Name, tick Shared with everyone if colleagues should see it, press Save.
6. To be alerted next time: Make this a rule opens the rule editor with this query (chapter 31).

If it fails: "The pipeline returned nothing" - widen the time range, or check the filter before the first |. A query the server cannot read is answered with a message under the query box; correct the query and search again.

30.2 The Dashboards tab#

Path: Logs > Dashboards. "Searches kept on screen, drawn as numbers, tables and charts."

• The list of dashboards; built-in dashboards are read-only ("copy one first to add to it").
• Range for every panel and Refresh dashboard at the top of an open dashboard.

Permission for everything in this section: Search logs. Four-eyes: no. Nothing here changes a host or the collected events; a dashboard only stores searches.

Procedure: build a dashboard#

1. Open Logs > Dashboards and press New dashboard.
2. Fill Name and Description. Leave Shared with everyone ticked for colleagues to see it, or untick it to keep it to yourself.
3. Press Add panel. For the panel fill Title, Query, Shown as (number, table, bar, line or pie), Range (minutes), Width (of 4) and Height. Repeat for more panels.
4. Order the panels with the up and down arrows (Move panel N up / Move panel N down); the bin (Remove panel N) takes a panel out.
5. Press Save dashboard (disabled while Name is empty). Cancel discards the draft.

What you see: "Saved <name>." and the dashboard in the list, drawn with its panels.

Faster: run a search on the Search tab and press Add to dashboard. The Dashboards tab opens with "Add <query> as a panel to:"; press the name of one of your dashboards, or A new dashboard, then finish from step 2 or 3 and press Save dashboard.

Procedure: change, copy or delete a dashboard#

1. Open Logs > Dashboards and click the dashboard in the list.
2. Choose Range for every panel to view all panels over one range ("Each panel's own" uses each panel's setting); press Refresh dashboard to run them again.
3. To change it, press Edit, change the fields or panels as in "Build a dashboard", and press Save dashboard.
4. To make a variant, press Copy, change the copy, and press Save dashboard.
5. To delete it, press the bin (Delete dashboard). The browser asks "Delete the dashboard "<name>"? Its panels' saved queries go with it." Press OK. You see "Deleted <name>."

Built-in dashboards are read-only: they show Copy to edit instead of Edit, Copy and the bin. Press Copy to edit and work on the copy.

If it fails: a red line names the reason, for example "The dashboard could not be deleted." A dashboard someone else made and did not share is not shown to you.

Procedure: delete a saved search#

1. Open Logs (Search tab). Saved searches are listed under Saved.
2. Press the x next to the saved search (Delete saved search <name>). It is deleted at once, without a question.

If it fails: "That saved search could not be deleted." A saved search cannot be renamed in the console: save the query again under the new name and delete the old one.

30.3 What the agents send#

Installed agents ship their host logs on their own. Each agent creates its own source on first delivery, so nothing needs to be added for them. They appear in the Sources tab.

30.4 Silent and dropped#

• Silent sources: a source that has not reported in over 24 hours. "Searches over that period will come back empty, which is not the same as nothing having happened." Check the sender.
• Dropped on ingest: events the platform received but could not store, because the ingest queue was full or a database write failed. The Sources tab shows them under "Rejected on ingest - these events were lost". A rising number means the server is not keeping up: check its disk and load with ./lawliet doctor (chapter 7).

30.5 Sources#

Path: Logs > Sources. Columns: Source, Transport, Health, Events, Last event, Retention.

Procedure: add a syslog source#

Permission: Manage log sources.

1. Open Logs > Sources and press Add source.
2. Fill Add a log source:

3. Press Create.
4. On the sender, point syslog at the Lawliet server address, port 5514, same protocol.
5. Open the server firewall for 5514 only from those senders (Security hardening of the server).

What you see: after the first message arrives, Events and Last event fill in.

If nothing arrives: check the sender's target and protocol, the server firewall, and that the sender's address is in Accept from (lines from other addresses are refused and not stored).

Procedure: add an HTTP shipper#

1. Add source, Transport: HTTP shipper, a Name, optional Accept from, Create.
2. The Ingest token dialog shows the token once: "This is the only time this token is shown. It can be replaced, not recovered." Press Copy token, store it, then press I have stored it.
3. Configure the shipper to POST batches of raw lines to https://<server>/api/v1/log-store/ingest with the header X-Ingest-Token: <token> and a JSON body {"lines": ["<line 1>", "<line 2>"]} (up to 5000 lines per batch).

The token authenticates this source only; it cannot read, search or list.

Procedure: delete a source#

Permission: Manage log sources. Four-eyes: no. Risk: its collected events are deleted with it. Rollback: none - search and export what you need first.

1. Open Logs > Sources and press the bin on the source row (Delete <name>).
2. The browser asks "Delete <name>? Its collected events are deleted with it - N received so far." Press OK.
3. What you see: the row disappears. If it fails: "<name> could not be deleted."

A source cannot be edited in the console in this release, and an HTTP shipper's token cannot be replaced there. To change the transport, the accepted addresses, the retention or the token, add a new source with the right settings, move the sender to it, and delete the old source once you no longer need its events. Sources that agents created for themselves are recreated by the agent on its next delivery if you delete them.

30.6 If it fails#