Finding the one session that matters in a week of noise

Searching logs for something suspicious does not work, and it is worth understanding why before you spend a year doing it.

You cannot search for what you have not thought of. A query is a guess about what the attacker did, and the good ones are the guesses you would not have made. So the productive technique is almost the opposite of searching: instead of asking what is bad, ask what is rare, and then ask why.

Count, then read the bottom

Frequency analysis is the whole method, and it is embarrassingly simple. Take a field. Count how often each value appears. Read the bottom of the list.

Nearly every host in your estate talks to the same handful of destinations. The one workstation that resolved a domain nobody else has ever asked for is at the bottom of that count, and it took one command to find. The same trick works on user agents, on parent-child process pairs, on which accounts logged into which machines, on JA3 hashes, on outbound ports. Sort ascending. Read the tail.

What that is, on a terminal

Nothing above needs a product. The entire technique is four commands in a pipe, and the last one is the one that matters: sort -n puts the rare values at the top of what you read.

$ awk '{print $3}' conns.log | sort | uniq -c | sort -n
      2 hx9q2-cdn.example
     94 api.partner.example
    168 ntp.internal.example
    288 smtp.internal.example
    611 metrics.internal.example
    842 repo.internal.example
   1204 backup.internal.example

Note what you did not have to know. Not what the attacker's tooling is called, not which domains are malicious, not what the current campaign looks like. You asked one question — how often? — and the answer sorted itself.

Try it

Below is a morning of proxy traffic from a small office. Count by destination and read the bottom of the list. Then count by workstation, and notice that it tells you nothing: most people visit most things, so that field is not where the rarity lives.

03:41:09 ws-019  q4t8v-push.example
09:01:59 ws-012  docs.corp.example
09:08:49 ws-011  auth.corp.example
09:17:00 ws-040  cdn.assets.example
09:21:50 ws-023  docs.corp.example
09:26:01 ws-007  cdn.assets.example
09:37:27 ws-031  auth.corp.example
09:55:25 ws-007  print.corp.example
10:46:28 ws-023  docs.corp.example
10:58:36 ws-040  docs.corp.example
11:12:11 ws-011  docs.corp.example
11:20:36 ws-028  auth.corp.example
11:32:20 ws-004  docs.corp.example
11:35:09 ws-019  time.pool.example
11:46:15 ws-012  auth.corp.example
11:54:54 ws-031  cdn.assets.example
12:06:46 ws-023  cdn.assets.example
12:12:50 ws-019  time.pool.example
12:19:24 ws-019  cdn.assets.example
12:39:45 ws-011  time.pool.example
12:49:19 ws-040  print.corp.example
13:17:48 ws-004  docs.corp.example
13:22:06 ws-004  docs.corp.example
13:28:36 ws-023  docs.corp.example
13:30:44 ws-023  docs.corp.example
13:33:41 ws-012  cdn.assets.example
13:37:58 ws-007  auth.corp.example
13:49:48 ws-011  docs.corp.example
13:55:09 ws-012  cdn.assets.example
14:05:51 ws-023  time.pool.example
14:33:07 ws-019  auth.corp.example
14:50:24 ws-028  time.pool.example
15:15:49 ws-011  auth.corp.example
15:16:24 ws-031  auth.corp.example
15:16:31 ws-004  docs.corp.example
15:18:19 ws-023  auth.corp.example
15:25:14 ws-023  docs.corp.example
15:37:12 ws-019  print.corp.example
15:48:26 ws-023  print.corp.example
16:01:40 ws-012  auth.corp.example
16:17:45 ws-012  time.pool.example
16:21:20 ws-012  cdn.assets.example
16:23:37 ws-031  docs.corp.example
16:43:55 ws-023  auth.corp.example
16:57:36 ws-012  cdn.assets.example

Two more techniques

The second technique is stacking two fields that are normally independent. Account and workstation, for example. Most people log into one or two machines. An account that appears on eleven, over a weekend, is not doing anything a filename-based rule would catch, and it is the clearest signature of credential reuse there is.

Third: baseline against time rather than against a threshold. "More than fifty failed logins" is a number someone guessed. "Ten times this account's own weekly average" adapts to a service account that legitimately fails often and to a human who never does.

What goes wrong

Two practical warnings, both learned the hard way.

Absence is evidence. A host that logged steadily for months and went quiet on Tuesday is telling you something, and no search for bad things will surface it — you have to be looking for the gap. Alert on silence from sources that should be talking.

And write down what you ruled out. An investigation that ends "nothing found" is nearly worthless to the next person, who will repeat every query you already ran. An investigation that ends "checked A, B and C, here is why each was normal, did not have data for D" is a real handover, and D is where the next person starts.