Every analytics product eventually grows an "anomaly detection" checkbox, and most of them are useless within a week. Either the thresholds are so loose you never hear anything, or the detector fires on every quiet Saturday and you mute the channel. This is a deep dive into how I built Tracely's version, and specifically the decisions that exist to keep it from crying wolf.

The whole thing lives in two layers: a detector that computes anomalies from raw pageviews, and an alerting layer that decides which of them are worth a webhook. They have different jobs and different failure modes, so I'll take them in order.

Two signal families

The detector (AnomalyDetector) computes two kinds of signals over a per-site lookback window.

Z-score signals cover the site-wide metrics: daily pageviews, distinct visitors, and bounce rate. Each day in the window gets compared against a baseline, and days whose z-score exceeds a configurable threshold get flagged as spikes or drops. Anything at |z| ≥ 3 is marked high severity; the rest is medium.

Ratio signals cover dimensions: referrer source, content section, country, device, UTM campaign, and individual paths. For each of the top values in a dimension (top 12 by volume), every day's count is compared against that value's average across the other days of the window. A day gets flagged when it exceeds that average by a configured multiplier. This catches "this newsletter sent 6x its usual traffic on Thursday" — the kind of thing a site-wide z-score smears into nothing.

Both families are per-site configurable: the lookback window, the z threshold, the ratio multiplier, the minimum views for path spikes, and each signal can be toggled on or off individually. If a site never runs campaigns, you can turn utm_campaign_spike off and never think about it again.

Weekday-aware baselines

The single biggest source of false positives in naive anomaly detection is weekly seasonality. A B2B site does half its weekday traffic on Saturdays, every Saturday, forever. If your baseline is "the mean of the last 28 days," every single weekend produces two "traffic drop" alerts. That's the fastest possible route to the mute button.

So the z-score baseline is weekday-aware. When scoring a given day, the detector first looks for other days in the window that fall on the same weekday, excluding the day being scored. If there are at least three of those peers, the baseline mean and standard deviation come from them: a Saturday is judged against other Saturdays. Only when there aren't enough same-weekday samples does it fall back to the whole-window mean and deviation.

There's one subtle failure mode inside that: if the same-weekday peers happen to be identical (say, three Saturdays with exactly 412 views each), their standard deviation is zero, which would make every deviation an infinite z-score — or, with the guard inverted, hide every real outlier. In that case the detector keeps the weekday mean but borrows the window-wide spread, so a genuinely unusual Saturday still stands out without the math blowing up.

Two more guards worth mentioning:

Today is a partial day. If it's 10am, today's pageview count is obviously "low" compared to complete days. Volume drops for the current day are suppressed until the day is complete. Rate metrics like bounce rate are fine to evaluate mid-day, so those still fire.
Every anomaly carries its series. Each flagged day ships with the full daily series plus the baseline band (mean, low, high per day), so the dashboard can draw a sparkline showing exactly where the day sat relative to its band. An anomaly you can't see is an anomaly you don't trust.

Ratio spikes and noise floors

The ratio signals have their own false-positive trap: small numbers. A referrer that averages 0.4 views per day and gets 3 views today is technically 7.5x its baseline, and nobody cares. Non-path ratio signals therefore ignore any day below 10 views outright, and path spikes have their own per-site minimum. On top of that, each signal type is capped at 8 flagged results per run, sorted by how extreme the ratio is — so a chaotic week surfaces its worst offenders instead of burying you in forty near-identical rows.

Root-cause attribution

A "traffic spike" flag on its own just tells you to go dig. So for traffic and visitor anomalies, the detector does the first round of digging itself: for the anomalous day, it asks which referrer source, country, and device moved the most compared with that value's typical daily volume, and attaches the top movers (up to three) to the anomaly as an attribution list. A spike gets its biggest positive movers; a drop gets its biggest negative ones. Deltas under five views are ignored, and attribution runs for at most the three most recent anomalous dates to keep query cost bounded.

In practice this turns "Traffic spike: 4,200 pageviews (85% above typical Tuesday)" into "…and it's mostly one source, from one country, on mobile." That's usually enough to know whether you're looking at a press hit or a crawler.

Suppressions

Some anomalies are real but permanently uninteresting — a partner site that spikes you every month, a country that swings on VPN traffic. Suppression rules are applied last, after everything is computed, so a "don't flag this again" choice matched on the anomaly type and dimension value holds across every recompute rather than being a one-time dismissal.

The alerting layer

Detection and notification are deliberately separate. The scheduler runs alerts:evaluate hourly (alongside reports:send; the full table is in the project's scheduler docs), and each configured alert webhook gets evaluated by AlertEvaluator. A few kinds of rules exist:

Simple triggers. A spike fires when the last hour's pageviews reach your threshold and are at least 25% above the previous hour — the second condition stops a steady-state busy site from firing every hour. A drop fires when the last hour falls below half the previous hour, but only if the previous hour had enough volume to make that meaningful. A milestone fires when today's pageviews cross a number.

Anomaly alerts. This is where the two layers meet: an anomaly trigger fires when the background anomaly scan has persisted at least N new, still-open anomalies since the last delivery, bounded to the last 24 hours so a long-idle webhook doesn't replay old history when it wakes up. High-severity anomalies sort first, and the message includes the top three with their dates and z-scores.

Query-based rules. For anything more specific there's a criteria evaluator: count pageviews or a named custom event over a window (5–120 minutes, or 1–168 hours), compare against a threshold, optionally require a ≥25% lift versus the prior window, and narrow with up to eight filters (eq, contains, starts_with) on fields like path, referrer_source, utm_campaign, or country. "Alert me when signups from the pricing page pass 50 in an hour" is one rule, not a feature request.

Content decay is a specialized case I care about a lot, since Tracely treats content workflows as first-class. The evaluator compares each path's last-7-day pageviews against the average of the three prior 7-day windows, and fires on the worst offender that dropped past your percentage — with a minimum trailing-views floor (default 200 per week) so sparse pages don't generate noise. Traffic decay is slow and silent; it's precisely the thing you don't notice until a quarter later.

Delivery, cooldowns, and one security note

Every webhook has a six-hour cooldown after it fires. If a condition persists, you get reminded, not spammed. Payloads are shaped per destination — Discord gets content, Slack gets text, and a generic type gets a structured JSON body with the site's domain, ulid, and name so you can route it in your own tooling.

One detail I'd flag for anyone building outbound webhooks: Tracely refuses to POST to any URL that resolves to a private or internal IP. A user-supplied webhook URL is a textbook SSRF vector into your own infrastructure, and the guard runs before every delivery, not just at save time.

What I'm still tuning

The z threshold and ratio multiplier defaults are educated guesses that I expect to revise as more traffic shapes flow through this. The honest summary of the design is: seasonality-aware baselines to kill the predictable false positives, noise floors and caps to kill the small-number ones, attribution so a real alert arrives with its own first answer, and cooldowns plus suppressions so the system respects your attention. An anomaly detector's real output isn't anomalies — it's your continued willingness to read them.

Detecting anomalies without crying wolf