Edge ingest: terminating analytics traffic at the CDN

This week I'm building something I've wanted since the first month of the project: an edge ingest tier. It's landing in pieces right now — the Worker is written and deployed, the key synchronization works, and I'm still finding rough edges as real traffic hits it — so consider this a snapshot of work in motion rather than a finished-feature announcement.

The problem

Tracely's ingest path is already queue-backed: a tracker beacon hits POST /api/i/p (or the batch endpoint), the request is validated, and the actual database write happens in a queued job so the HTTP worker returns fast. That protects the dashboard from ingest load, but every beacon still terminates on origin PHP. Octane makes that cheap per request, yet "cheap" multiplied by a viral article's worth of pageviews is still a lot of PHP invocations doing little more than validating JSON and enqueueing work.

The observation: for the hot path, the origin adds almost no value. Validating that a site exists and queueing a payload doesn't need PHP, a framework, or even my server. It needs a key-value lookup and a durable queue — which is exactly what Cloudflare Workers, Workers KV, and Cloudflare Queues provide, running in the same data centers that already sit between visitors and most sites.

The shape of it

Browser tracker  →  Cloudflare Worker (/api/i/*, sid in JSON)
                        ↓ KV validate, 202 immediately
                   Cloudflare Queue
                        ↓ Bearer TRACELY_INGEST_EDGE_TOKEN
                   Tracely backend  POST /api/i/edge  →  ingest queue

The contract for customer sites doesn't change at all. The same sid (tracking ID), the same /api/i/* paths, the same payloads. The only thing that moves is the tracker's apiUrl — instead of pointing at the main app, it points at an edge hostname served by the Worker. There's even a config key so the hosted tracker script does this automatically: when TRACELY_INGEST_EDGE_TRACKER_URL is set (and edge ingest is enabled), the tracker script builder embeds <tracker_url>/api as the apiUrl, so beacon traffic terminates at the Worker without anyone editing a snippet.

Two different authentications guard the two hops. Browser-to-Worker is validated by looking up the sid in Workers KV — no secrets in the browser, same as origin ingest. Worker-to-backend uses a shared secret, TRACELY_INGEST_EDGE_TOKEN, sent as a Bearer header. That token is generated once (openssl rand -hex 32) and stored in two places: the Tracely .env and a Wrangler secret.

Inside the Worker

The Worker itself is small and deliberately dumb. It keeps a set of known ingest paths — /api/i/p, /api/i/b, /api/i/e, /api/i/m, /api/i/h, plus the legacy aliases like /api/pageview — parses the JSON body, extracts the sid, and looks it up in the INGEST_SITES KV namespace. The KV record carries site_id, organization_id, billing_ok, and revoked_at; an unknown or revoked sid gets a 404, and a suspended workspace gets a 402 so the tracker fails fast instead of piling payloads onto a dead account.

If the record checks out, the Worker enqueues an envelope — site ID, path, method, body, plus the client IP, user agent, and country header that Cloudflare already knows — and returns 202 with {"status":"ok"}. The enqueue happens in ctx.waitUntil, so the response goes out without waiting on it. One nice consequence of holding the payload at the edge: there's also a GET beacon path (/api/i/a) that responds with a 1×1 GIF, which covers contexts where a JSON POST isn't possible.

Anything under /api/ that the Worker doesn't recognize is passed straight through to the backend rather than 404ing — future endpoints shouldn't require a Worker deploy to exist.

The queue consumer runs in the same Worker script. It pulls batches and forwards each envelope to the backend:

response = await fetch(`${env.TRACELY_BACKEND_URL}/api/i/edge`, {
  method: 'POST',
  headers: {
    ...JSON_HEADERS,
    Authorization: `Bearer ${env.TRACELY_INGEST_EDGE_TOKEN}`,
    'X-Tracely-Response-Id': envelope.response_id,
    'X-Tracely-Client-Ip': envelope.client_ip || '',
    ...
  },
  ...
});

Failures call message.retry() on that one message rather than throwing — throwing would re-drive the whole batch, and I only want to retry what actually failed. After max_retries (five, per wrangler.toml), the message parks in a dead-letter queue (tracely-ingest-dlq) instead of being dropped. The consumer's max_concurrency is capped at 25, which doubles as backpressure: if the backend slows down, the queue absorbs the burst instead of the origin.

The backend side

On the Tracely side there's a single internal route, POST /api/i/edge. It's registered without the public ingest throttle — the comment in routes/web.php says it plainly: "No public throttle; bearer token only." That's intentional. Public rate limits exist to protect against abusive browsers; the forward route only ever hears from my own Worker, authenticated by the shared token, and its natural rate limit is the queue consumer's concurrency. The route returns 503 until TRACELY_INGEST_EDGE_TOKEN is configured, so a half-deployed setup fails loudly rather than accepting unauthenticated traffic.

From there the payload rejoins the normal pipeline — same validation, same queued persistence jobs — so edge-ingested events are indistinguishable from origin-ingested ones downstream.

Configuration all lives under ingest_edge in config/tracely.php: enabled (TRACELY_INGEST_EDGE_ENABLED), token, backend_url, the tracker_url mentioned above, and three Cloudflare credentials (CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_KV_NAMESPACE_ID, CLOUDFLARE_API_TOKEN) used for key sync.

Keeping KV in sync

The Worker can only validate a sid if it's in KV, which makes key synchronization the load-bearing piece. IngestEdgeKeySynchronizer builds a small JSON entry per site — site_id, organization_id, billing_ok (derived from whether the organization is suspended), revoked_at — and writes it to Cloudflare's KV REST API keyed by tracking ID.

Two mechanisms keep it current. php artisan tracely:sync-ingest-edge-keys does a full sweep, and it's scheduled every five minutes when edge ingest is enabled. On top of that, sites re-sync individually when they're saved or deleted, so the common cases — new site, changed domain, deleted site — propagate immediately rather than waiting for the sweep. The sweep is the safety net for anything the hooks miss, and for bootstrapping a fresh KV namespace.

Putting billing_ok in the KV entry deserves a note, because it's the one place edge validation is more than existence-checking. When an organization is suspended, the next sync flips that flag and the Worker starts answering 402 at the edge — the payloads never reach my queue, my backend, or my storage. In the origin-only world, a suspended account's beacons still consumed PHP workers just to be rejected. At the edge, policy enforcement costs one KV read.

Verifying a deployment

Since this is an operator feature, the runbook ends with a smoke test rather than a screenshot:

curl -sS -o /dev/null -w "%{http_code}\n" \
  -X POST "https://ingest.example.com/api/i/p" \
  -H "Content-Type: application/json" \
  -d '{"sid":"YOUR_TRACKING_ID","url":"https://your-site.test/"}'

Expect 202. A 404 means the sid isn't in KV yet (run the sync command); a 503 from the backend means the edge token isn't set on the Tracely side. Each failure mode has exactly one cause, which is the property I most want in infrastructure I'll be debugging at odd hours.

Rough edges, honestly

A few things I've already hit, and a few I'm still uneasy about:

CORS bit me first. Tracker beacons are cross-origin fetch()es with a JSON content type, so every browser preflights with OPTIONS and enforces CORS on the response. The first version of the Worker didn't handle that at all — preflights got 405, and even accepted POSTs were opaque failures to the page. The Worker now answers OPTIONS with 204 and proper CORS headers on everything, but it was a humbling reminder that "just forward the request" is never the whole job.

202 means accepted, not persisted. The Worker acknowledges before the payload has traveled through the queue to my database. That's the entire point — it's what makes the edge fast — but it changes failure semantics. If the backend is down for an extended period, retries and then the dead-letter queue hold the data; replaying the DLQ is currently a manual affair, and I want better tooling there.

Eventual consistency on new sites. A brand-new site whose KV write failed (or raced) will get 404s at the edge until the five-minute sweep catches it. The save-hook makes this rare, but "rare" isn't "never," and the troubleshooting table in my runbook exists because I've needed it.

No edge-side rate limiting yet. Origin ingest has per-IP-and-site throttles; the Worker currently has none of its own, leaning on Cloudflare's platform protections and the consumer's concurrency cap. That's probably fine and definitely unproven.

This tier is strictly optional — self-hosted deployments at normal volume don't need it, and nothing about the default path changes. But if analytics infrastructure is going to be boring at scale, the boring has to be engineered somewhere, and this week that somewhere is a hundred-odd lines of JavaScript running at the edge. More notes as it hardens — the feed is here.