Observability · 2026-09-04 · 5 min read

The drop rule that ate the error logs

A client runs FlowFuse — a platform for managing Node-RED instances — on Kubernetes, across AWS and on-premise clusters. The flows running inside those instances are production logic: they move data, talk to databases, feed other systems. The request was reasonable and small: alert us when flows throw errors.

Between that sentence and a working alert there turned out to be three layers of problems, each hiding the next. The alert itself was the least interesting part.

Layer 1: the logs didn't exist

You can't alert on logs that never leave the container. Two enablers had to land first:

• logPassthrough in the FlowFuse Helm values, so each Node-RED instance's flow logs are written through to the container's stdout instead of staying internal to the platform.
• A relabel rule in Alloy deriving an nr_instance label from the pod's name label (guarded by the platform's nodered=true label), so Loki can tell instances apart. Without it, every instance's logs land in one undifferentiated stream and a per-flow alert can't say which flow.

Pipeline complete: instance → stdout → Alloy → Loki, queryable per instance. Or so it seemed.

Layer 2: the logs were being eaten

The Alloy config carried a drop stage whose intent was to discard the platform's noisy access logs:

stage.drop {
  expression = "INFO"
}

The drop stage takes an RE2 regular expression — and, as we verified the hard way, it matches case-sensitively and unanchored: a partial match anywhere in the line drops the whole line. So this rule doesn't mean "drop INFO-level logs". It means "drop any line that contains the substring INFO anywhere". Two verified consequences:

• On the AWS clusters it discarded the platform's entire log output (every line carries "level":"INFO"), so the platform's namespace simply didn't exist in Loki. Nobody had noticed, because who alerts on the absence of logs?
• Worse: any flow log whose message happened to contain "INFO" was silently dropped too — including error-level lines that mentioned, say, INFORMATION_SCHEMA. A query error against a database's information schema is exactly the kind of line you're building alerts for, and the pipeline was discarding it before Loki ever saw it.

The fix is one line — anchor the expression to the JSON key it always meant to match:

stage.drop {
  expression = "\"level\":\"INFO\""
}

The lesson generalizes: a drop stage is destructive filtering. Data it discards is gone with no trace, no metric, no error. Any unanchored substring match in a drop rule is a silent data-loss bug waiting for the right log line — audit what your expression actually matches, not what you meant it to match.

Layer 3: the restart that GitOps can't see

With the config merged and synced, the instances still emitted nothing. The passthrough flag gets baked into each Node-RED instance's Deployment as an environment variable — and that Deployment is not managed by ArgoCD. It's created dynamically by the platform's Kubernetes driver. A kubectl rollout restart doesn't help (same baked env), and GitOps has no lever on the resource at all: the only real path is Suspend → Start from the platform's own panel, which regenerates the Deployment with the new flag.

That's a boundary worth knowing in any platform-on-Kubernetes setup: resources created by the platform's own orchestrator live outside your GitOps net. Your repo can be green while the actual workloads run stale config.

Method note for the config change itself: everything was validated live on the test cluster first, editing the running resources by hand — and then the Helm values were written to reproduce the validated ConfigMap byte for byte, so the ArgoCD sync was a verified no-op rather than a leap of faith. Rollback for the whole change: one git revert.

Then: measure before you threshold

With logs finally flowing and complete, the temptation is to write the alert rule. First we measured a 24-hour baseline — and it reframed the whole request:

• Production: 313 flow errors in 24 hours, and every single one was the same error — ECONNRESET, a dropped connection from the flows' MySQL client node. Not query bugs: infrastructure.
• The pattern: a chronic base of 1–14 errors per hour (average ~7), plus one spike of 176 errors in a single hour, at 5 AM — a real incident that had passed completely unnoticed.

That shape dictates the alert design. Alerting per event means ~7 notifications an hour of pure background noise — the alert gets muted within a week, and mute is where alerts go to die. A rate threshold (on the order of 30 errors in 15 minutes) cleanly separates the 5 AM-style incident from the chronic hum.

And it surfaces the honest conversation to have with the client: there's a chronic connectivity problem to the database that nobody is attending. An alert on top of an unfixed chronic condition is just a scheduled reminder of the problem. The recommendation went out as two parallel tracks — rate-based alert now, root-cause investigation of the connection drops alongside — rather than pretending the alert alone was the deliverable. (With one more honesty note: the baseline numbers are a floor, measured while the drop rule was still eating lines. Re-measure after the fix.)

Small print for the queries

Two parsing traps found on the way, worth their weight in avoided debugging:

• Don't trust the ts field. The platform's launcher builds it as logEntry.ts + ('' + count).padStart(4,'0') — a number plus a string, which in JavaScript is concatenation, not addition. The published timestamp has four extra digits and arrives as a quoted JSON string. Use Loki's ingest timestamp instead.
• msg isn't always a string. When a flow node passes an error object, msg arrives as a JSON object. Cast before doing string operations, or the query breaks precisely on the lines you care about — the errors.

Takeaways

• 1. "Add an alert" is an observability audit in disguise. Verify the logs exist, arrive complete, and are attributable before writing rules on top of them.
• 2. Unanchored drop rules are silent data loss. A destructive filter deserves the same review rigor as a DELETE without a WHERE.
• 3. Know where your GitOps net ends. Platform-created resources can run stale config while the repo shows green.
• 4. Baseline before threshold. Twenty-four hours of measurement turned "alert on errors" into "rate alert + a chronic problem your client didn't know it had".
• 5. An alert on a known-chronic condition is a snooze button. Pair it with the root-cause track, or mute fatigue will win.