Welcome to InsightFinder Docs!
Categories
Rare Event Detection
Detecting novel and statistically infrequent behavior without predefined rules
The Core Idea
InsightFinder learns a behavioral baseline directly from the data itself. Patterns that are sparsely represented in that learned model are surfaced as rare events—including first-occurrence failure modes that rule-based monitoring cannot anticipate.
How It Works
Unsupervised Behavior Learning (UBL)
Rare event detection is built on InsightFinder’s Unsupervised Behavior Learning (UBL) approach. UBL learns normal behavior directly from observed data, so it does not require labeled training data, a catalog of known failures, or hand-maintained signatures.
Why this matters: Rule-based detection answers: “Does this match something we already know is bad?”
Rare event detection answers: “How much of the learned model was shaped by behavior like this?”
A first occurrence is rare by definition—which is exactly why the model can surface a previously unseen failure mode.
Detection Sequence
- Ingest + Parse: Logs are parsed into structured observations such as timestamp, instance, component, message content, and extracted fields.
- Train: UBL learns multi-dimensional behavior patterns and represents them on a topology-preserving neuron map.
- Group: Each observation maps to its best-matching neuron. Observations mapped to the same neuron form a behavior group.
- Classify: Groups at or below the configured Rare event cluster size are classified as rare and receive an anomaly score.
- Rank + Escalate: Events are ranked, capped by the number limit, and optionally promoted into incidents for RCA, webhooks, and automation.
Key Operational Parameters
The Most Important Tuning Knob: Rare event cluster size defines the boundary between normal and rare. A larger value widens the net and increases candidate volume; a smaller value limits detection to near-unique behavior. In the reference environment, the value is 2—behavior seen only once or twice qualifies.
Model Health: Incremental learning keeps the model current, but a major workload shift can make an old map less representative. Re-bootstrap when real problems are repeatedly missed or when rare-event volume spikes without an operational explanation.
Configuration
Path: Project Settings → Advanced → Sensitivity → Log Anomaly Detection
“Enable anomaly detection” is the prerequisite; the rare-event controls then define what qualifies and how much is reported.
| Setting | Reference Value | Purpose |
|---|---|---|
| Enable anomaly detection | Enabled | Prerequisite for model-based anomaly detection. |
| Enable all rare event detection | Enabled | Turns on rare-event detection across all learned patterns. |
| Rare event cluster size | 2 | Group-size threshold. Groups with ≤ 2 observations qualify as rare. |
| Rare event base score | 5 | Baseline anomaly score used for ranking and downstream severity. |
| Rare event number limit | 20 | Flood guard: maximum rare events reported per detection interval. |
| Rare anomaly type | Words only | Treats word/token content as meaningful while avoiding noise from routine numeric variation. |
Escalation + Detection Scope
Promote Rare Anomalies into the Incident Workflow
Path: Project Settings → Core functions → Incident escalations
Detection alone creates anomalies. Enable escalation when rare events should appear in Incidents and Root Cause views and become eligible for RCA, webhooks, and automated remediation.
Note: Click Update on the Incident escalations panel, then continue through the wizard. Changes apply to newly processed data; previously processed intervals are not retroactively re-evaluated.
What Rare Event Detection Can Surface
- First / Near-Unique: New or low-frequency exceptions, code paths, or third-party responses that rule-based rates may miss.
- Novel Tokens: Unseen status codes, regions, customer/asset identifiers, or other tokens inside familiar messages.
- Rare Integrated Alerts: Uncommon alert types or descriptions from sources such as Conviva when ingested as logs.
- Instance-Scoped Rarity: Behavior common across the fleet but new for a specific instance or component.
Results + Validation
How a Detected Rare Event Appears
When escalation is enabled, a rare event appears as a standard incident and participates in downstream analysis. The incident retains the source project, component/instance context, extracted fields, anomaly score/severity, learned incident pattern, and an RCA entry point.
Validation Checklist
| Check Item | What to Verify |
|---|---|
| Project | Filter to the project whose settings were changed. |
| Detection Source | Use Anomaly Detection to isolate model-detected incidents. |
| Anomaly Score | Confirm the base score produces the intended ordering and severity behavior. |
| Problem Category / Incident Source | Separate rare events from other anomaly classes while tuning. |
| RCA Context | Review contributing fields and surrounding signals rather than treating rarity alone as causality. |
Recommended Operating Posture: Start with the reference values, validate event volume and usefulness, then tune cluster size and number limit to the project’s log diversity. Escalate only when the organization wants rare events to enter the incident/RCA workflow.
Quick Reference & Research Basis
Rare Anomaly Types Defined
- All: Evaluates both word content and numerical values. Broadest coverage; typically requires the most tuning.
- Words Only: Evaluates tokens/word content while ignoring numerical values as standalone novelty. Best fit for typical log text (reference setting).
- Numerical Value Only: Evaluates numerical values while treating wording as fixed context. Best for metric-like messages.
From the Blog
Blogs 
See how InsightFinder helps your team deliver reliable services across every layer of the stack
Take InsightFinder AI for a no-obligation test drive. We’ll provide you with a detailed report on your outages to uncover what could have been prevented.
