Welcome to InsightFinder AI Reliability Docs!
Categories
AI Observability
Product naming update
The source document refers to the model testing and comparison environment as “LLM Labs.” Per current product naming, this guide uses “AI Eval.” Screenshots used in this document have been updated where the former navigation label was visible; all other screenshot content is preserved.
This guide explains the purpose, source-supported capabilities, signals, outputs, and relationship of each InsightFinder AI Observability component. It intentionally does not assert implementation details that are not defined in the supplied source.
Platform model: closed-loop AI reliability
InsightFinder AI Observability is positioned as a lifecycle platform for evaluating, deploying, observing, governing, and continuously improving AI agents, LLMs, and ML models. The central design principle is a closed feedback loop: production behavior and user outcomes are captured, analyzed, and reused to improve prompts, models, and operating controls.
| OBSERVE
Trace agents & models |
→ | EVALUATE
Quality & safety |
→ | OPTIMIZE
Prompt & model |
→ | GOVERN
Route & control |
→ | IMPROVE
Feedback & tuning |
Component map
Component |
Primary role | Source-supported signals / controls | Primary outcome |
|---|---|---|---|
| Multi-Agent Tracing | Agent workflow visibility | Agents, tools, handoffs, performance anomalies, token use, failed evaluations | Explain where and why agent workflows fail |
| LLM Prompt Comparison | Prompt/model optimization | Failures, execution time, token cost, success rates | Select better prompt/model combinations |
| Model Fine-Tuning | Production-driven improvement | Failed prompts, generated training data, RL jobs | Adapt foundation models to domain behavior |
| AI Eval | Model evaluation & selection | Foundational/open-source models, hallucination, safety, bias, quality dimensions | Choose fit-for-purpose models with evidence |
| AI Gateway | Production routing & governance | Model routing, availability, rate limits, trust/safety screening | Balance reliability, control, and cost in production |
| LLM Observability | LLM production monitoring | Input/output tokens, latency, changes, failed evals, traces | Detect and diagnose quality/cost/performance issues |
| Data Integrity Insights | Data quality assurance | Missing data, type mismatches, outliers, custom conditions | Prevent bad data from degrading downstream AI |
| ML Observability | ML risk monitoring | Data/concept drift, feature bias, SHAP, RCA, remediation | Maintain model quality and trust over time |
Multi-Agent Tracing
Multi-Agent Tracing provides end-to-end visibility into agentic workflows. It is designed for systems where a user request passes through multiple agents, LLM calls, tools, and handoffs before a final outcome is produced.
Figure 1. Multi-Agent Tracing with anomaly details and failed-evaluation context. The navigation label has been updated to AI Eval.
What it captures
- The sequence of agents, tools, and handoffs involved in a workflow.
- Performance anomalies within the trace.
- Token consumption associated with the workflow.
- Failed evaluations and their context.
- Common reasons evaluations fail, allowing reliability issues to be prioritized.
How to use it technically
| Task | Technical use |
|---|---|
| Trace reconstruction | Follow a request across the complete agent/tool path rather than inspecting isolated model calls. |
| Failure localization | Identify which agent, LLM step, or handoff is associated with failed evaluations or anomalous behavior. |
| Performance analysis | Compare trace duration and anomalous duration to understand where execution time is being introduced. |
| Token analysis | Use token consumption in context with workflow behavior rather than as a standalone cost metric. |
| Evaluation analysis | Review failed-evaluation categories and associated trace steps to narrow the reliability problem. |
| Technical boundary
The supplied source describes distributed tracing and the signals surfaced in the trace, but it does not specify the trace-ingestion protocol, exact span schema, sampling behavior, or storage implementation. |
LLM Prompt Comparison
Prompt Comparison is used to test the same prompt behavior across multiple underlying LLMs and prompt versions. The goal is to make model selection and prompt optimization measurable across quality, cost, and execution characteristics.
Figure 2. Model comparison details showing evaluation outcomes, prompt counts, processing time, token usage, and response success.
Source-supported evaluation dimensions
- Failed evaluations and failure categories.
- Execution / processing time.
- Input and output token consumption.
- Success rates.
- Prompt-pack versions and model versions.
Technical workflow
| Step | What happens |
|---|---|
| 1. Version | Create or select prompt-pack versions that represent the behavior being tested. |
| 2. Compare | Run comparable prompt sets across multiple candidate LLMs. |
| 3. Evaluate | Review failures, timing, token cost, and success rate side by side. |
| 4. Select | Choose the model/prompt combination that best balances quality, performance, and cost for the use case. |
| 5. Iterate | Use the result as an optimization input before or during production rollout. |
| Key distinction
Prompt Comparison is comparative: it helps decide which prompt/model combination performs best. LLM Observability is operational: it monitors how deployed LLMs behave in production. |
Model Fine-Tuning
Model Fine-Tuning turns production failures into training inputs. The source describes a continuous-improvement workflow where failed prompts can be converted into training datasets and used to run reinforcement-learning jobs against foundational models.
Figure 3. Fine-tuning job creation and preview dataset. The former LLM Labs navigation label has been updated to AI Eval.
Technical workflow
| Stage | Source-supported behavior |
|---|---|
| Failure collection | Production failures and failed prompts provide evidence of behaviors that need improvement. |
| Dataset generation | Failed prompts can be converted into training datasets. |
| Job definition | A fine-tuning job is configured against a selected base model and dataset. |
| Reinforcement learning | The source states that reinforcement-learning jobs can adapt foundational models into custom models. |
| Domain adaptation | The resulting model is tuned using production evidence from the organization’s actual use case. |
Why it matters
- Moves improvement from manual guesswork toward evidence collected from real usage.
- Creates a closed loop between production reliability problems and model improvement.
- Supports more domain-specific behavior than relying only on a general-purpose foundation model.
Technical boundary
The source does not define the training algorithm configuration beyond the stated reinforcement-learning workflow, nor does it specify supported model providers, hardware requirements, training infrastructure, or deployment pipeline details.
AI Eval
AI Eval is the current name for the component referred to as LLM Labs in the source. It is the model testing and comparison environment used to evaluate foundational and open-source LLMs before selecting them for a use case.
Figure 4. AI Eval model-testing environment. The screenshot navigation has been updated from LLM Labs to AI Eval.
What AI Eval is designed to evaluate
- Model behavior across the quality dimensions that matter to the use case.
- Guardrail results for hallucination, safety, bias, and other configured evaluation dimensions.
- Comparative performance of different foundational and open-source models.
- Prompt behavior across models before production selection.
Technical use
| Function | Explanation |
|---|---|
| Model selection | Compare candidate models side by side rather than selecting by benchmark reputation alone. |
| Evaluation sessions | Run defined model/prompt evaluations and review the resulting status and outcomes. |
| Guardrail testing | Measure model behavior against reliability dimensions such as hallucination, safety, and bias. |
| Experimentation | Test model versions and prompt behavior before deployment, reducing production trial-and-error. |
| Evidence-based decision | Use measured evaluation results to select the model that best fits the workload. |
| Naming note
In this guide, AI Eval replaces the former product name LLM Labs. The underlying source description – model testing, comparison, and guardrail evaluation – is otherwise preserved. |
AI Gateway
AI Gateway is the production control layer for routing and governing model traffic. It is designed to balance performance, cost, availability, and trust controls as AI requests move through production.
Figure 5. AI Gateway model routing and usage view. The former LLM Labs navigation label has been updated to AI Eval.
Source-supported controls
- Route traffic across foundational and open-source models.
- Manage model availability.
- Work around or manage provider rate limits.
- Apply continuous trust and safety screening to requests.
- Use built-in open-source LLM hosting where appropriate.
Technical operating model
| Concern | Gateway role |
|---|---|
| Availability | Provide an operational point for managing model availability and alternate model paths. |
| Capacity / limits | Account for provider rate limits when routing production requests. |
| Cost / utilization | Track model usage and token consumption as part of production control. |
| Trust / safety | Screen requests continuously using configured trust and safety controls. |
| Model strategy | Use a mix of foundational and open-source models instead of binding the application to one model path. |
| Technical boundary
The source does not define the gateway protocol, routing algorithm, failover order, authentication model, supported providers, or exact trust/safety enforcement mechanism. |
LLM Observability
LLM Observability provides production monitoring for model performance, cost, change, and trust. It is the component used to understand deployed LLM behavior over time and drill from aggregate health into individual prompt-level traces.
Figure 6. LLM Insights view showing model usage, token consumption, prompt request counts, and health/performance context.
Signals described in the source
- Input and output token usage.
- Response times.
- Change events.
- Failed evaluations.
- Overall model behavior.
- Prompt-level LLM traces.
- Dedicated monitoring/workbench views for trust and safety, cost, and performance.
Technical workflow
| View | Purpose |
|---|---|
| Portfolio / fleet | Monitor all LLMs in one place rather than treating each model deployment independently. |
| Consumption | Understand token usage and prompt volume as production workload and cost signals. |
| Performance | Track response time and behavior changes that may indicate degradation. |
| Evaluation health | Surface failed evaluations alongside performance and cost rather than in a separate workflow. |
| Prompt drill-down | Use traces to investigate issues at the individual prompt level. |
| Relationship to Multi-Agent Tracing
LLM Observability focuses on model behavior, cost, performance, and prompt-level diagnostics. Multi-Agent Tracing expands the view across the complete multi-agent workflow, including tool calls and handoffs. |
Data Integrity Insights
Data Integrity Insights monitors the quality and consistency of data before those problems affect models or downstream systems. It gives teams a way to identify data-quality failures as an upstream reliability problem rather than treating model degradation only as a model issue.
Figure 7. Data Integrity Insights showing anomaly counts, data-quality issues, and trends.
Data-quality conditions described in the source
- Missing data.
- Field type mismatches.
- Outliers.
- Custom conditions.
- Trend tracking over time.
- Tracing issues back to their source.
Technical use
| Use case | Why it matters |
|---|---|
| Schema consistency | Catch field-type changes that can break downstream processing or alter model behavior. |
| Completeness | Detect missing data before it produces misleading or incomplete AI results. |
| Outlier monitoring | Identify abnormal values that can distort model input or downstream analysis. |
| Custom validation | Apply business-specific data-quality conditions beyond generic checks. |
| Source analysis | Trace detected integrity issues upstream so the data problem can be corrected at its origin. |
| Reliability role
Data Integrity Insights protects the inputs to AI. It complements LLM/ML observability by addressing data conditions that can create unreliable outputs even when the model itself has not changed. |
ML Observability
ML Observability monitors production machine-learning models for drift, bias, and performance risk. The source combines model-health detection with explainability, root cause analysis, and remediation capabilities.
Source-supported monitoring dimensions
- Data drift.
- Concept drift.
- Feature-level bias.
- Local and global explainability using SHAP values.
- Root cause analysis.
- Auto-remediation for bias, model drift, and data drift.
Technical interpretation
| Capability | Operational use |
|---|---|
| Data drift | Detect when the statistical characteristics of model inputs change from the learned/expected distribution. |
| Concept drift | Identify changes in the relationship between inputs and expected outcomes. |
| Feature bias | Surface feature-level behavior that may create unfair or unreliable model decisions. |
| SHAP explainability | Use local and global SHAP-based explanations to understand which features influence model behavior. |
| Root cause analysis | Connect detected model-health problems to likely contributing factors. |
| Auto-remediation | Use the platform’s stated remediation capability to respond to bias or drift conditions. |
| Technical boundary
The source does not define drift mathematics, SHAP computation mode, retraining triggers, remediation policy, or how auto-remediation is executed. Those details should be documented separately if implementation-specific guidance is required. |
How the components work together
The components are most useful when treated as one reliability lifecycle rather than separate point tools. The source positions InsightFinder as a cohesive workflow that connects development decisions, production observation, governance, and continuous improvement.
| Lifecycle question | Component(s) that answer it |
|---|---|
| Which model should we use? | AI Eval + LLM Prompt Comparison |
| How should requests be routed and governed? | AI Gateway |
| How are production LLMs behaving? | LLM Observability |
| Where is a multi-agent workflow failing? | Multi-Agent Tracing |
| Is poor behavior caused by bad data? | Data Integrity Insights |
| Is a production ML model drifting or biased? | ML Observability |
| How do we improve behavior using real failures? | Model Fine-Tuning |
Closed-loop example
| Stage | Example signal | Component | Result |
|---|---|---|---|
| 1. Observe | An agent trace contains failed evaluations and high duration. | Multi-Agent Tracing | Locate the problematic agent/LLM step. |
| 2. Diagnose | Prompt-level failures repeat across production traffic. | LLM Observability | Identify failure pattern and production impact. |
| 3. Evaluate | Alternative models/prompts are tested against quality, cost, and safety. | AI Eval + Prompt Comparison | Select a better configuration. |
| 4. Govern | Production traffic is routed with model-availability and safety controls. | AI Gateway | Deploy with operational guardrails. |
| 5. Improve | Failed prompts are converted into training evidence. | Model Fine-Tuning | Adapt model behavior using production evidence. |
| 6. Protect inputs | Data-quality anomalies or ML drift are detected upstream. | Data Integrity + ML Observability | Prevent recurring quality degradation. |
Source limitations and implementation details not specified
- Exact ingestion APIs, collectors, protocols, and telemetry schemas for AI Observability.
- Evaluation formulae, scoring thresholds, and guardrail algorithms.
- AI Gateway provider compatibility, routing policy syntax, and failover rules.
- Fine-tuning infrastructure, supported model families, and training parameter constraints.
- Drift-detection mathematics and automated remediation policy.
- Data retention, access control, deployment topology, and scale limits for these components.
Those implementation-specific details should be added from product engineering or platform configuration documentation when available; they are not defined in the supplied source.
From the Blog
Blogs 
See how InsightFinder helps your team deliver reliable services across every layer of the stack
Take InsightFinder AI for a no-obligation test drive. We’ll provide you with a detailed report on your outages to uncover what could have been prevented.
