Case Study: Maintaining accurate and highly available AI Models with InsightFinder

Introduction

Many payment companies like Visa face big challenges in maintaining high availability and effectiveness of their real time transaction systems. The key challenges were detecting and fixing production problems such as transaction failures and  model drift before customers report them. A leading financial services provider turned to InsightFinder, a proactive AI platform, to address these challenges using InsightFinder patented machine learning technologies. This case study explores the solutions implemented and the outcomes achieved.

Why InsightFinder?

InsightFinder was chosen for its advanced capabilities in multivariate anomaly detection and its ability to provide deep insights into complex data patterns and root causes. The platform’s use of unsupervised machine learning made it an ideal fit for production system operation needs, offering:

• Early Detection of Problems: Proactively identifying issues before they escalate.
• Root Cause Analysis: Providing clear insights into the sources of problems.
• Continuous Monitoring: Ensuring ongoing system performance and reliability.
• Scalability: Handling large volumes of transaction data and system logs efficiently.

Use Case 1: AI Model Drift

Problem Definition

AI model drift occurs when a machine learning model’s accuracy deteriorates over time. Model drift is notoriously difficult to detect and the underlying root causes for the model drift can be a myriad of factors such as data changes, database issues, or hardware failures. The model drift could lead to significant revenue loss and customer dissatisfaction with the company’s AI products. 

Solution

InsightFinder’s AI-driven insight discovery platform was employed to monitor Visa’s AI models for signs of drift. The process involved:

1. Establishing a dynamic baseline for the model. InsightFinder can dynamically build baselines by extracting key indicators such as risk scores from raw transaction logs and building statistical patterns to model the baseline. This could be based on a combination of reference models and historical data.
2. Detection of Anomalies: Flagging anomalous model behavior changes when compared to reference or historical models indicative of model drift.
3. Root Cause Analysis: This included understanding the direction of the model changes—to the left or right of the statistical data—and identifying the reasons behind the drift, whether due to new fraud tactics, changes in user behavior, or other factors.
4. Model Retraining: Retraining and fine-tuning the models based on the insights provided by InsightFinder.
5. Validation and Deployment: Validating the updated models to ensure they met performance standards before redeployment.

Results

By leveraging InsightFinder, Visa achieved:

• Early Detection of Model Drift: Proactively addressing drift before it impacts customers.
• Confidence in model accuracy through automatic tracking at scale.
• Maintained Model Accuracy: Ensuring fraud detection models remained effective against evolving fraud tactics.
• Improved Customer Satisfaction: Reducing false positives and false negatives, leading to a better user experience.

Use Case 2: High Availability in AI Systems

Problem Definition

System availability is critical for the seamless operation of AI services. Any downtime or performance degradation can impact end-users and business operations. Ensuring high availability involves monitoring transaction logs and relevant metrics. Visa already had fault-tolerant systems that could detect failures, such as offline database nodes, and operate seamlessly with these problems. However, a problem is sometimes subtle and hard to detect, especially when the system is not down or offline.

Solution

InsightFinder’s proactive AI algorithms monitored Visa’s systems for availability issues, focusing on:

1. Continuous Monitoring: Analyzing transaction logs and system metrics to detect low availability.
2. Real-time Availability Anomaly Detection: Identifying availability anomalies that could indicate potential system availability issues.
3. Root Cause Isolation: Isolating the root causes of availability problems for quick resolution.
4. Proactive Maintenance: Performing maintenance based on early warnings to prevent downtime.

Results

Implementing InsightFinder’s solutions led to:

• Reduced Downtime: Early detection and resolution of issues minimized system downtime.
• Enhanced Performance: Continuous monitoring ensured fraud detection models operated at optimal levels.
• Improved User Experience: Maintaining high AI service availability ensured a seamless experience for Visa’s users.

Timeframe

InsightFinder’s AI engine was set up within a few hours in production environments and started to deliver problem detection and root cause analysis results within just a few weeks. 

Conclusion

InsightFinder’s proactive AI solutions were crucial in addressing Visa’s model drift and system availability challenges. By providing early detection of anomalies, deep insights into root causes, and continuous monitoring, InsightFinder helped Visa maintain the effectiveness and reliability of its fraud detection systems. The result was improved business outcomes, enhanced system performance, and increased customer satisfaction.