Confluent Kafka Integration

Insightfinder – Confluent Kafka Integration
Insightfinder can source the metrics and logs data from any user source and stream real time using the confluent kafka cluster for analysis. This makes InsightFinder analysis and prediction more real time and predictive. Please refer to the rest of this document for steps to configure the projects.

Title :- Confluent Kafka Integration for Logs

Summary – InsightFinder can stream the logs from Confluent Kafka topics and correlate it with other data to generate anomalies, automatically identify root causes and predict incidents. Below documentation is a walk through of how to setup and configure a confluent kafka agent and project.

Project Creation :-

1. Go to “Settings”->“System Settings”. Click on “Add New Project”.
2. Select “Custom” from the list and click on “Create Project” on the next page.
3. This is where you will start configuring the project.
   1. Instance Type – Private Cloud
   2. Data Type – Log
   3. Agent Type – Live Streaming
4. Click on “Next” and you will be taken to the next page.
5. Then on the next page, you can type in the “Project Name” and “System Name” and click on Register. You can also define detection keywords and Incident labels.
   1. Detection keywords/regular expressions are used to detect which log entries will produce allowlist alerts.
   2. Incident labels/regex are used to identify which log entries indicate incidents.
6. Once a project is successfully created, you will see a message like below.

Once a project is successfully created, you need to set-up an agent.

Agent Setup :-
You would need a server where you can run this agent
For container –

1. Access the image – https://hub.docker.com/r/insightfinderagent/insightfinder-kafka-log-agent
2. Follow the steps from the overview section to use the agent

For github –

1. Download the agent from the link (https://github.com/2623238476/InsightAgent/blob/master/kafka_confluent_log/kafka_confluent_log.tar.gz) on the server.
2. Follow the steps from the link (https://github.com/2623238476/InsightAgent/blob/master/kafka_confluent_log/README.md) to install the agent on the server.

Once you have an agent running, you will be able to see the data streaming into the InsightFinder project. To check,

1. Go to Analysis->Log/Trace Analysis.
2. Select your project and date. Click on the Refresh button on the right top. You will get the calendar view of log data.
   

Title :- Confluent Kafka Integration for Metrics

Summary – InsightFinder can source the metrics from Confluent Kafka and correlate it with other data to generate anomalies, automatically identify root causes and predict incidents. Below documentation is a walk through of how to setup and configure a confluent kafka agent and project.

Project Creation :-

1. Go to “Settings”->“System Settings”. Click on “Add New Project”.
2. Select “Custom” from the list and click on “Create Project” on the next page.
3. This is where you will start configuring the project.
   1. Instance Type – Private Cloud
   2. Data Type – Metric
   3. Agent Type – Streaming

   

4. Click on “Next” and you will be taken to the next page.
5. Then on the next page, you can type in the “Project Name” and “System Name” and click on Register.
6. Once a project is successfully created, you will see a message like below.

Once a project is successfully created, you need to set-up the agent.

Agent Setup :-
You would need a server where you can run this agent
For container –

1. Access the image – https://hub.docker.com/r/insightfinderagent/insightfinder-kafka-metric-agent
2. Follow the steps from the overview section to use the agent

For github –

1. Download the agent from the link (https://github.com/2623238476/InsightAgent/blob/master/kafka_confluent_metric/kafka_confluent_metric.tar.gz) on the server.
2. Follow the steps from the link (https://github.com/2623238476/InsightAgent/blob/master/kafka_confluent_metric/README.md) to install the agent on the server.

Once you have an agent running, you will be able to see the data streaming into the InsightFinder project. To check,

1. Go to Analysis->Metric Analysis->Overall Line Chart.
2. Select your project and date. Click on Reload button on the right top.