Fluentd Integration

InsightFinder can stream the logs from Confluent Kafka topics and correlate it with other data to generate anomalies, automatically identify root causes and predict incidents. Below documentation is a walk through of how to setup and configure a confluent kafka agent and project.

Single Node Manual Installation

  1. Go to InsightFinder Agent td-agent repo
  2. Download fluent-plugin-out-InsightFinder.gemspec and out_InsightFinder.rb
  3. Move these 2 files to either /etc/fluent/plugin or /etc/td-agent/plugin depending on how you install the Fluentd
  4. By default, Fluentd adds the /etc/fluent/plugin directory to its load path. Thus, any additional plugins that are placed in /etc/fluent/plugin will be loaded automatically. If using td-agent, use /etc/td-agent/plugin
  5. Modify the configuration file (should be located in /etc/td-agent/td-agent.conf) of Fluentd, add the section for insight finder plugin according to example file td-agent.conf.example
  6. Restart Fluentd and the data will be ingested into InsightFinder.

Tested with Fluentd version 1.0

Multi-node installation using Ansible