Skip to main content

Monitoring

Raccoon provides monitoring for server connection, publisher, resource usage, and event delivery. Reference for available metrics is available here. The metrics are reported using Statsd and Prometheus

How To Setup​

TL;DR
- Run Statsd supported metric collector
- Configure `METRIC_STATSD_ADDRESS` on Raccoon to send to the metric collector
- Visualize and create alerting from the collected metrics

Generally, you can follow the steps above and use any metric collector that supports statsd like Telegraf or Datadog.

This section will cover a setup example using Telegraf, Influx, Kapacitor, and Grafana stack based on the steps above.

Run Statsd Supported Metric Collector To enable statsd on Telegraf you need to enable statsd input on telegraf.conf file. Following are default configurations that you can add based on statsd input readme.

[[inputs.statsd]]
  protocol = "udp"
  max_tcp_connections = 250
  tcp_keep_alive = false
  service_address = ":8125"

  delete_gauges = true
  delete_counters = true
  delete_sets = true
  delete_timings = true

  percentiles = [50.0, 90.0, 99.0, 99.9, 99.95, 100.0]

  metric_separator = "_"

  parse_data_dog_tags = false
  datadog_extensions = false
  datadog_distributions = false

  allowed_pending_messages = 10000
  percentile_limit = 1000

[[outputs.influxdb]]
  urls = ["http://127.0.0.1:8086"]
  database = "raccoon"
  retention_policy = "autogen"
  write_consistency = "any"
  timeout = "5s"

Configure Raccoon To Send To The Metric Collector After you have the collector with the port configured, you need to set METRIC_STATSD_ADDRESS to match the metric collector address. Suppose you deploy the telegraf using the default configuration above as sidecar or in localhost, you need to set the value to :8125.

Visualize And Create Alerting From The Collected Metrics Now that you have Raccoon and Telegraf as metric collector set, next is to use the metrics reported. You may notice that the Telegraf config above contains outputs.influxdb. That config will send the metric received to Influxdb. Make sure you have influx service accessible from the configured URL. You can visualize the metrics using Grafana. To do that, you need to add influx datasource to make the data available on Grafana. After that, you can use the data to You can visualize the metrics using Grafana.

Metrics Usages​

Following are key monitoring statistics that you can infer from Raccoon metrics. Refer to this section to understand how to build alerting, dashboard, or analyze the metrics.

Data Loss​

To infer data loss, you can count kafka_messages_delivered_total with tag success=false. You can also infer the loss rate by calculating the following.

count(kafka_messages_delivered_total success=false)/count(kafka_messages_delivered_total)

For other publishers, just replace kafka in the metric name with the name of the publisher. For instance, analogs of kafka_messages_delivered_total for PubSub and Kinesis would be:

Latency​

Raccoon provides fine-grained metrics that denote latency. That gives clues as to where to look in case something goes wrong. In summary, these are key metrics for latency:

Dashboard​

There is a pre-built grafana dashboard available with support for Prometheus data source.

If you're running the statsd + telegraf setup, you can configure telegraf to push metrics to Prometheus.