Project

General

Profile

task #9144

Enable org.eclipse.jetty.io.ConnectionStatistics jmx bean and collect request statistics with collectd

Added by Andreas Kohlbecker 29 days ago.

Status:
New
Priority:
Highest
Category:
cdmserver
Target version:
Start date:
07/10/2020
Due date:
% Done:

0%

Severity:
major

Description

  1. Enable org.eclipse.jetty.io.ConnectionStatistics jmx bean https://www.eclipse.org/jetty/documentation/current/statistics-handler.html
  2. Configure collectd for all cdm-server installations to collect the measures: https://gist.github.com/pmauduit/ed906f5439fd1aae44a73d5e67472e38
  3. configure facette graphs to show the measures for the four golden signals mentioned in (https://itnext.io/monitoring-distributed-jetty-servers-in-k8s-using-prometheus-and-grafana-7b02fe3506ad)

The four golden signals. These have been championed by the Google SRE team as the most fundamental metrics for tracking service health and performance.

  1. Latency: The time it takes to service a request.
  2. Traffic: A measure of how much demand is being placed on the service. In the case of Jetty, it is a number of requests per second.
  3. Errors: The rate of requests that fail. In the case of Jetty, we can count 4xx and 5xx errors.
  4. Saturation: How “full” is the service. This is a measure of the system utilization like memory, I/O or CPU. Services degrade in performance as they approach high saturation.

Let’s get back to the Jetty and JMX exporter agent and pick the right metrics

  1. org.eclipse.jetty.server.handler.statisticshandler.requesttimetotal measures the total time (in milliseconds) of requests handling.
  2. org.eclipse.jetty.server.handler.statisticshandler.requests — measures the total number of requests
  3. org.eclipse.jetty.server.handler.statisticshandler.responses4xx, org.eclipse.jetty.server.handler.statisticshandler.responses5xx — measures the total number of errors
  4. process_cpu_seconds_total, process_resident_memory_bytes — can be used as system metrics.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 40 MB)