Enable org.eclipse.jetty.io.ConnectionStatistics jmx bean and collect request statistics with collectd
- Enable org.eclipse.jetty.io.ConnectionStatistics jmx bean https://www.eclipse.org/jetty/documentation/current/statistics-handler.html
- Configure collectd for all cdm-server installations to collect the measures: https://gist.github.com/pmauduit/ed906f5439fd1aae44a73d5e67472e38
- configure facette graphs to show the measures for the four golden signals mentioned in (https://itnext.io/monitoring-distributed-jetty-servers-in-k8s-using-prometheus-and-grafana-7b02fe3506ad)
The four golden signals. These have been championed by the Google SRE team as the most fundamental metrics for tracking service health and performance.
- Latency: The time it takes to service a request.
- Traffic: A measure of how much demand is being placed on the service. In the case of Jetty, it is a number of requests per second.
- Errors: The rate of requests that fail. In the case of Jetty, we can count 4xx and 5xx errors.
- Saturation: How “full” is the service. This is a measure of the system utilization like memory, I/O or CPU. Services degrade in performance as they approach high saturation.
Let’s get back to the Jetty and JMX exporter agent and pick the right metrics
- org.eclipse.jetty.server.handler.statisticshandler.requesttimetotal measures the total time (in milliseconds) of requests handling.
- org.eclipse.jetty.server.handler.statisticshandler.requests — measures the total number of requests
- org.eclipse.jetty.server.handler.statisticshandler.responses4xx, org.eclipse.jetty.server.handler.statisticshandler.responses5xx — measures the total number of errors
- process_cpu_seconds_total, process_resident_memory_bytes — can be used as system metrics.