Project

General

Profile

task #8635

install collectd + facette for server monitoring

Added by Andreas Kohlbecker almost 2 years ago. Updated 11 months ago.

Status:
Closed
Priority:
New
Category:
devOps
Target version:
Start date:
10/29/2019
Due date:
% Done:

100%

Severity:
normal

Description

We will use collectd + facette for now:

Test installation on edit-test as client and edit-staging as master to collect the time series data

The facette server is running at: http://edit-staging.bgbm.fu-berlin.de/facette/

collectd

https://www.21x9.org/de/serverueberwachung-mit-collectd-und-facette.html
https://www.tecmint.com/configure-collectd-as-central-monitoring-server-for-clients/2/

installation

Collectd will be installed on the client and master

apt-get install collectd

Configuration

create types.db file with custom types (needed for GenericJMX), create this file for the server and client machine

echo "jmx_memory      value:GAUGE:0:U
time_ms      value:GAUGE:0:U" > /etc/collectd/my_types.db    

and uncomment the below line in /etc/collectd/collectd.conf

TypesDB "/usr/share/collectd/types.db" "/etc/collectd/my_types.db"

Apache

https://collectd.org/wiki/index.php/Plugin:Apache

enable and configure mod_status:

a2enmod status # should be enabled already
systemctl reload apache2 # only if needed

in /etc/apache2/mods-available/status.conf:

ExtendedStatus on
<Location /mod_status>
  SetHandler server-status
</Location>

MySQL

https://collectd.org/wiki/index.php/Plugin:MySQL

  1. create mysql user collectd (see link above)
  2. configure collectd:
<Plugin mysql>
  <Database localhost>
    Port "3306"
    User "collectd"
    Password "*****"
    MasterStats true
  </Database>
</Plugin>

Docker

No official collectd plugin so far. So are using a contributed plugin https://github.com/lebauce/docker-collectd-plugin. A good installation guide is found in the fork https://github.com/signalfx/docker-collectd-plugin

Facette

https://github.com/facette/facette

Documentation

There is currently only sparse documentation on the configuration of facette. The only source of documentation are the issues:

Templates:

  • A graph or a collection is considered being a template once markups are detected in certain fields
  • Both graphs and collections can be templates (e.g. if you use {{ .source }} in the source field of a metric in a graph, you'll be able a reuse this graph for multiple hosts, providing a value for source in either a graph created from a template or in a collection referencing it);
  • Create new graph/collection will be templates only if template markups are detected (e.g. {{ .var }}). Create from template is used to create resources based on existing templates, it'll ask you to provide values for markups variables existing in the source template;

dump and restore of facette setups

facettectl -a http://$(hostname -I):12003 library dump -o facette-lib_$(date -I).tar.gz
facettectl -a http://$(hostname -I):12003 library restore -i facette-lib****.tar.gz

Installation

installing facette from binaries

https://github.com/facette/facette/releases

mkdir /root/installer
wget https://github.com/facette/facette/releases/download/0.5.1/facette_0.5.1_stretch-amd64.deb
dpkg -i facette_0.5.1_stretch-amd64.deb

Install from sources

apt-get install build-essential golang-go librrd-dev pkg-config pandoc protobuf-compiler

install node_js:

https://github.com/nodesource/distributions/blob/master/README.md#deb

curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
sudo apt-get install -y nodejs
npm install yarn -g

....

Setup and configuration

edit /etc/facette/facette.yaml:

listen: 160.45.63.177:12003

restart facette:

systemctl restart facette.service 

Admin panel

The admin panel is usually set readonly. see /etc/facette/facette.yaml

  # Allow/Prevent modifications through API calls
  read_only: true

Configure the provider

(See https://github.com/facette/facette/issues/336)

curl -s -H 'Content-Type: application/json' http://160.45.63.177:12003/api/v1/providers -d '                                                                                                                ─╯
{
  "name": "collectd",
  "description": null,
  "created": "2017-06-14T06:09:19Z",
  "modified": "2017-06-14T06:09:19Z",
  "connector": "rrd",
  "settings": {
    "path": "/var/lib/collectd",
    "pattern": "rrd/(?P<source>[^/]+)/(?P<metric>.+)\\.rrd"
  },
  "filters": [
    {"action": "sieve", "target": "metric", "pattern": "/average$", "into": ""},
    {"action": "rewrite", "target": "metric", "pattern": "/average$", "into": ""},
    {"action": "rewrite", "target": "metric", "pattern": "/value$", "into": ""},
    {"action": "rewrite", "target": "metric", "pattern": "/", "into": "."},
    {"action": "rewrite", "target": "metric", "pattern": "^cpu\\.percent-", "into": "cpu.percent."},
    {"action": "rewrite", "target": "metric", "pattern": "^df-(.+)\\.(?:df_)?(.+)-(.+)", "into": "df.$1.$2.$3"},
    {"action": "rewrite", "target": "metric", "pattern": "^interface-(.+).if_(.+)\\.", "into": "net.$1.$2."},
    {"action": "rewrite", "target": "metric", "pattern": "^load\\.load\\.", "into": "load."},
    {"action": "rewrite", "target": "metric", "pattern": "^memory\\.memory-", "into": "memory."}
  ],
  "refresh_interval": 0,
  "priority": 0,
  "enabled": true
}
'

the new provider should now be available at http://edit-staging:12003/admin/providers/

History

#1 Updated by Andreas Kohlbecker almost 2 years ago

  • Status changed from New to In Progress

#2 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)
  • Private changed from No to Yes

#3 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#4 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#5 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#6 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#7 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#8 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#9 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#10 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#11 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#12 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#13 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#14 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#15 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#16 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#17 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#18 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#19 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#20 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#21 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#22 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#23 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#24 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#25 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#26 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#27 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#28 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#29 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#30 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#31 Updated by Andreas Kohlbecker almost 2 years ago

  • Subject changed from install server monitoring solution to install collectd + facette for server monitoring
  • Description updated (diff)
  • Private changed from Yes to No

#32 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#33 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#34 Updated by Andreas Kohlbecker almost 2 years ago

  • Target version changed from Unassigned CDM tickets to Release 5.12

#35 Updated by Andreas Kohlbecker almost 2 years ago

  • % Done changed from 0 to 30

#36 Updated by Andreas Kohlbecker almost 2 years ago

  • Target version changed from Release 5.12 to Release 5.13

#37 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#38 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#39 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#40 Updated by Andreas Kohlbecker almost 2 years ago

  • Description updated (diff)

#41 Updated by Andreas Kohlbecker over 1 year ago

  • Description updated (diff)

#42 Updated by Andreas Kohlbecker over 1 year ago

  • Description updated (diff)

#43 Updated by Andreas Kohlbecker over 1 year ago

  • Description updated (diff)

#44 Updated by Andreas Kohlbecker over 1 year ago

  • Description updated (diff)

#45 Updated by Andreas Kohlbecker over 1 year ago

  • Description updated (diff)

#46 Updated by Andreas Kohlbecker over 1 year ago

  • Target version changed from Release 5.13 to Release 5.14
  • % Done changed from 30 to 40

mostly done.

#47 Updated by Andreas Müller over 1 year ago

  • Target version changed from Release 5.14 to Release 5.15

#48 Updated by Andreas Kohlbecker over 1 year ago

  • Target version changed from Release 5.15 to Release 5.18

#49 Updated by Andreas Kohlbecker over 1 year ago

  • Description updated (diff)

#50 Updated by Andreas Kohlbecker 11 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 40 to 100

this is now fairly complete. the ticket is linked from the ServerAdministration wiki page, so it can be closed

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 40 MB)