task #8635
closedinstall collectd + facette for server monitoring
100%
Description
We will use collectd + facette for now:
Test installation on edit-test as client and edit-staging as master to collect the time series data
The facette server is running at: http://edit-staging.bgbm.fu-berlin.de/facette/
collectd¶
https://www.21x9.org/de/serverueberwachung-mit-collectd-und-facette.html
https://www.tecmint.com/configure-collectd-as-central-monitoring-server-for-clients/2/
installation¶
Collectd will be installed on the client and master
apt-get install collectd
Configuration¶
create types.db file with custom types (needed for GenericJMX), create this file for the server and client machine
echo "jmx_memory value:GAUGE:0:U time_ms value:GAUGE:0:U" > /etc/collectd/my_types.db
and uncomment the below line in /etc/collectd/collectd.conf
TypesDB "/usr/share/collectd/types.db" "/etc/collectd/my_types.db"
Apache
https://collectd.org/wiki/index.php/Plugin:Apache
enable and configure mod_status:
a2enmod status # should be enabled already systemctl reload apache2 # only if needed
in /etc/apache2/mods-available/status.conf:
ExtendedStatus on <Location /mod_status> SetHandler server-status </Location>
MySQL
https://collectd.org/wiki/index.php/Plugin:MySQL
- create mysql user collectd (see link above)
- configure collectd:
<Plugin mysql>
<Database localhost>
Port "3306"
User "collectd"
Password "*****"
MasterStats true
</Database>
</Plugin>
Docker
No official collectd plugin so far. So are using a contributed plugin https://github.com/lebauce/docker-collectd-plugin. A good installation guide is found in the fork https://github.com/signalfx/docker-collectd-plugin
Facette¶
https://github.com/facette/facette
Documentation¶
There is currently only sparse documentation on the configuration of facette. The only source of documentation are the issues:
- RRD Provider setup in Admin Panel
- RRD Provider setup via API
- facette-graph-templates
- Add collections templates #170
- create new graph/collection -> create new graph/collection template
- Graphs/Collections and Templates
- Aggregation function over metric group in a graph template
Templates:
- A graph or a collection is considered being a template once markups are detected in certain fields
- Both graphs and collections can be templates (e.g. if you use
{{ .source }}
in the source field of a metric in a graph, you'll be able a reuse this graph for multiple hosts, providing a value for source in either a graph created from a template or in a collection referencing it); - Create new graph/collection will be templates only if template markups are detected (e.g.
{{ .var }}
). Create from template is used to create resources based on existing templates, it'll ask you to provide values for markups variables existing in the source template;
dump and restore of facette setups
facettectl -a http://$(hostname -I):12003 library dump -o facette-lib_$(date -I).tar.gz facettectl -a http://$(hostname -I):12003 library restore -i facette-lib****.tar.gz
Installation¶
installing facette from binaries¶
https://github.com/facette/facette/releases
mkdir /root/installer wget https://github.com/facette/facette/releases/download/0.5.1/facette_0.5.1_stretch-amd64.deb dpkg -i facette_0.5.1_stretch-amd64.deb
Install from sources¶
apt-get install build-essential golang-go librrd-dev pkg-config pandoc protobuf-compiler
install node_js:
https://github.com/nodesource/distributions/blob/master/README.md#deb
curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash - sudo apt-get install -y nodejs npm install yarn -g
....
Setup and configuration¶
edit /etc/facette/facette.yaml:
listen: 160.45.63.177:12003
restart facette:
systemctl restart facette.service
Admin panel¶
The admin panel is usually set readonly. see /etc/facette/facette.yaml
# Allow/Prevent modifications through API calls read_only: true
Configure the provider
(See https://github.com/facette/facette/issues/336)
curl -s -H 'Content-Type: application/json' http://160.45.63.177:12003/api/v1/providers -d ' ─╯ { "name": "collectd", "description": null, "created": "2017-06-14T06:09:19Z", "modified": "2017-06-14T06:09:19Z", "connector": "rrd", "settings": { "path": "/var/lib/collectd", "pattern": "rrd/(?P<source>[^/]+)/(?P<metric>.+)\\.rrd" }, "filters": [ {"action": "sieve", "target": "metric", "pattern": "/average$", "into": ""}, {"action": "rewrite", "target": "metric", "pattern": "/average$", "into": ""}, {"action": "rewrite", "target": "metric", "pattern": "/value$", "into": ""}, {"action": "rewrite", "target": "metric", "pattern": "/", "into": "."}, {"action": "rewrite", "target": "metric", "pattern": "^cpu\\.percent-", "into": "cpu.percent."}, {"action": "rewrite", "target": "metric", "pattern": "^df-(.+)\\.(?:df_)?(.+)-(.+)", "into": "df.$1.$2.$3"}, {"action": "rewrite", "target": "metric", "pattern": "^interface-(.+).if_(.+)\\.", "into": "net.$1.$2."}, {"action": "rewrite", "target": "metric", "pattern": "^load\\.load\\.", "into": "load."}, {"action": "rewrite", "target": "metric", "pattern": "^memory\\.memory-", "into": "memory."} ], "refresh_interval": 0, "priority": 0, "enabled": true } '
the new provider should now be available at http://edit-staging:12003/admin/providers/