Project

General

Profile

Actions

bug #7085

closed

cdmserver logfiles getting too big, use RollingFileAppender

Added by Andreas Kohlbecker over 6 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Highest
Category:
cdmserver
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:
Severity:
major
Found in Version:

Description

cdmserver logfiles getting too big, use DailyRollingFileAppender

Keeping 7 daily logs should be sufficient since all logs are also preserved in the backupserver

Also delete all /var/log/cdmserver/out.log files once this is released.

see https://github.com/GoogleCloudPlatform/appengine-java-vm-runtime/issues/231

Configuring the logging consistently for the cdm-server and for all instances is complicated by the fact that the instances are running separately from the cdm-server in an own thread with own classloader for each instance. Therefore the root logger configured for the cdm-server is not being used by the instances.

TODO

  • Setup the logging for the cdm-server so that the server and all instances are using the same DailyRollingFileAppender and logfile.
  • remove FileAppenders configured in /opt/cdmserver/.cdmLibrary/log4j.properties of edit-test, edit-prod, edit-int, edit-demo-*, edit-jobber-1
  • consider using log4j2, see #7821
  • logfile paths in the cdm-server/src/main/resources/log4j.properties must not be hardcoded and platform-specific, like log4j.appender.FILE.File= /var/log/cdmserver/cdmserver_log.out - fixed by cdm-server|bb431d84


Solution

Jetty can be configured to use the Managed Diagnostic Context of either log4j or logback to centralize the logging from multiple webapps in to a central log file. This technique is explained in https://www.eclipse.org/jetty/documentation/9.4.29.v20200521/example-logging-logback-centralized.html for a jetty server which uses a deployment manager. In our situation of an embedded jetty which manages the cdm-webapp instance directly the configuration need to be a bit different, see eu.etaxonomy.cdm.server.logging.LoggingConfigurator for details.

Consequences of this change:

  • The cdm-server start option logdir has now been replanced by the system property cdm.logDir so all server installations need to checked and modified -logdir=... ==> -Dcdm.logDir=file://...
    • adapt the docker configuration
    • adapt the debian package?
  • The system property -Dlog4j.configuration=file://.. can no longer be used and needs to be replaced by e.g.: -Dlogback.configurationFile=file:/app/logback.xml

Related issues

Related to EDIT - feature request #7819: Log4j.properties file is not properly packaged by the cdmserver install scriptNewAndreas Kohlbecker

Actions
Related to EDIT - feature request #7820: Add cron job for deleting cdmserver log files after a specified time periodRejectedAndreas Kohlbecker

Actions
Related to EDIT - feature request #6249: better logging strategyClosedAndreas Kohlbecker

Actions
Related to EDIT - bug #9236: stopping an instance breaks the centralized loggingClosedAndreas Kohlbecker

Actions
Has duplicate EDIT - feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log filesDuplicateAndreas Kohlbecker

Actions
Has duplicate EDIT - bug #6287: Rolling cdmserver.log file only contains entries from startup phaseDuplicateAndreas Kohlbecker

Actions
Actions #1

Updated by Andreas Kohlbecker over 6 years ago

Also delete all /var/log/cdmserver/out.log files once this is released

Actions #2

Updated by Andreas Kohlbecker over 6 years ago

  • Priority changed from New to Highest
Actions #3

Updated by Andreas Kohlbecker over 6 years ago

  • Description updated (diff)
Actions #4

Updated by Andreas Müller over 6 years ago

  • Target version changed from Release 4.12 to Release 4.13
Actions #5

Updated by Andreas Müller about 6 years ago

  • Target version changed from Release 4.13 to Release 4.14
Actions #6

Updated by Andreas Müller about 6 years ago

  • Target version changed from Release 4.14 to Release 5.0
Actions #7

Updated by Andreas Kohlbecker almost 6 years ago

  • Target version changed from Release 5.0 to Release 5.1
Actions #8

Updated by Andreas Kohlbecker almost 6 years ago

  • Status changed from New to In Progress
Actions #9

Updated by Andreas Kohlbecker almost 6 years ago

  • Description updated (diff)
Actions #10

Updated by Andreas Kohlbecker over 5 years ago

  • Target version changed from Release 5.1 to Release 5.2
Actions #11

Updated by Andreas Kohlbecker over 5 years ago

  • Target version changed from Release 5.2 to Release 5.3
Actions #12

Updated by Andreas Kohlbecker over 5 years ago

  • Status changed from In Progress to New
  • Target version changed from Release 5.3 to Release 5.4
Actions #13

Updated by Andreas Kohlbecker over 5 years ago

  • Assignee changed from Andreas Kohlbecker to Patrick Plitzner
Actions #14

Updated by Patrick Plitzner over 5 years ago

  • Related to bug #6287: Rolling cdmserver.log file only contains entries from startup phase added
Actions #15

Updated by Patrick Plitzner over 5 years ago

  • Related to feature request #7819: Log4j.properties file is not properly packaged by the cdmserver install script added
Actions #16

Updated by Patrick Plitzner over 5 years ago

  • Status changed from New to Resolved
  • Assignee changed from Patrick Plitzner to Andreas Kohlbecker
  • % Done changed from 0 to 50

The logging is now configured to daily logging i.e. every day a new log file is generated.

The change was done manually in the /opt/cdmserver/.cdmLibrary/log4j.properties file. This should be handled by the install script of the cdm server -> #7819

Remaining issues is deleting the old log files after a specified time period -> #7820

Actions #17

Updated by Patrick Plitzner over 5 years ago

  • Related to feature request #7820: Add cron job for deleting cdmserver log files after a specified time period added
Actions #18

Updated by Patrick Plitzner over 5 years ago

  • Related to feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files added
Actions #19

Updated by Andreas Kohlbecker over 5 years ago

  • Status changed from Resolved to In Progress
  • Assignee changed from Andreas Kohlbecker to Patrick Plitzner

The rolled daily log files have a whitespace character, please remove the whitespace from the filenames.

-rw-r--r-- 1 cdm adm 409569 Oct 19 12:18 cdmserver_log.out
-rw-r--r-- 1 cdm adm 262768 Oct 12 09:43 cdmserver_log.out. 2018-10-12
-rw-r--r-- 1 cdm adm 143578 Oct 13 05:07 cdmserver_log.out. 2018-10-13
-rw-r--r-- 1 cdm adm 150123 Oct 14 09:40 cdmserver_log.out. 2018-10-14
-rw-r--r-- 1 cdm adm 549825 Oct 15 19:11 cdmserver_log.out. 2018-10-15
-rw-r--r-- 1 cdm adm 343656 Oct 16 18:27 cdmserver_log.out. 2018-10-16
-rw-r--r-- 1 cdm adm 599338 Oct 17 20:14 cdmserver_log.out. 2018-10-17
-rw-r--r-- 1 cdm adm 623044 Oct 18 23:11 cdmserver_log.out. 2018-10-18
Actions #20

Updated by Andreas Kohlbecker over 5 years ago

i found a second issue:

The rolling daily file appender only works if the custom loj4j.properties are defined in the /etc/default/cdmserver

#
# user custom loj4j.properties
#
JAVA_OPTIONS="$JAVA_OPTIONS -Dlog4j.configuration=file:///opt/cdmserver/.cdmLibrary/log4j.properties"

The rolling daily file appender should be used in default configuration.

Actions #21

Updated by Andreas Kohlbecker over 5 years ago

third issue:

unfortunately you committed all changes to the master branch:

* 469b75d - Fri, 12 Oct 2018 11:26:09 +0200 (7 days ago) (origin/master, origin/HEAD)
|   ref #7085 Adjust log path - Patrick Plitzner
* 0465659 - Fri, 12 Oct 2018 10:47:59 +0200 (7 days ago)
|   ref #7085 Set logging to daily rollover - Patrick Plitzner
* 72c3d33 - Fri, 12 Oct 2018 09:18:02 +0200 (7 days ago)
|   ref #7085 Adjust log path - Patrick Plitzner
* b78e167 - Thu, 11 Oct 2018 17:13:40 +0200 (8 days ago)
|   Revert "ref #7085 Use DailyRollingFileAppender (in code)" - Patrick Plitzner
* 02ef1e4 - Thu, 11 Oct 2018 16:57:36 +0200 (8 days ago)
|   ref #7085 Use DailyRollingFileAppender (in code) - Patrick Plitzner
* 99c113d - Thu, 11 Oct 2018 16:43:44 +0200 (8 days ago)
|   ref #7085 Use DailyRollingFileAppender to generate one log per day - Patrick Plitzner
*   c172618 - Mon, 17 Sep 2018 11:28:56 +0200 (5 weeks ago) (tag: 5.3.0)
|\    jenkins merging release branch into master (strategy: theirs) - jenkins
| * 9efc161 - Mon, 17 Sep 2018 11:28:15 +0200 (5 weeks ago)
| |   updating poms for branch'release/5.3.0' with non-snapshot versions - jenkins
* |   83415cd - Fri, 17 Aug 2018 16:08:00 +0200 (9 weeks ago) (tag: 5.2.0)

This will break the next release since git-flow can not deal with intermediate commits to the master branch, we will run into merge conflicts. Your changes being committed to the master branch could also be the reason why the daily rolling fileappender is not working in those servers which are not configured to use the custom log4j.properties

Please fix the git repo. The clone at git hub must also be fixed. You must whipe out your changes completely from master. A git revert might not be enough in this case.

BTW: Did you replace the old log4j.properties on the edit-test server or did exchange the old appender by the daily rolling fileappender?

Actions #22

Updated by Patrick Plitzner over 5 years ago

  • Assignee changed from Patrick Plitzner to Andreas Kohlbecker
Actions #23

Updated by Andreas Kohlbecker over 5 years ago

  • Description updated (diff)
Actions #24

Updated by Andreas Kohlbecker over 5 years ago

  • % Done changed from 50 to 20
Actions #25

Updated by Andreas Kohlbecker over 5 years ago

we can use the Log4J config provided as

-Dlog4j.configuration=file:///opt/cdmserver/.cdmLibrary/log4j.properties

in the /etc/default/cdmserver as preliminary solution. But this is not reliable since it does not work out of the box when the cdm-server is installed newly.

Interestingly with -Dlog4j.configuration the log4j.properties is used in the bootloader and in the instances. In contrast to that consistent configuration seems to be hard to achieve out of the Bootloader, see cdm-server|bb431d84

Actions #26

Updated by Andreas Kohlbecker over 5 years ago

for the coming release I will add the following configuration to edit-int, edit-test and edit-production:

##
## NOTICE !!!!
## no need to configure a FileAppender here
## use eu.etaxonomy.cdm.remote.config.LoggingConfigurer instead
##
#--------------------------------------------------------------------------------------------------
# as interim solution in order to have a daily rolling file appender we configer it here
# see https://dev.e-taxonomy.eu/redmine/issues/7085
log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender
log4j.appender.FILE.File= ${LOGGING_FOLDER}/cdmserver.log
log4j.appender.FILE.ImmediateFlush=true
log4j.appender.FILE.Threshold=debug
log4j.appender.FILE.Append=true
log4j.appender.FILE.DatePattern='.'yyyy-MM-dd
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.ConversionPattern=%d %p [%c] - %m%n
#--------------------------------------------------------------------------------------------------

This needs to be removed once the DailyRollingFileAppender is being applied out of the cdm-server.

Actions #27

Updated by Andreas Kohlbecker over 5 years ago

  • Description updated (diff)
Actions #28

Updated by Andreas Kohlbecker over 5 years ago

  • Description updated (diff)
Actions #29

Updated by Andreas Kohlbecker over 5 years ago

  • Target version changed from Release 5.4 to Release 5.5
Actions #30

Updated by Andreas Kohlbecker about 5 years ago

  • Target version changed from Release 5.5 to Release 5.6
Actions #31

Updated by Andreas Kohlbecker about 5 years ago

  • Target version changed from Release 5.6 to Reviewed Next Major Release
Actions #32

Updated by Andreas Kohlbecker about 4 years ago

  • Target version changed from Reviewed Next Major Release to Release 5.13
Actions #33

Updated by Andreas Kohlbecker about 4 years ago

  • Target version changed from Release 5.13 to Release 5.14
Actions #34

Updated by Andreas Müller almost 4 years ago

  • Target version changed from Release 5.14 to Release 5.15
Actions #35

Updated by Andreas Kohlbecker almost 4 years ago

  • Description updated (diff)
Actions #36

Updated by Andreas Kohlbecker almost 4 years ago

  • Description updated (diff)
Actions #37

Updated by Andreas Kohlbecker almost 4 years ago

  • Description updated (diff)
Actions #38

Updated by Andreas Kohlbecker almost 4 years ago

  • Description updated (diff)
  • % Done changed from 20 to 30
Actions #39

Updated by Andreas Kohlbecker almost 4 years ago

  • Description updated (diff)
Actions #40

Updated by Andreas Kohlbecker almost 4 years ago

  • Subject changed from cdmserver logfiles getting too big, use DailyRollingFileAppender to cdmserver logfiles getting too big, use RollingFileAppender
Actions #41

Updated by Andreas Kohlbecker almost 4 years ago

Actions #42

Updated by Andreas Kohlbecker almost 4 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 30 to 100

implemented and tested locally, on edit-test and edit-int.

Actions #43

Updated by Andreas Kohlbecker almost 4 years ago

  • Related to deleted (feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files)
Actions #44

Updated by Andreas Kohlbecker almost 4 years ago

  • Has duplicate feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files added
Actions #45

Updated by Andreas Kohlbecker over 3 years ago

  • Related to deleted (bug #6287: Rolling cdmserver.log file only contains entries from startup phase)
Actions #46

Updated by Andreas Kohlbecker over 3 years ago

  • Has duplicate bug #6287: Rolling cdmserver.log file only contains entries from startup phase added
Actions #47

Updated by Andreas Kohlbecker over 3 years ago

  • Related to bug #9236: stopping an instance breaks the centralized logging added
Actions

Also available in: Atom PDF