Project

General

Profile

bug #7085

cdmserver logfiles getting too big, use RollingFileAppender

Added by Andreas Kohlbecker almost 3 years ago. Updated 4 months ago.

Status:
Closed
Priority:
Highest
Category:
cdmserver
Target version:
Start date:
11/23/2017
Due date:
% Done:

100%

Severity:
major
Found in Version:

Description

cdmserver logfiles getting too big, use DailyRollingFileAppender

Keeping 7 daily logs should be sufficient since all logs are also preserved in the backupserver

Also delete all /var/log/cdmserver/out.log files once this is released.

see https://github.com/GoogleCloudPlatform/appengine-java-vm-runtime/issues/231

Configuring the logging consistently for the cdm-server and for all instances is complicated by the fact that the instances are running separately from the cdm-server in an own thread with own classloader for each instance. Therefore the root logger configured for the cdm-server is not being used by the instances.

TODO

  • Setup the logging for the cdm-server so that the server and all instances are using the same DailyRollingFileAppender and logfile.
  • remove FileAppenders configured in /opt/cdmserver/.cdmLibrary/log4j.properties of edit-test, edit-prod, edit-int, edit-demo-*, edit-jobber-1
  • consider using log4j2, see #7821
  • logfile paths in the cdm-server/src/main/resources/log4j.properties must not be hardcoded and platform-specific, like log4j.appender.FILE.File= /var/log/cdmserver/cdmserver_log.out - fixed by cdm-server|bb431d84


Solution

Jetty can be configured to use the Managed Diagnostic Context of either log4j or logback to centralize the logging from multiple webapps in to a central log file. This technique is explained in https://www.eclipse.org/jetty/documentation/9.4.29.v20200521/example-logging-logback-centralized.html for a jetty server which uses a deployment manager. In our situation of an embedded jetty which manages the cdm-webapp instance directly the configuration need to be a bit different, see eu.etaxonomy.cdm.server.logging.LoggingConfigurator for details.

Consequences of this change:

  • The cdm-server start option logdir has now been replanced by the system property cdm.logDir so all server installations need to checked and modified -logdir=... ==> -Dcdm.logDir=file://...
    • adapt the docker configuration
    • adapt the debian package?
  • The system property -Dlog4j.configuration=file://.. can no longer be used and needs to be replaced by e.g.: -Dlogback.configurationFile=file:/app/logback.xml

Related issues

Related to Edit - feature request #7819: Log4j.properties file is not properly packaged by the cdmserver install script New 10/12/2018
Related to Edit - feature request #7820: Add cron job for deleting cdmserver log files after a specified time period Rejected 10/15/2018 10/15/2018
Related to Edit - feature request #6249: better logging strategy Closed 12/06/2016
Related to Edit - bug #9236: stopping an instance breaks the centralized logging New 10/01/2020
Duplicated by Edit - feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files Duplicate 10/12/2018
Duplicated by Edit - bug #6287: Rolling cdmserver.log file only contains entries from startup phase Duplicate 12/19/2016

Associated revisions

Revision 3775575b (diff)
Added by Patrick Plitzner about 2 years ago

ref #7085 Use DailyRollingFileAppender to generate one log per day

Revision cf681ac9 (diff)
Added by Patrick Plitzner about 2 years ago

ref #7085 Use DailyRollingFileAppender (in code)

Revision 045673ca (diff)
Added by Patrick Plitzner about 2 years ago

ref #7085 Adjust log path

Revision 3aae2857 (diff)
Added by Patrick Plitzner about 2 years ago

ref #7085 Set logging to daily rollover

Revision ca5c272c (diff)
Added by Patrick Plitzner about 2 years ago

ref #7085 Adjust log path

Revision dc6c8364 (diff)
Added by Patrick Plitzner about 2 years ago

ref #7085 Remove whitespace in log file time stamp

Revision bb431d84 (diff)
Added by Andreas Kohlbecker about 2 years ago

ref #7085 changing logging initialization for more consistent logging configuration for bootloader and instances - experimental!

Revision 72ddab96 (diff)
Added by Andreas Kohlbecker 5 months ago

ref #7085 using logback MDC for centralized logging into a rolling file appender and to system out

Revision 9526e8ec (diff)
Added by Andreas Kohlbecker 5 months ago

ref #7085 adapting ide launch configs

Revision 20884bdb (diff)
Added by Andreas Kohlbecker 5 months ago

ref #7085 removing obsolete code from LoggingConfigurer

Revision e1944271 (diff)
Added by Andreas Kohlbecker 5 months ago

ref #7085 removing obsolete attribute which is no longer used in the LoggingConfigurer

Revision 0c5f74e6 (diff)
Added by Andreas Kohlbecker 5 months ago

ref #7085 adapting docker files and debian package to changes

Revision 8881fca1 (diff)
Added by Andreas Kohlbecker 5 months ago

ref #7085 adapting debian package - 2

Revision e895826c (diff)
Added by Andreas Kohlbecker 5 months ago

ref #7085 better logging configuration and improved init script

Revision 1f9ea18c (diff)
Added by Andreas Kohlbecker 4 months ago

ref #7085 adding the InstanceManager explicitly to fix broken JSP compilation

History

#1 Updated by Andreas Kohlbecker almost 3 years ago

Also delete all /var/log/cdmserver/out.log files once this is released

#2 Updated by Andreas Kohlbecker almost 3 years ago

  • Priority changed from New to Highest

#3 Updated by Andreas Kohlbecker almost 3 years ago

  • Description updated (diff)

#4 Updated by Andreas Müller almost 3 years ago

  • Target version changed from Release 4.12 to Release 4.13

#5 Updated by Andreas Müller over 2 years ago

  • Target version changed from Release 4.13 to Release 4.14

#6 Updated by Andreas Müller over 2 years ago

  • Target version changed from Release 4.14 to Release 5.0

#7 Updated by Andreas Kohlbecker over 2 years ago

  • Target version changed from Release 5.0 to Release 5.1

#8 Updated by Andreas Kohlbecker over 2 years ago

  • Status changed from New to In Progress

#9 Updated by Andreas Kohlbecker over 2 years ago

  • Description updated (diff)

#10 Updated by Andreas Kohlbecker over 2 years ago

  • Target version changed from Release 5.1 to Release 5.2

#11 Updated by Andreas Kohlbecker about 2 years ago

  • Target version changed from Release 5.2 to Release 5.3

#12 Updated by Andreas Kohlbecker about 2 years ago

  • Status changed from In Progress to New
  • Target version changed from Release 5.3 to Release 5.4

#13 Updated by Andreas Kohlbecker about 2 years ago

  • Assignee changed from Andreas Kohlbecker to Patrick Plitzner

#14 Updated by Patrick Plitzner about 2 years ago

  • Related to bug #6287: Rolling cdmserver.log file only contains entries from startup phase added

#15 Updated by Patrick Plitzner about 2 years ago

  • Related to feature request #7819: Log4j.properties file is not properly packaged by the cdmserver install script added

#16 Updated by Patrick Plitzner about 2 years ago

  • Status changed from New to Resolved
  • Assignee changed from Patrick Plitzner to Andreas Kohlbecker
  • % Done changed from 0 to 50

The logging is now configured to daily logging i.e. every day a new log file is generated.

The change was done manually in the /opt/cdmserver/.cdmLibrary/log4j.properties file. This should be handled by the install script of the cdm server -> #7819

Remaining issues is deleting the old log files after a specified time period -> #7820

#17 Updated by Patrick Plitzner about 2 years ago

  • Related to feature request #7820: Add cron job for deleting cdmserver log files after a specified time period added

#18 Updated by Patrick Plitzner about 2 years ago

  • Related to feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files added

#19 Updated by Andreas Kohlbecker about 2 years ago

  • Status changed from Resolved to In Progress
  • Assignee changed from Andreas Kohlbecker to Patrick Plitzner

The rolled daily log files have a whitespace character, please remove the whitespace from the filenames.

-rw-r--r-- 1 cdm adm 409569 Oct 19 12:18 cdmserver_log.out
-rw-r--r-- 1 cdm adm 262768 Oct 12 09:43 cdmserver_log.out. 2018-10-12
-rw-r--r-- 1 cdm adm 143578 Oct 13 05:07 cdmserver_log.out. 2018-10-13
-rw-r--r-- 1 cdm adm 150123 Oct 14 09:40 cdmserver_log.out. 2018-10-14
-rw-r--r-- 1 cdm adm 549825 Oct 15 19:11 cdmserver_log.out. 2018-10-15
-rw-r--r-- 1 cdm adm 343656 Oct 16 18:27 cdmserver_log.out. 2018-10-16
-rw-r--r-- 1 cdm adm 599338 Oct 17 20:14 cdmserver_log.out. 2018-10-17
-rw-r--r-- 1 cdm adm 623044 Oct 18 23:11 cdmserver_log.out. 2018-10-18

#20 Updated by Andreas Kohlbecker about 2 years ago

i found a second issue:

The rolling daily file appender only works if the custom loj4j.properties are defined in the /etc/default/cdmserver

#
# user custom loj4j.properties
#
JAVA_OPTIONS="$JAVA_OPTIONS -Dlog4j.configuration=file:///opt/cdmserver/.cdmLibrary/log4j.properties"

The rolling daily file appender should be used in default configuration.

#21 Updated by Andreas Kohlbecker about 2 years ago

third issue:

unfortunately you committed all changes to the master branch:

* 469b75d - Fri, 12 Oct 2018 11:26:09 +0200 (7 days ago) (origin/master, origin/HEAD)
|   ref #7085 Adjust log path - Patrick Plitzner
* 0465659 - Fri, 12 Oct 2018 10:47:59 +0200 (7 days ago)
|   ref #7085 Set logging to daily rollover - Patrick Plitzner
* 72c3d33 - Fri, 12 Oct 2018 09:18:02 +0200 (7 days ago)
|   ref #7085 Adjust log path - Patrick Plitzner
* b78e167 - Thu, 11 Oct 2018 17:13:40 +0200 (8 days ago)
|   Revert "ref #7085 Use DailyRollingFileAppender (in code)" - Patrick Plitzner
* 02ef1e4 - Thu, 11 Oct 2018 16:57:36 +0200 (8 days ago)
|   ref #7085 Use DailyRollingFileAppender (in code) - Patrick Plitzner
* 99c113d - Thu, 11 Oct 2018 16:43:44 +0200 (8 days ago)
|   ref #7085 Use DailyRollingFileAppender to generate one log per day - Patrick Plitzner
*   c172618 - Mon, 17 Sep 2018 11:28:56 +0200 (5 weeks ago) (tag: 5.3.0)
|\    jenkins merging release branch into master (strategy: theirs) - jenkins
| * 9efc161 - Mon, 17 Sep 2018 11:28:15 +0200 (5 weeks ago)
| |   updating poms for branch'release/5.3.0' with non-snapshot versions - jenkins
* |   83415cd - Fri, 17 Aug 2018 16:08:00 +0200 (9 weeks ago) (tag: 5.2.0)

This will break the next release since git-flow can not deal with intermediate commits to the master branch, we will run into merge conflicts. Your changes being committed to the master branch could also be the reason why the daily rolling fileappender is not working in those servers which are not configured to use the custom log4j.properties

Please fix the git repo. The clone at git hub must also be fixed. You must whipe out your changes completely from master. A git revert might not be enough in this case.

BTW: Did you replace the old log4j.properties on the edit-test server or did exchange the old appender by the daily rolling fileappender?

#22 Updated by Patrick Plitzner about 2 years ago

  • Assignee changed from Patrick Plitzner to Andreas Kohlbecker

#23 Updated by Andreas Kohlbecker about 2 years ago

  • Description updated (diff)

#24 Updated by Andreas Kohlbecker about 2 years ago

  • % Done changed from 50 to 20

#25 Updated by Andreas Kohlbecker about 2 years ago

we can use the Log4J config provided as

-Dlog4j.configuration=file:///opt/cdmserver/.cdmLibrary/log4j.properties

in the /etc/default/cdmserver as preliminary solution. But this is not reliable since it does not work out of the box when the cdm-server is installed newly.

Interestingly with -Dlog4j.configuration the log4j.properties is used in the bootloader and in the instances. In contrast to that consistent configuration seems to be hard to achieve out of the Bootloader, see cdm-server|bb431d84

#26 Updated by Andreas Kohlbecker about 2 years ago

for the coming release I will add the following configuration to edit-int, edit-test and edit-production:

##
## NOTICE !!!!
## no need to configure a FileAppender here
## use eu.etaxonomy.cdm.remote.config.LoggingConfigurer instead
##
#--------------------------------------------------------------------------------------------------
# as interim solution in order to have a daily rolling file appender we configer it here
# see https://dev.e-taxonomy.eu/redmine/issues/7085
log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender
log4j.appender.FILE.File= ${LOGGING_FOLDER}/cdmserver.log
log4j.appender.FILE.ImmediateFlush=true
log4j.appender.FILE.Threshold=debug
log4j.appender.FILE.Append=true
log4j.appender.FILE.DatePattern='.'yyyy-MM-dd
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.ConversionPattern=%d %p [%c] - %m%n
#--------------------------------------------------------------------------------------------------

This needs to be removed once the DailyRollingFileAppender is being applied out of the cdm-server.

#27 Updated by Andreas Kohlbecker about 2 years ago

  • Description updated (diff)

#28 Updated by Andreas Kohlbecker about 2 years ago

  • Description updated (diff)

#29 Updated by Andreas Kohlbecker about 2 years ago

  • Target version changed from Release 5.4 to Release 5.5

#30 Updated by Andreas Kohlbecker over 1 year ago

  • Target version changed from Release 5.5 to Release 5.6

#31 Updated by Andreas Kohlbecker over 1 year ago

  • Target version changed from Release 5.6 to Reviewed Next Major Release

#32 Updated by Andreas Kohlbecker 8 months ago

  • Target version changed from Reviewed Next Major Release to Release 5.13

#33 Updated by Andreas Kohlbecker 7 months ago

  • Target version changed from Release 5.13 to Release 5.14

#34 Updated by Andreas Müller 7 months ago

  • Target version changed from Release 5.14 to Release 5.15

#35 Updated by Andreas Kohlbecker 5 months ago

  • Description updated (diff)

#36 Updated by Andreas Kohlbecker 5 months ago

  • Description updated (diff)

#37 Updated by Andreas Kohlbecker 5 months ago

  • Description updated (diff)

#38 Updated by Andreas Kohlbecker 5 months ago

  • Description updated (diff)
  • % Done changed from 20 to 30

#39 Updated by Andreas Kohlbecker 5 months ago

  • Description updated (diff)

#40 Updated by Andreas Kohlbecker 5 months ago

  • Subject changed from cdmserver logfiles getting too big, use DailyRollingFileAppender to cdmserver logfiles getting too big, use RollingFileAppender

#41 Updated by Andreas Kohlbecker 5 months ago

#42 Updated by Andreas Kohlbecker 4 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 30 to 100

implemented and tested locally, on edit-test and edit-int.

#43 Updated by Andreas Kohlbecker 4 months ago

  • Related to deleted (feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files)

#44 Updated by Andreas Kohlbecker 4 months ago

  • Duplicated by feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files added

#45 Updated by Andreas Kohlbecker 24 days ago

  • Related to deleted (bug #6287: Rolling cdmserver.log file only contains entries from startup phase)

#46 Updated by Andreas Kohlbecker 24 days ago

  • Duplicated by bug #6287: Rolling cdmserver.log file only contains entries from startup phase added

#47 Updated by Andreas Kohlbecker 24 days ago

  • Related to bug #9236: stopping an instance breaks the centralized logging added

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 40 MB)