Project

General

Profile

bug #7085

cdmserver logfiles getting too big, use DailyRollingFileAppender

Added by Andreas Kohlbecker over 1 year ago. Updated about 1 month ago.

Status:
In Progress
Priority:
Highest
Category:
cdmserver
Start date:
11/23/2017
Due date:
% Done:

20%

Severity:
major
Found in Version:

Description

cdmserver logfiles getting too big, use DailyRollingFileAppender

Keeping 7 daily logs should be sufficient since all logs are also preserved in the backupserver

Also delete all /var/log/cdmserver/out.log files once this is released.

see https://github.com/GoogleCloudPlatform/appengine-java-vm-runtime/issues/231

Configuring the logging consistently for the cdm-server and for all instances is complicated by the fact that the instances are running separately from the cdm-server in an own thread with own classloader for each instance. Therefore the root logger configured for the cdm-server is not being used by the instances.

TODO

  • Setup the logging for the cdm-server so that the server and all instances are using the same DailyRollingFileAppender and logfile.
  • remove FileAppenders configured in /opt/cdmserver/.cdmLibrary/log4j.properties of edit-test, edit-prod, edit-int, edit-demo-*, edit-jobber-1
  • consider using log4j2, see #7821 *logfile paths in the cdm-server/src/main/resources/log4j.properties must not be hardcoded an platform-specific, like log4j.appender.FILE.File= /var/log/cdmserver/cdmserver_log.out - fixed by cdm-server|bb431d84

Related issues

Related to Edit - bug #6287: Rolling cdmserver.log file only contains entries from startup phase In Progress 12/19/2016
Related to Edit - feature request #7819: Log4j.properties file is not properly packaged by the cdmserver install script New 10/12/2018
Related to Edit - feature request #7820: Add cron job for deleting cdmserver log files after a specified time period New 10/15/2018 10/15/2018
Related to Edit - feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files New 10/12/2018

Associated revisions

Revision 3775575b (diff)
Added by Patrick Plitzner 5 months ago

ref #7085 Use DailyRollingFileAppender to generate one log per day

Revision cf681ac9 (diff)
Added by Patrick Plitzner 5 months ago

ref #7085 Use DailyRollingFileAppender (in code)

Revision 045673ca (diff)
Added by Patrick Plitzner 5 months ago

ref #7085 Adjust log path

Revision 3aae2857 (diff)
Added by Patrick Plitzner 5 months ago

ref #7085 Set logging to daily rollover

Revision ca5c272c (diff)
Added by Patrick Plitzner 5 months ago

ref #7085 Adjust log path

Revision dc6c8364 (diff)
Added by Patrick Plitzner 5 months ago

ref #7085 Remove whitespace in log file time stamp

Revision bb431d84 (diff)
Added by Andreas Kohlbecker 5 months ago

ref #7085 changing logging initialization for more consistent logging configuration for bootloader and instances - experimental!

History

#1 Updated by Andreas Kohlbecker over 1 year ago

Also delete all /var/log/cdmserver/out.log files once this is released

#2 Updated by Andreas Kohlbecker over 1 year ago

  • Priority changed from New to Highest

#3 Updated by Andreas Kohlbecker over 1 year ago

  • Description updated (diff)

#4 Updated by Andreas Müller over 1 year ago

  • Target version changed from Release 4.12 to Release 4.13

#5 Updated by Andreas Müller about 1 year ago

  • Target version changed from Release 4.13 to Release 4.14

#6 Updated by Andreas Müller about 1 year ago

  • Target version changed from Release 4.14 to Release 5.0

#7 Updated by Andreas Kohlbecker 10 months ago

  • Target version changed from Release 5.0 to Release 5.1

#8 Updated by Andreas Kohlbecker 9 months ago

  • Status changed from New to In Progress

#9 Updated by Andreas Kohlbecker 9 months ago

  • Description updated (diff)

#10 Updated by Andreas Kohlbecker 9 months ago

  • Target version changed from Release 5.1 to Release 5.2

#11 Updated by Andreas Kohlbecker 7 months ago

  • Target version changed from Release 5.2 to Release 5.3

#12 Updated by Andreas Kohlbecker 6 months ago

  • Status changed from In Progress to New
  • Target version changed from Release 5.3 to Release 5.4

#13 Updated by Andreas Kohlbecker 6 months ago

  • Assignee changed from Andreas Kohlbecker to Patrick Plitzner

#14 Updated by Patrick Plitzner 5 months ago

  • Related to bug #6287: Rolling cdmserver.log file only contains entries from startup phase added

#15 Updated by Patrick Plitzner 5 months ago

  • Related to feature request #7819: Log4j.properties file is not properly packaged by the cdmserver install script added

#16 Updated by Patrick Plitzner 5 months ago

  • Status changed from New to Resolved
  • Assignee changed from Patrick Plitzner to Andreas Kohlbecker
  • % Done changed from 0 to 50

The logging is now configured to daily logging i.e. every day a new log file is generated.

The change was done manually in the /opt/cdmserver/.cdmLibrary/log4j.properties file. This should be handled by the install script of the cdm server -> #7819

Remaining issues is deleting the old log files after a specified time period -> #7820

#17 Updated by Patrick Plitzner 5 months ago

  • Related to feature request #7820: Add cron job for deleting cdmserver log files after a specified time period added

#18 Updated by Patrick Plitzner 5 months ago

  • Related to feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files added

#19 Updated by Andreas Kohlbecker 5 months ago

  • Status changed from Resolved to In Progress
  • Assignee changed from Andreas Kohlbecker to Patrick Plitzner

The rolled daily log files have a whitespace character, please remove the whitespace from the filenames.

-rw-r--r-- 1 cdm adm 409569 Oct 19 12:18 cdmserver_log.out
-rw-r--r-- 1 cdm adm 262768 Oct 12 09:43 cdmserver_log.out. 2018-10-12
-rw-r--r-- 1 cdm adm 143578 Oct 13 05:07 cdmserver_log.out. 2018-10-13
-rw-r--r-- 1 cdm adm 150123 Oct 14 09:40 cdmserver_log.out. 2018-10-14
-rw-r--r-- 1 cdm adm 549825 Oct 15 19:11 cdmserver_log.out. 2018-10-15
-rw-r--r-- 1 cdm adm 343656 Oct 16 18:27 cdmserver_log.out. 2018-10-16
-rw-r--r-- 1 cdm adm 599338 Oct 17 20:14 cdmserver_log.out. 2018-10-17
-rw-r--r-- 1 cdm adm 623044 Oct 18 23:11 cdmserver_log.out. 2018-10-18

#20 Updated by Andreas Kohlbecker 5 months ago

i found a second issue:

The rolling daily file appender only works if the custom loj4j.properties are defined in the /etc/default/cdmserver

#
# user custom loj4j.properties
#
JAVA_OPTIONS="$JAVA_OPTIONS -Dlog4j.configuration=file:///opt/cdmserver/.cdmLibrary/log4j.properties"

The rolling daily file appender should be used in default configuration.

#21 Updated by Andreas Kohlbecker 5 months ago

third issue:

unfortunately you committed all changes to the master branch:

* 469b75d - Fri, 12 Oct 2018 11:26:09 +0200 (7 days ago) (origin/master, origin/HEAD)
|   ref #7085 Adjust log path - Patrick Plitzner
* 0465659 - Fri, 12 Oct 2018 10:47:59 +0200 (7 days ago)
|   ref #7085 Set logging to daily rollover - Patrick Plitzner
* 72c3d33 - Fri, 12 Oct 2018 09:18:02 +0200 (7 days ago)
|   ref #7085 Adjust log path - Patrick Plitzner
* b78e167 - Thu, 11 Oct 2018 17:13:40 +0200 (8 days ago)
|   Revert "ref #7085 Use DailyRollingFileAppender (in code)" - Patrick Plitzner
* 02ef1e4 - Thu, 11 Oct 2018 16:57:36 +0200 (8 days ago)
|   ref #7085 Use DailyRollingFileAppender (in code) - Patrick Plitzner
* 99c113d - Thu, 11 Oct 2018 16:43:44 +0200 (8 days ago)
|   ref #7085 Use DailyRollingFileAppender to generate one log per day - Patrick Plitzner
*   c172618 - Mon, 17 Sep 2018 11:28:56 +0200 (5 weeks ago) (tag: 5.3.0)
|\    jenkins merging release branch into master (strategy: theirs) - jenkins
| * 9efc161 - Mon, 17 Sep 2018 11:28:15 +0200 (5 weeks ago)
| |   updating poms for branch'release/5.3.0' with non-snapshot versions - jenkins
* |   83415cd - Fri, 17 Aug 2018 16:08:00 +0200 (9 weeks ago) (tag: 5.2.0)

This will break the next release since git-flow can not deal with intermediate commits to the master branch, we will run into merge conflicts. Your changes being committed to the master branch could also be the reason why the daily rolling fileappender is not working in those servers which are not configured to use the custom log4j.properties

Please fix the git repo. The clone at git hub must also be fixed. You must whipe out your changes completely from master. A git revert might not be enough in this case.

BTW: Did you replace the old log4j.properties on the edit-test server or did exchange the old appender by the daily rolling fileappender?

#22 Updated by Patrick Plitzner 5 months ago

  • Assignee changed from Patrick Plitzner to Andreas Kohlbecker

#23 Updated by Andreas Kohlbecker 5 months ago

  • Description updated (diff)

#24 Updated by Andreas Kohlbecker 5 months ago

  • % Done changed from 50 to 20

#25 Updated by Andreas Kohlbecker 5 months ago

we can use the Log4J config provided as

-Dlog4j.configuration=file:///opt/cdmserver/.cdmLibrary/log4j.properties

in the /etc/default/cdmserver as preliminary solution. But this is not reliable since it does not work out of the box when the cdm-server is installed newly.

Interestingly with -Dlog4j.configuration the log4j.properties is used in the bootloader and in the instances. In contrast to that consistent configuration seems to be hard to achieve out of the Bootloader, see cdm-server|bb431d84

#26 Updated by Andreas Kohlbecker 5 months ago

for the coming release I will add the following configuration to edit-int, edit-test and edit-production:

##
## NOTICE !!!!
## no need to configure a FileAppender here
## use eu.etaxonomy.cdm.remote.config.LoggingConfigurer instead
##
#--------------------------------------------------------------------------------------------------
# as interim solution in order to have a daily rolling file appender we configer it here
# see https://dev.e-taxonomy.eu/redmine/issues/7085
log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender
log4j.appender.FILE.File= ${LOGGING_FOLDER}/cdmserver.log
log4j.appender.FILE.ImmediateFlush=true
log4j.appender.FILE.Threshold=debug
log4j.appender.FILE.Append=true
log4j.appender.FILE.DatePattern='.'yyyy-MM-dd
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.ConversionPattern=%d %p [%c] - %m%n
#--------------------------------------------------------------------------------------------------

This needs to be removed once the DailyRollingFileAppender is being applied out of the cdm-server.

#27 Updated by Andreas Kohlbecker 5 months ago

  • Description updated (diff)

#28 Updated by Andreas Kohlbecker 5 months ago

  • Description updated (diff)

#29 Updated by Andreas Kohlbecker 5 months ago

  • Target version changed from Release 5.4 to Release 5.5

#30 Updated by Andreas Kohlbecker about 2 months ago

  • Target version changed from Release 5.5 to Release 5.6

#31 Updated by Andreas Kohlbecker about 1 month ago

  • Target version changed from Release 5.6 to Reviewed Next Major Release

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 40 MB)