bug #7085
cdmserver logfiles getting too big, use RollingFileAppender
100%
Description
cdmserver logfiles getting too big, use DailyRollingFileAppender
Keeping 7 daily logs should be sufficient since all logs are also preserved in the backupserver
Also delete all /var/log/cdmserver/out.log files once this is released.
see https://github.com/GoogleCloudPlatform/appengine-java-vm-runtime/issues/231
Configuring the logging consistently for the cdm-server and for all instances is complicated by the fact that the instances are running separately from the cdm-server in an own thread with own classloader for each instance. Therefore the root logger configured for the cdm-server is not being used by the instances.
TODO
- Setup the logging for the cdm-server so that the server and all instances are using the same DailyRollingFileAppender and logfile.
- remove FileAppenders configured in /opt/cdmserver/.cdmLibrary/log4j.properties of edit-test, edit-prod, edit-int, edit-demo-*, edit-jobber-1
- consider using log4j2, see #7821
- logfile paths in the cdm-server/src/main/resources/log4j.properties must not be hardcoded and platform-specific, like
log4j.appender.FILE.File= /var/log/cdmserver/cdmserver_log.out
- fixed by cdm-server|bb431d84
- On Configuring Jetty logging: https://www.eclipse.org/jetty/documentation/9.4.29.v20200521/configuring-logging.html
Solution
Jetty can be configured to use the Managed Diagnostic Context of either log4j or logback to centralize the logging from multiple webapps in to a central log file. This technique is explained in https://www.eclipse.org/jetty/documentation/9.4.29.v20200521/example-logging-logback-centralized.html for a jetty server which uses a deployment manager. In our situation of an embedded jetty which manages the cdm-webapp instance directly the configuration need to be a bit different, see eu.etaxonomy.cdm.server.logging.LoggingConfigurator
for details.
Consequences of this change:
- The cdm-server start option
logdir
has now been replanced by the system propertycdm.logDir
so all server installations need to checked and modified-logdir=... ==> -Dcdm.logDir=file://...
- adapt the docker configuration
- adapt the debian package?
- The system property
-Dlog4j.configuration=file://..
can no longer be used and needs to be replaced by e.g.:-Dlogback.configurationFile=file:/app/logback.xml
Related issues
Associated revisions
ref #7085 Use DailyRollingFileAppender to generate one log per day
ref #7085 Use DailyRollingFileAppender (in code)
ref #7085 Adjust log path
ref #7085 Set logging to daily rollover
ref #7085 Adjust log path
ref #7085 Remove whitespace in log file time stamp
ref #7085 changing logging initialization for more consistent logging configuration for bootloader and instances - experimental!
ref #7085 using logback MDC for centralized logging into a rolling file appender and to system out
ref #7085 adapting ide launch configs
ref #7085 removing obsolete code from LoggingConfigurer
ref #7085 removing obsolete attribute which is no longer used in the LoggingConfigurer
ref #7085 adapting docker files and debian package to changes
ref #7085 adapting debian package - 2
ref #7085 better logging configuration and improved init script
ref #7085 adding the InstanceManager explicitly to fix broken JSP compilation
History
#1 Updated by Andreas Kohlbecker over 3 years ago
Also delete all /var/log/cdmserver/out.log files once this is released
#2 Updated by Andreas Kohlbecker over 3 years ago
- Priority changed from New to Highest
#3 Updated by Andreas Kohlbecker over 3 years ago
- Description updated (diff)
#4 Updated by Andreas Müller over 3 years ago
- Target version changed from Release 4.12 to Release 4.13
#5 Updated by Andreas Müller about 3 years ago
- Target version changed from Release 4.13 to Release 4.14
#6 Updated by Andreas Müller about 3 years ago
- Target version changed from Release 4.14 to Release 5.0
#7 Updated by Andreas Kohlbecker almost 3 years ago
- Target version changed from Release 5.0 to Release 5.1
#8 Updated by Andreas Kohlbecker almost 3 years ago
- Status changed from New to In Progress
#9 Updated by Andreas Kohlbecker almost 3 years ago
- Description updated (diff)
#10 Updated by Andreas Kohlbecker almost 3 years ago
- Target version changed from Release 5.1 to Release 5.2
#11 Updated by Andreas Kohlbecker over 2 years ago
- Target version changed from Release 5.2 to Release 5.3
#12 Updated by Andreas Kohlbecker over 2 years ago
- Status changed from In Progress to New
- Target version changed from Release 5.3 to Release 5.4
#13 Updated by Andreas Kohlbecker over 2 years ago
- Assignee changed from Andreas Kohlbecker to Patrick Plitzner
#14 Updated by Patrick Plitzner over 2 years ago
- Related to bug #6287: Rolling cdmserver.log file only contains entries from startup phase added
#15 Updated by Patrick Plitzner over 2 years ago
- Related to feature request #7819: Log4j.properties file is not properly packaged by the cdmserver install script added
#16 Updated by Patrick Plitzner over 2 years ago
- Status changed from New to Resolved
- Assignee changed from Patrick Plitzner to Andreas Kohlbecker
- % Done changed from 0 to 50
The logging is now configured to daily logging i.e. every day a new log file is generated.
The change was done manually in the /opt/cdmserver/.cdmLibrary/log4j.properties
file. This should be handled by the install script of the cdm server -> #7819
Remaining issues is deleting the old log files after a specified time period -> #7820
#17 Updated by Patrick Plitzner over 2 years ago
- Related to feature request #7820: Add cron job for deleting cdmserver log files after a specified time period added
#18 Updated by Patrick Plitzner over 2 years ago
- Related to feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files added
#19 Updated by Andreas Kohlbecker over 2 years ago
- Status changed from Resolved to In Progress
- Assignee changed from Andreas Kohlbecker to Patrick Plitzner
The rolled daily log files have a whitespace character, please remove the whitespace from the filenames.
-rw-r--r-- 1 cdm adm 409569 Oct 19 12:18 cdmserver_log.out -rw-r--r-- 1 cdm adm 262768 Oct 12 09:43 cdmserver_log.out. 2018-10-12 -rw-r--r-- 1 cdm adm 143578 Oct 13 05:07 cdmserver_log.out. 2018-10-13 -rw-r--r-- 1 cdm adm 150123 Oct 14 09:40 cdmserver_log.out. 2018-10-14 -rw-r--r-- 1 cdm adm 549825 Oct 15 19:11 cdmserver_log.out. 2018-10-15 -rw-r--r-- 1 cdm adm 343656 Oct 16 18:27 cdmserver_log.out. 2018-10-16 -rw-r--r-- 1 cdm adm 599338 Oct 17 20:14 cdmserver_log.out. 2018-10-17 -rw-r--r-- 1 cdm adm 623044 Oct 18 23:11 cdmserver_log.out. 2018-10-18
#20 Updated by Andreas Kohlbecker over 2 years ago
i found a second issue:
The rolling daily file appender only works if the custom loj4j.properties are defined in the /etc/default/cdmserver
# # user custom loj4j.properties # JAVA_OPTIONS="$JAVA_OPTIONS -Dlog4j.configuration=file:///opt/cdmserver/.cdmLibrary/log4j.properties"
The rolling daily file appender should be used in default configuration.
#21 Updated by Andreas Kohlbecker over 2 years ago
third issue:
unfortunately you committed all changes to the master branch:
* 469b75d - Fri, 12 Oct 2018 11:26:09 +0200 (7 days ago) (origin/master, origin/HEAD) | ref #7085 Adjust log path - Patrick Plitzner * 0465659 - Fri, 12 Oct 2018 10:47:59 +0200 (7 days ago) | ref #7085 Set logging to daily rollover - Patrick Plitzner * 72c3d33 - Fri, 12 Oct 2018 09:18:02 +0200 (7 days ago) | ref #7085 Adjust log path - Patrick Plitzner * b78e167 - Thu, 11 Oct 2018 17:13:40 +0200 (8 days ago) | Revert "ref #7085 Use DailyRollingFileAppender (in code)" - Patrick Plitzner * 02ef1e4 - Thu, 11 Oct 2018 16:57:36 +0200 (8 days ago) | ref #7085 Use DailyRollingFileAppender (in code) - Patrick Plitzner * 99c113d - Thu, 11 Oct 2018 16:43:44 +0200 (8 days ago) | ref #7085 Use DailyRollingFileAppender to generate one log per day - Patrick Plitzner * c172618 - Mon, 17 Sep 2018 11:28:56 +0200 (5 weeks ago) (tag: 5.3.0) |\ jenkins merging release branch into master (strategy: theirs) - jenkins | * 9efc161 - Mon, 17 Sep 2018 11:28:15 +0200 (5 weeks ago) | | updating poms for branch'release/5.3.0' with non-snapshot versions - jenkins * | 83415cd - Fri, 17 Aug 2018 16:08:00 +0200 (9 weeks ago) (tag: 5.2.0)
This will break the next release since git-flow can not deal with intermediate commits to the master branch, we will run into merge conflicts. Your changes being committed to the master branch could also be the reason why the daily rolling fileappender is not working in those servers which are not configured to use the custom log4j.properties
Please fix the git repo. The clone at git hub must also be fixed. You must whipe out your changes completely from master. A git revert
might not be enough in this case.
BTW: Did you replace the old log4j.properties
on the edit-test server or did exchange the old appender by the daily rolling fileappender?
#22 Updated by Patrick Plitzner over 2 years ago
- Assignee changed from Patrick Plitzner to Andreas Kohlbecker
#23 Updated by Andreas Kohlbecker over 2 years ago
- Description updated (diff)
#24 Updated by Andreas Kohlbecker over 2 years ago
- % Done changed from 50 to 20
#25 Updated by Andreas Kohlbecker over 2 years ago
we can use the Log4J config provided as
-Dlog4j.configuration=file:///opt/cdmserver/.cdmLibrary/log4j.properties
in the /etc/default/cdmserver
as preliminary solution. But this is not reliable since it does not work out of the box when the cdm-server is installed newly.
Interestingly with -Dlog4j.configuration
the log4j.properties is used in the bootloader and in the instances. In contrast to that consistent configuration seems to be hard to achieve out of the Bootloader, see cdm-server|bb431d84
#26 Updated by Andreas Kohlbecker over 2 years ago
for the coming release I will add the following configuration to edit-int, edit-test and edit-production:
## ## NOTICE !!!! ## no need to configure a FileAppender here ## use eu.etaxonomy.cdm.remote.config.LoggingConfigurer instead ## #-------------------------------------------------------------------------------------------------- # as interim solution in order to have a daily rolling file appender we configer it here # see https://dev.e-taxonomy.eu/redmine/issues/7085 log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender log4j.appender.FILE.File= ${LOGGING_FOLDER}/cdmserver.log log4j.appender.FILE.ImmediateFlush=true log4j.appender.FILE.Threshold=debug log4j.appender.FILE.Append=true log4j.appender.FILE.DatePattern='.'yyyy-MM-dd log4j.appender.FILE.layout=org.apache.log4j.PatternLayout log4j.appender.FILE.layout.ConversionPattern=%d %p [%c] - %m%n #--------------------------------------------------------------------------------------------------
This needs to be removed once the DailyRollingFileAppender is being applied out of the cdm-server.
#27 Updated by Andreas Kohlbecker over 2 years ago
- Description updated (diff)
#28 Updated by Andreas Kohlbecker over 2 years ago
- Description updated (diff)
#29 Updated by Andreas Kohlbecker over 2 years ago
- Target version changed from Release 5.4 to Release 5.5
#30 Updated by Andreas Kohlbecker about 2 years ago
- Target version changed from Release 5.5 to Release 5.6
#31 Updated by Andreas Kohlbecker about 2 years ago
- Target version changed from Release 5.6 to Reviewed Next Major Release
#32 Updated by Andreas Kohlbecker about 1 year ago
- Target version changed from Reviewed Next Major Release to Release 5.13
#33 Updated by Andreas Kohlbecker about 1 year ago
- Target version changed from Release 5.13 to Release 5.14
#34 Updated by Andreas Müller about 1 year ago
- Target version changed from Release 5.14 to Release 5.15
#35 Updated by Andreas Kohlbecker 11 months ago
- Description updated (diff)
#36 Updated by Andreas Kohlbecker 11 months ago
- Description updated (diff)
#37 Updated by Andreas Kohlbecker 11 months ago
- Description updated (diff)
#38 Updated by Andreas Kohlbecker 10 months ago
- Description updated (diff)
- % Done changed from 20 to 30
#39 Updated by Andreas Kohlbecker 10 months ago
- Description updated (diff)
#40 Updated by Andreas Kohlbecker 10 months ago
- Subject changed from cdmserver logfiles getting too big, use DailyRollingFileAppender to cdmserver logfiles getting too big, use RollingFileAppender
#41 Updated by Andreas Kohlbecker 10 months ago
- Related to feature request #6249: better logging strategy added
#42 Updated by Andreas Kohlbecker 10 months ago
- Status changed from In Progress to Closed
- % Done changed from 30 to 100
implemented and tested locally, on edit-test and edit-int.
#43 Updated by Andreas Kohlbecker 10 months ago
- Related to deleted (feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files)
#44 Updated by Andreas Kohlbecker 10 months ago
- Duplicated by feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files added
#45 Updated by Andreas Kohlbecker 7 months ago
- Related to deleted (bug #6287: Rolling cdmserver.log file only contains entries from startup phase)
#46 Updated by Andreas Kohlbecker 7 months ago
- Duplicated by bug #6287: Rolling cdmserver.log file only contains entries from startup phase added
#47 Updated by Andreas Kohlbecker 7 months ago
- Related to bug #9236: stopping an instance breaks the centralized logging added