bug #7085
closedcdmserver logfiles getting too big, use RollingFileAppender
100%
Description
cdmserver logfiles getting too big, use DailyRollingFileAppender
Keeping 7 daily logs should be sufficient since all logs are also preserved in the backupserver
Also delete all /var/log/cdmserver/out.log files once this is released.
see https://github.com/GoogleCloudPlatform/appengine-java-vm-runtime/issues/231
Configuring the logging consistently for the cdm-server and for all instances is complicated by the fact that the instances are running separately from the cdm-server in an own thread with own classloader for each instance. Therefore the root logger configured for the cdm-server is not being used by the instances.
TODO
- Setup the logging for the cdm-server so that the server and all instances are using the same DailyRollingFileAppender and logfile.
- remove FileAppenders configured in /opt/cdmserver/.cdmLibrary/log4j.properties of edit-test, edit-prod, edit-int, edit-demo-*, edit-jobber-1
- consider using log4j2, see #7821
- logfile paths in the cdm-server/src/main/resources/log4j.properties must not be hardcoded and platform-specific, like
log4j.appender.FILE.File= /var/log/cdmserver/cdmserver_log.out
- fixed by cdm-server|bb431d84
- On Configuring Jetty logging: https://www.eclipse.org/jetty/documentation/9.4.29.v20200521/configuring-logging.html
Solution
Jetty can be configured to use the Managed Diagnostic Context of either log4j or logback to centralize the logging from multiple webapps in to a central log file. This technique is explained in https://www.eclipse.org/jetty/documentation/9.4.29.v20200521/example-logging-logback-centralized.html for a jetty server which uses a deployment manager. In our situation of an embedded jetty which manages the cdm-webapp instance directly the configuration need to be a bit different, see eu.etaxonomy.cdm.server.logging.LoggingConfigurator
for details.
Consequences of this change:
- The cdm-server start option
logdir
has now been replanced by the system propertycdm.logDir
so all server installations need to checked and modified-logdir=... ==> -Dcdm.logDir=file://...
- adapt the docker configuration
- adapt the debian package?
- The system property
-Dlog4j.configuration=file://..
can no longer be used and needs to be replaced by e.g.:-Dlogback.configurationFile=file:/app/logback.xml
Related issues
Updated by Andreas Kohlbecker about 6 years ago
Also delete all /var/log/cdmserver/out.log files once this is released
Updated by Andreas Kohlbecker about 6 years ago
- Priority changed from New to Highest
Updated by Andreas Müller about 6 years ago
- Target version changed from Release 4.12 to Release 4.13
Updated by Andreas Müller almost 6 years ago
- Target version changed from Release 4.13 to Release 4.14
Updated by Andreas Müller almost 6 years ago
- Target version changed from Release 4.14 to Release 5.0
Updated by Andreas Kohlbecker over 5 years ago
- Target version changed from Release 5.0 to Release 5.1
Updated by Andreas Kohlbecker over 5 years ago
- Status changed from New to In Progress
Updated by Andreas Kohlbecker over 5 years ago
- Target version changed from Release 5.1 to Release 5.2
Updated by Andreas Kohlbecker over 5 years ago
- Target version changed from Release 5.2 to Release 5.3
Updated by Andreas Kohlbecker about 5 years ago
- Status changed from In Progress to New
- Target version changed from Release 5.3 to Release 5.4
Updated by Andreas Kohlbecker about 5 years ago
- Assignee changed from Andreas Kohlbecker to Patrick Plitzner
Updated by Patrick Plitzner about 5 years ago
- Related to bug #6287: Rolling cdmserver.log file only contains entries from startup phase added
Updated by Patrick Plitzner about 5 years ago
- Related to feature request #7819: Log4j.properties file is not properly packaged by the cdmserver install script added
Updated by Patrick Plitzner about 5 years ago
- Status changed from New to Resolved
- Assignee changed from Patrick Plitzner to Andreas Kohlbecker
- % Done changed from 0 to 50
The logging is now configured to daily logging i.e. every day a new log file is generated.
The change was done manually in the /opt/cdmserver/.cdmLibrary/log4j.properties
file. This should be handled by the install script of the cdm server -> #7819
Remaining issues is deleting the old log files after a specified time period -> #7820
Updated by Patrick Plitzner about 5 years ago
- Related to feature request #7820: Add cron job for deleting cdmserver log files after a specified time period added
Updated by Patrick Plitzner about 5 years ago
- Related to feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files added
Updated by Andreas Kohlbecker about 5 years ago
- Status changed from Resolved to In Progress
- Assignee changed from Andreas Kohlbecker to Patrick Plitzner
The rolled daily log files have a whitespace character, please remove the whitespace from the filenames.
-rw-r--r-- 1 cdm adm 409569 Oct 19 12:18 cdmserver_log.out -rw-r--r-- 1 cdm adm 262768 Oct 12 09:43 cdmserver_log.out. 2018-10-12 -rw-r--r-- 1 cdm adm 143578 Oct 13 05:07 cdmserver_log.out. 2018-10-13 -rw-r--r-- 1 cdm adm 150123 Oct 14 09:40 cdmserver_log.out. 2018-10-14 -rw-r--r-- 1 cdm adm 549825 Oct 15 19:11 cdmserver_log.out. 2018-10-15 -rw-r--r-- 1 cdm adm 343656 Oct 16 18:27 cdmserver_log.out. 2018-10-16 -rw-r--r-- 1 cdm adm 599338 Oct 17 20:14 cdmserver_log.out. 2018-10-17 -rw-r--r-- 1 cdm adm 623044 Oct 18 23:11 cdmserver_log.out. 2018-10-18
Updated by Andreas Kohlbecker about 5 years ago
i found a second issue:
The rolling daily file appender only works if the custom loj4j.properties are defined in the /etc/default/cdmserver
# # user custom loj4j.properties # JAVA_OPTIONS="$JAVA_OPTIONS -Dlog4j.configuration=file:///opt/cdmserver/.cdmLibrary/log4j.properties"
The rolling daily file appender should be used in default configuration.
Updated by Andreas Kohlbecker about 5 years ago
third issue:
unfortunately you committed all changes to the master branch:
* 469b75d - Fri, 12 Oct 2018 11:26:09 +0200 (7 days ago) (origin/master, origin/HEAD) | ref #7085 Adjust log path - Patrick Plitzner * 0465659 - Fri, 12 Oct 2018 10:47:59 +0200 (7 days ago) | ref #7085 Set logging to daily rollover - Patrick Plitzner * 72c3d33 - Fri, 12 Oct 2018 09:18:02 +0200 (7 days ago) | ref #7085 Adjust log path - Patrick Plitzner * b78e167 - Thu, 11 Oct 2018 17:13:40 +0200 (8 days ago) | Revert "ref #7085 Use DailyRollingFileAppender (in code)" - Patrick Plitzner * 02ef1e4 - Thu, 11 Oct 2018 16:57:36 +0200 (8 days ago) | ref #7085 Use DailyRollingFileAppender (in code) - Patrick Plitzner * 99c113d - Thu, 11 Oct 2018 16:43:44 +0200 (8 days ago) | ref #7085 Use DailyRollingFileAppender to generate one log per day - Patrick Plitzner * c172618 - Mon, 17 Sep 2018 11:28:56 +0200 (5 weeks ago) (tag: 5.3.0) |\ jenkins merging release branch into master (strategy: theirs) - jenkins | * 9efc161 - Mon, 17 Sep 2018 11:28:15 +0200 (5 weeks ago) | | updating poms for branch'release/5.3.0' with non-snapshot versions - jenkins * | 83415cd - Fri, 17 Aug 2018 16:08:00 +0200 (9 weeks ago) (tag: 5.2.0)
This will break the next release since git-flow can not deal with intermediate commits to the master branch, we will run into merge conflicts. Your changes being committed to the master branch could also be the reason why the daily rolling fileappender is not working in those servers which are not configured to use the custom log4j.properties
Please fix the git repo. The clone at git hub must also be fixed. You must whipe out your changes completely from master. A git revert
might not be enough in this case.
BTW: Did you replace the old log4j.properties
on the edit-test server or did exchange the old appender by the daily rolling fileappender?
Updated by Patrick Plitzner about 5 years ago
- Assignee changed from Patrick Plitzner to Andreas Kohlbecker
Updated by Andreas Kohlbecker about 5 years ago
we can use the Log4J config provided as
-Dlog4j.configuration=file:///opt/cdmserver/.cdmLibrary/log4j.properties
in the /etc/default/cdmserver
as preliminary solution. But this is not reliable since it does not work out of the box when the cdm-server is installed newly.
Interestingly with -Dlog4j.configuration
the log4j.properties is used in the bootloader and in the instances. In contrast to that consistent configuration seems to be hard to achieve out of the Bootloader, see cdm-server|bb431d84
Updated by Andreas Kohlbecker about 5 years ago
for the coming release I will add the following configuration to edit-int, edit-test and edit-production:
## ## NOTICE !!!! ## no need to configure a FileAppender here ## use eu.etaxonomy.cdm.remote.config.LoggingConfigurer instead ## #-------------------------------------------------------------------------------------------------- # as interim solution in order to have a daily rolling file appender we configer it here # see https://dev.e-taxonomy.eu/redmine/issues/7085 log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender log4j.appender.FILE.File= ${LOGGING_FOLDER}/cdmserver.log log4j.appender.FILE.ImmediateFlush=true log4j.appender.FILE.Threshold=debug log4j.appender.FILE.Append=true log4j.appender.FILE.DatePattern='.'yyyy-MM-dd log4j.appender.FILE.layout=org.apache.log4j.PatternLayout log4j.appender.FILE.layout.ConversionPattern=%d %p [%c] - %m%n #--------------------------------------------------------------------------------------------------
This needs to be removed once the DailyRollingFileAppender is being applied out of the cdm-server.
Updated by Andreas Kohlbecker about 5 years ago
- Target version changed from Release 5.4 to Release 5.5
Updated by Andreas Kohlbecker almost 5 years ago
- Target version changed from Release 5.5 to Release 5.6
Updated by Andreas Kohlbecker almost 5 years ago
- Target version changed from Release 5.6 to Reviewed Next Major Release
Updated by Andreas Kohlbecker almost 4 years ago
- Target version changed from Reviewed Next Major Release to Release 5.13
Updated by Andreas Kohlbecker over 3 years ago
- Target version changed from Release 5.13 to Release 5.14
Updated by Andreas Müller over 3 years ago
- Target version changed from Release 5.14 to Release 5.15
Updated by Andreas Kohlbecker over 3 years ago
- Description updated (diff)
- % Done changed from 20 to 30
Updated by Andreas Kohlbecker over 3 years ago
- Subject changed from cdmserver logfiles getting too big, use DailyRollingFileAppender to cdmserver logfiles getting too big, use RollingFileAppender
Updated by Andreas Kohlbecker over 3 years ago
- Related to feature request #6249: better logging strategy added
Updated by Andreas Kohlbecker over 3 years ago
- Status changed from In Progress to Closed
- % Done changed from 30 to 100
implemented and tested locally, on edit-test and edit-int.
Updated by Andreas Kohlbecker over 3 years ago
- Related to deleted (feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files)
Updated by Andreas Kohlbecker over 3 years ago
- Has duplicate feature request #7821: Use log4j2 for cdmserver logging to avoid having a separate cron job for deleting log files added
Updated by Andreas Kohlbecker about 3 years ago
- Related to deleted (bug #6287: Rolling cdmserver.log file only contains entries from startup phase)
Updated by Andreas Kohlbecker about 3 years ago
- Has duplicate bug #6287: Rolling cdmserver.log file only contains entries from startup phase added
Updated by Andreas Kohlbecker about 3 years ago
- Related to bug #9236: stopping an instance breaks the centralized logging added