Make DSpace logs rotate using org.apache.log4j.DailyRollingFileAppender

[pmather]

Logs produced by VTechWorks DSpace rotate log files via the application
using the log4j framework. The logging is defined via log4j properties
files. Currently, different log files are rotated using a daily file
rotation logger. The logs for "dspace.log", "checker.log", and
"cocoon.log" are rotated using org.dspace.app.util.DailyFileAppender
(defined in /dspace/config/log4j.properties). Solr logs are rotated
using org.apache.log4j.DailyRollingFileAppender (defined in
/dspace/config/log4j-solr.properties). Finally, "handle-plugin.log" is
using org.apache.log4j.DailyRollingFileAppender (defined in
/dspace/config/log4j-handle-plugin.properties)

Although both logging methods superficially achieve the same ends
(rotating a main log file on a daily basis) they differ subtly in
precise details. With org.apache.log4j.DailyRollingFileAppender, when
the base file is rotated it is renamed with a suffix indicating the date
its contents represent. The base file is then truncated to zero size
and continues to be the current active log file. With
org.dspace.app.util.DailyFileAppender, the base file never exists.
Logging is always done to a base file name with a date suffix. When a
daily rotation period rolls around, a new log file is created with the
new daily date suffix.

The main practical difference between the two schemes is that with
org.apache.log4j.DailyRollingFileAppender the base log file (e.g.,
"solr.log") is always "today's logs" whereas with
org.dspace.app.util.DailyFileAppender the fixed concept of "today's
logs" does not exist. (You need to know today's date and then use that
to find the appropriate file.)

The VT Minimum Security Standards
(https://it.vt.edu/content/dam/it_vt_edu/policies/Minimum-Security-Standards.pdf)
mandate the following for medium and high risk servers: "Forward logs to
a remote log server. Use of the university's centralized log server is
recommended." We currently do this for our servers and use filebeat to
ship the logs. With org.apache.log4j.DailyRollingFileAppender, we can
get filebeat to monitor and ship logs from a single file. When
searching log entries in Kibana, only a single log file name need be
searched over time. With org.dspace.app.util.DailyFileAppender, we have
to have filebeat monitor and ship logs from all files matching the base
file + date suffix pattern. When searching these in Kibana, many
different files must be searched. It's easier to construct dashboards
and queries in the org.apache.log4j.DailyRollingFileAppender case.

Given all the above, it would be preferable if the log files currently
rotated using org.dspace.app.util.DailyFileAppender could be switched to
rotate via org.apache.log4j.DailyRollingFileAppender (or some other
rotation than maintains a fixed base log file).

[alawvt]

@pmather, it seems like we have the dspace and solr log rotation working by other means. Shall I close this issue?

[pmather]

This issue concerns logs, not stats, so the recent sharding of stats in the Solr cores would not be a reason to close this issue in my opinion.

It may be that DSpace log rotation has changed in DSpace 7, so this issue may be superseded by events. However, it would be nice if @keithgee could take a look as to whether the current logging class could be substituted for the one that Solr is using.

A perhaps more important issue that arose after I filed this issue is the log4j vulnerability. Although we did confirm our current DSpace is not vulnerable, we need to address the reason why it is not vulnerable...

[alawvt]

Thanks, @pmather, our agenda for the DevOps meeting next week is pretty full but we'll keep this on the backlog.