Support scanning a directory for log4j2 configuration files

[mattrpav]

New property log4j2.configurationDirectory would work similar to the comma separated list in log4j2.configurationFile where multiple configuration files can be used for an application.

This behavior would allow dynamic loading of configuration files, instead of requiring the name of each configuration file ahead of time, which is the case with log4j2.configurationFile.

1. If property log4j2.configurationDirectory is set, recursively scan the directory for matching configuration files.
2. Limit directory depth to something reasonable (16 or so by default)

[vy]

That is a very interesting idea Matt, thanks for bringing it to our attention. 💯 Do you have a particular use case that you want to solve with this mechanism?

[mattrpav]

@vy the use case is to enable a runtime to use a fixed configuration value and then allow for various deployments use cases to modify 0..n number of log4j2 configuration files without having to change the runtime's configured value for log4j2.configurationFile.

Use Case 1: Log config per-environment

1. Build a Java-based container and configure log4j2.configurationDirectory=/opt/app/conf/log.d
2. Deploy the application to 3 different environments: dev, test and prod
3. On a per-env basis deploy the corresponding log4j2 configuration: log4j2-dev.xml, log4j2-test.xml, log4j2-prod.xml

Use Case 2: Dynamic multi-tenant application

1. Build a Java-based container that supports multi-tenant applications (aka monolithic/modulith app) and configure log4j2.configurationDirectory=/opt/app/conf/log.d
2. Deploy the application to support 3 different tenants: order, quote and billing
3. On a per-tenant basis deploy the corresponding log4j2 configuration: log4j2-order.xml, log4j2-quote.xml, log4j2-billing.xml

[ppkarwasz]

Hi @mattrpav,

Your proposal certainly addresses several limitations in the way Log4j Core determines the appropriate configuration file, although I am not certain how these limitations should be addressed. From a maintainer perspective I would prefer not to have too many configuration knobs that I need to care about. If some feature applies to a very small number of environments, I would prefer to externalize it.

Let's analyze the current situation and how we can improve it.

Current status

Currently Log4j Core has already a mechanism that looks for files named log4j2<contextName>.xml and falls back to log4j2.xml if nothing is found (see configuration file location). This mechanism is useless for Java SE applications (the <contextName> is random), but can be used with some success in "use case 2", if you integrate the runtime environment with Log4j Core. An example of such an integration is provided by the log4j-jakarta-web module (see the documentation of the log4jContextName and log4jConfiguration Servlet context parameters).

Admittedly this mechanism is limited:

1. As I stated before <contextName> is initialized to a random value in a Java SE application. Even in the favorable case of a Servlet Environment <contextName> does not necessarily give a valid file name, e.g. it can be the unescaped Servlet context path /my-app. However, you can always change it at runtime with a call like:

   Configurator.initialize("-dev", null).reconfigure();

2. The search path for the log4j2<contextName>.xml file is limited to the classpath. You could work around this limitation by setting the log4j2.configurationFile property to something like log4j2-${spring:profiles.active[0]}.xml. This would enlarge the search path to the classpath or current working directory.

Possible solutions

Resolution of relative filenames

As explained above, the search path for Log4j Core configuration files differs, depending on the value of log4j2.configurationFile:

• If log4j2.configurationFile has a value that is not an absolute URI or file path, Log4j Core tries to find it in the current working directory or the classpath. This might resemble the way Spring Boot does it, but in my experience the cwd is a value the should not be relied upon, especially in a production environment, where applications are not started from the CLI.
• if log4j2.configurationFile is not set, Log4j Core looks exclusively in the classpath.

We should probably:

1. Introduce a log4j2.configurationArea or log4j2.configurationDirectory configuration property as you propose, to provide a list of directories to be searched for configuration files. Runtime environments could set it to the value that makes most sense. For example Eclipse Equinox could use ${osgi.configuration.area}, while Tomcat could use ${catalina.base}/conf/:${catalina.home}/conf/.
2. Fix the automatic configuration mechanism to also look for files in log4j2.configurationDirectory.

Note: We don't necessarily need a new configuration property. We could extend the definition of log4j2.configurationFile and use the same convention as Spring Boot (cf. External Application Properties: if a value in log4j2.configurationFile ends in /, it is treated as a directory and Log4j appends log4j2-<contextName>.<extension> to it.

The default value of log4j2.configurationFile will change from null to file:./,classpath:/ to provide backward compatibility.

Determination of <contextName>

To fully support the use cases you propose, we need some support from the runtime environment.

Use Case 1

I assume we are running in a Spring Boot environment. This environment is quite specific since Log4j Core is configured twice:

• when the JVM starts it is configured without any access to Spring's environment. So a logger context is created, but we can not give it any significant name.
• when the Spring Boot Environment is ready, the logger context is reconfigured programmatically by Spring Boot. Therefore we might expect Spring Boot to choose the appropriate configuration file among log4j2-dev.xml, log4j2-prod.xml. This can already be done by setting Spring Boot's logger.config property to a different value, depending on the environment.

Use Case 2

We might support this use case by improving the way ContextSelector determine the default <contextName>.

Currently the ClassLoaderContextSelector uses ClassLoader.hashCode() as context name, which is pretty much useless. We could improve that by using ClassLoader.getName() available since Java 9. The runtime environment could create classloaders named order, quote or billing to allow users to configure those services separately.

[mattrpav]

@ppkarwasz 'log4j2.configurationDirectory' configuration property that supports comma separated sounds great!

Note: This is not for Spring Boot, but approach would be the similar-- initial log4j2 context that is then programmatically reconfigured.

[rgoers]

I actually prefer Piotr's suggestion of using a trailing '/' to indicate it is a directory.

[mattrpav]

I'm coming up to speed on log4j2 internals, so this is may be a rough riff--

re: ContextNameSelector -- perhaps a package prefix selector?

Multi-tenant app could deploy:

log4j2-com.company.order.[xml|json|yaml]
log4j2-com.company.quote.[xml|json|yaml]
log4j2-com.company.billing.[xml|json|yaml]

Then:

LogManger.getLogger(com.company.order.status.OrderStatusService.class)

..could resolve to the 'com.company.order' package prefix

This has the added benefit of being able to fallback to a simple default context (log4j2.xml) on the desktop during development and unit testing.

Classloader approach is a nice option, but can be incomplete since high-density apps may share a classloader to support shared JSON/XML model class marshallers.

Perhaps a configuration to name the context selector class?

[ppkarwasz]

Perhaps a configuration to name the context selector class?

There is already log4j2.contextSelector configuration property that does that.

re: ContextNameSelector -- perhaps a package prefix selector?

While you can create a ContextSelector based on the FQCN of the caller, that wouldn't necessarily be a good way to split loggers between logger context.

Ideally you don't only want to split the loggers of the classes in the order, quote and billing packages into 3 separate logger contexts, you would like also to split the loggers of the common libraries into 3 separate logger contexts. Of course this requires those libraries to be carefully written with log separation in mind: they need to use exclusively instance logger fields instead of the more popular static logger fields.

Note: In practice separating the loggers of common libraries of a runtime environment is almost impossible. A more practical approach is to add some context data (see ThreadContext for example) to each log entry, log everything to one big file and split it afterwards.

[vy]

Build a Java-based container ...

@mattrpav, are you referring to OS containers (Docker, Podman, etc.) or Java servlet containers? (I have the impression that you're referring to the former, while @ppkarwasz is talking about the latter. Hence, I feel like we are not on the same page.)

We could extend the definition of log4j2.configurationFile and ... if a value in log4j2.configurationFile ends in /, it is treated as a directory

@ppkarwasz, I liked this approach.

I had two other alternatives in mind:

1. Support wildcards: e.g., -Dlog4j2.configurationFile=/opt/app/conf/log.d/* – This matches the way -cp JVM option works and the JEP 458: Launch Multi-File Source-Code Programs features freshly delivered in Java 22.
2. Support glob patterns: e.g., log4j2.configurationFile=/opt/app/conf/log.d/**/* – This would be pretty powerful, but requires an in-house glob implementation, which sort of renders this option an overkill.

[mattrpav]

Build a Java-based container ...

@mattrpav, are you referring to OS containers (Docker, Podman, etc.) or Java servlet containers? (I have the impression that you're referring to the former, while @ppkarwasz is talking about the latter. Hence, I feel like we are not on the same page.)

What specifically does not seem to align to you?

I had two other alternatives in mind:

1. **Support wildcards:** e.g., `-Dlog4j2.configurationFile=/opt/app/conf/log.d/*` – This matches the way `-cp` JVM option works and the [JEP 458: Launch Multi-File Source-Code Programs](https://openjdk.org/jeps/458) features freshly delivered in Java 22.

A reasonable set of default filename globs should suffice without having to draw on a JDK 22 feature (keep in mind the next LTS after JDK 21 is JDK 25) -- log4j2*.json, log4j2*.yaml, log4j2*.xml, log4j2*.properties, etc.

2. **Support glob patterns:** e.g., `log4j2.configurationFile=/opt/app/conf/log.d/**/*` – This would be pretty powerful, but requires an in-house glob implementation, which sort of renders this option an overkill.

If the default behavior is defined to scan sub-directories, this should not be needed.

Defaults:

1. Follow symlinks
2. Scan sub-directories
3. Limit sub-directory depth to a reasonably high value (ie. 16 or 32). This will provide flexibility, be performant, and provide an escape hatch for problematic scenarios (symlink loops, etc)

[vy]

@mattrpav, what do you mean by a Java-based container? A Docker container? Op ma 26 aug 2024 om 15:40 schreef Matt Pavlovich ***@***.***>

…

Build a Java-based container ... @mattrpav <https://github.com/mattrpav>, are you referring to OS containers (Docker, Podman, etc.) or Java servlet containers? (I have the impression that you're referring to the former, while @ppkarwasz <https://github.com/ppkarwasz> is talking about the latter. Hence, I feel like we are not on the same page.) What specifically does not seem to align to you? I had two other alternatives in mind: 1. **Support wildcards:** e.g., `-Dlog4j2.configurationFile=/opt/app/conf/log.d/*` – This matches the way `-cp` JVM option works and the [JEP 458: Launch Multi-File Source-Code Programs](https://openjdk.org/jeps/458) features freshly delivered in Java 22. A reasonable set of default filename globs should suffice without having to draw on a JDK 22 feature -- log4j2*.json, log4j2*.yaml, log4j2*.xml, log4j2*.properties, etc. 2. **Support glob patterns:** e.g., `log4j2.configurationFile=/opt/app/conf/log.d/**/*` – This would be pretty powerful, but requires an in-house glob implementation, which sort of renders this option an overkill. If the default behavior is defined to scan sub-directories, this should not be needed. Defaults: 1. Follow symlinks 2. Scan sub-directories 3. Limit sub-directory depth to a reasonably high value (ie. 16 or 32). This will provide flexibility, be performant, and provide an escape hatch for problematic scenarios (symlink loops, etc) — Reply to this email directly, view it on GitHub <#2852 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAARTSOCHCR7MENZ4VRS7KLZTMV5NAVCNFSM6AAAAABMVTSIHWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJQGI2DKNJXGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ppkarwasz]

If the default behavior is defined to scan sub-directories, this should not be needed.

Defaults:

1. Follow symlinks
2. Scan sub-directories
3. Limit sub-directory depth to a reasonably high value (ie. 16 or 32). This will provide flexibility, be performant, and provide an escape hatch for problematic scenarios (symlink loops, etc)

@mattrpav, what purpose would the recursion serve? What happens if more than one log4j2* file is found?

[mattrpav]

@mattrpav, what purpose would the recursion serve? What happens if more than one log4j2* file is found?

Allows developers/devops to deploy Kubernetes config maps (one file per-subdirectory) without using a special type of volume (aka projected volume).

Standard one ConfigMap per-subdirectory

/opt/app/conf/log.d/order/log4j2-order.xml
/opt/app/conf/log.d/quote/log4j2-quote.xml

Projected volume would allow:

/opt/app/conf/log.d/log4j2-order.xml
/opt/app/conf/log.d/log4j2-quote.xml

[vy]

Allows developers/devops to deploy Kubernetes config maps (one file per-subdirectory) without using a special type of volume (aka projected volume).

@mattrpav, thanks for clarifying that you were referring to OS containers – as I indicated earlier, @ppkarwasz thought you were referring to Servlet containers.

I am still keen on supporting configuration directories you proposed, but to better understand what is falling short in the current knobs: You can attach your Log4j configuration files in whichever way you want (bundle them in JARs, mount them using ConfigMaps, regular/projected volumes, etc.) and use the LOG4J_CONFIGURATION_FILE environment variable to point to the effective one at runtime. Why is this not a viable option for you?

[ppkarwasz]

Note also that the log4j2.configurationFile variable can contain property substitution expressions. If you export the name of your container as CONTAINER_NAME, you can use:

/opt/app/conf/log4j.d/${env:CONTAINER_NAME}/log4j2-${env:CONTAINER_NAME}.xml