MicroProfile 4.x: OutOfMemoryError: GC overhead limit exceeded when starting/stopping lots of apps

[brideck]

When running the CDI TCK with microProfile-4.1 added to the Liberty server feature list, the server eventually runs out of memory, throwing java.lang.OutOfMemoryError: GC overhead limit exceeded from all of its threads. This behavior does not normally occur with this TCK, and is only present when the microProfile feature is included in the server config.

The CDI TCK consists of hundreds of applications that are added to the server, started, and then stopped as the tests execute. Analysis of a heap dump collected when the server was actively throwing the OOM errors has indicated that OLSmallRyeConfigProviderResolver is failing to release resources when the test applications are stopped. There is a known memory leak in SmallRye (smallrye/smallrye-config#277) and an attempt has been made to work around it in Open Liberty (#14550). The current suspicion is that the GLOBAL_CONFIG_APPLICATION_NAME case is not being handled correctly, so those applications are never released.

[brideck]

Access to the heap dumps is available upon request.

[tevans78]

The root cause of this issue is the MP Config spec's assumption that there is a 1:1 relationship between Applications and ClassLoaders.... eclipse/microprofile-config#546

We're fighting a losing battle until the spec is fixed ... and that would have to be a major version change.