Quarkus App fail to restart: SmallRyeMetricsRecorder cause ArrayIndexOutOfBoundsException in CgroupV2Subsystem #27248

rmanibus · 2022-08-11T14:33:25Z

Describe the bug

I have a quarkus application with quarkus 2.10.1.Final running on a RHEEL vm. Everything is working fine when I start the app for the first time,

however when I stop it I am not able to restart it. I have to reboot the VM to make it start again.
The stacktrace indicates that the failure occurs in SmallRyeMetricsRecorder.
I am not sure I will be able to produce a reproducer for this one.

Aug 11, 2022 4:20:09 PM org.hibernate.Version
INFO: HHH000412: Hibernate ORM core version %s
Aug 11, 2022 4:20:09 PM org.hibernate.annotations.common.Version
INFO: HCANN000001: Hibernate Commons Annotations {%1$s}
Aug 11, 2022 4:20:09 PM org.hibernate.dialect.Dialect
INFO: HHH000400: Using dialect: %s
Aug 11, 2022 4:20:10 PM org.hibernate.validator.internal.util.Version
INFO: HV000001: Hibernate Validator %s
Aug 11, 2022 4:20:11 PM org.jboss.threads
INFO: JBoss Threads version %s
LogManager error of type WRITE_FAILURE: The delayed handler's queue was overrun and log record(s) were lost (Did you forget to configure logging?): 
	 - Hibernate Types: [DEBUG]
	 - io.netty.channel.DefaultChannelId: [DEBUG]
	 - io.netty.util.internal.CleanerJava9: [DEBUG]
	 - io.netty.util.internal.PlatformDependent: [DEBUG]
	 - io.netty.util.internal.PlatformDependent0: [DEBUG]
	 - io.netty.util.internal.logging.InternalLoggerFactory: [DEBUG]
	 - io.quarkus.arc.impl: [DEBUG]
	 - io.quarkus.arc.runtime.ArcRecorder: [DEBUG]
	 - io.quarkus.arc.runtime.BeanContainerImpl: [DEBUG]
	 - io.quarkus.cache.runtime.caffeine.CaffeineCacheBuildRecorder: [DEBUG]
	 - io.quarkus.cache.runtime.caffeine.CaffeineCacheImpl: [TRACE]
	 - io.quarkus.flyway.runtime.FlywayRecorder: [DEBUG]
	 - io.quarkus.hibernate.orm: [DEBUG]
	 - io.quarkus.hibernate.orm.runtime.service.FlatClassLoaderService: [TRACE]
	 - io.quarkus.hibernate.orm.runtime.service.QuarkusSimplifiedIdentifierGeneratorFactory: [DEBUG]
	 - io.quarkus.resteasy.runtime: [DEBUG]
	 - io.smallrye.metrics: [TRACE, DEBUG]
	 - io.undertow.session: [DEBUG]
	 - org.hibernate.boot.internal.BootstrapContextImpl: [DEBUG]
	 - org.hibernate.boot.internal.ClassLoaderAccessImpl: [DEBUG]
	 - org.hibernate.boot.internal.InFlightMetadataCollectorImpl: [DEBUG, TRACE]
	 - org.hibernate.boot.model.relational.Namespace: [DEBUG]
	 - org.hibernate.boot.model.source.internal.annotations.AnnotationMetadataSourceProcessorImpl: [DEBUG]
	 - org.hibernate.boot.registry.internal.BootstrapServiceRegistryImpl: [DEBUG]
	 - org.hibernate.boot.registry.selector.internal.StrategySelectorImpl: [TRACE]
	 - org.hibernate.cfg.AbstractPropertyHolder: [DEBUG]
	 - org.hibernate.cfg.AnnotationBinder: [TRACE, DEBUG]
	 - org.hibernate.cfg.BinderHelper: [DEBUG]
	 - org.hibernate.cfg.CollectionPropertyHolder: [DEBUG]
	 - org.hibernate.cfg.CollectionSecondPass: [DEBUG]
	 - org.hibernate.cfg.Ejb3Column: [DEBUG]
	 - org.hibernate.cfg.Environment: [DEBUG]
	 - org.hibernate.cfg.annotations.CollectionBinder: [DEBUG]
	 - org.hibernate.cfg.annotations.EntityBinder: [DEBUG]
	 - org.hibernate.cfg.annotations.PropertyBinder: [TRACE, DEBUG]
	 - org.hibernate.cfg.annotations.ResultsetMappingSecondPass: [DEBUG]
	 - org.hibernate.cfg.annotations.SimpleValueBinder: [DEBUG]
	 - org.hibernate.cfg.annotations.TableBinder: [DEBUG]
	 - org.hibernate.engine.jdbc.env.internal.LobCreatorBuilderImpl: [DEBUG]
	 - org.hibernate.engine.jdbc.env.internal.NormalizingIdentifierHelperImpl: [TRACE]
	 - org.hibernate.jpa.internal.util.LogHelper: [DEBUG]
	 - org.hibernate.mapping.PrimaryKey: [DEBUG]
	 - org.hibernate.quarkus.feature: [DEBUG]
	 - org.hibernate.service.internal.AbstractServiceRegistryImpl: [TRACE]
	 - org.hibernate.type.BasicTypeRegistry: [DEBUG]
	 - org.hibernate.type.EnumType: [DEBUG]
	 - org.hibernate.type.spi.TypeConfiguration$Scope: [DEBUG]
	 - org.hibernate.validator.internal.engine.AbstractConfigurationImpl: [DEBUG]
	 - org.hibernate.validator.internal.engine.ValidatorFactoryConfigurationHelper: [DEBUG]
	 - org.hibernate.validator.messageinterpolation.ResourceBundleMessageInterpolator: [DEBUG]
	 - org.hibernate.validator.resourceloading.PlatformResourceBundleLocator: [DEBUG, TRACE]
	 - org.infinispan.quarkus.hibernate.cache.QuarkusInfinispanRegionFactory: [DEBUG]
	 - org.jboss.logging: [DEBUG]
Aug 11, 2022 4:20:11 PM io.quarkus.runtime.ApplicationLifecycleManager run
ERROR: Failed to start application (with profile prod)
java.lang.ArrayIndexOutOfBoundsException: Index 4 out of bounds for length 1
	at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.initSubsystem(Unknown Source)
	at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.<clinit>(Unknown Source)
	at java.base/jdk.internal.platform.CgroupSubsystemFactory.create(Unknown Source)
	at java.base/jdk.internal.platform.CgroupMetrics.getInstance(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at java.base/jdk.internal.platform.Metrics.systemMetrics(Unknown Source)
	at java.base/jdk.internal.platform.Container.metrics(Unknown Source)
	at jdk.management/com.sun.management.internal.OperatingSystemImpl.<init>(Unknown Source)
	at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl.getOperatingSystemMXBean(Unknown Source)
	at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl$3.nameToMBeanMap(Unknown Source)
	at java.management/sun.management.spi.PlatformMBeanProvider$PlatformComponent.getMBeans(Unknown Source)
	at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(Unknown Source)
	at java.management/java.lang.management.ManagementFactory.getOperatingSystemMXBean(Unknown Source)
	at io.quarkus.smallrye.metrics.runtime.SmallRyeMetricsRecorder.baseOperatingSystemMetrics(SmallRyeMetricsRecorder.java:291)
	at io.quarkus.smallrye.metrics.runtime.SmallRyeMetricsRecorder.registerBaseMetrics(SmallRyeMetricsRecorder.java:105)
	at io.quarkus.deployment.steps.SmallRyeMetricsProcessor$registerBaseAndVendorMetrics349750083.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.SmallRyeMetricsProcessor$registerBaseAndVendorMetrics349750083.deploy(Unknown Source)
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:103)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:67)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:41)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:120)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

No response

Output of `uname -a` or `ver`

Linux [REDACTED] 3.10.0-1160.31.1.el7.x86_64 #1 SMP Thu Jun 10 13:32:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Output of `java -version`

openjdk version "15.0.2" 2021-01-19 OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.2+7) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.2+7, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.10.1.Final

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

No response

Additional information

No response

The text was updated successfully, but these errors were encountered:

rmanibus · 2022-08-17T14:19:34Z

/cc @ebullient , since this looks related to quarkus-metrics, maybe you have an idea ?

ebullient · 2022-08-17T15:20:34Z

Does this happen if you use the micrometer extension instead?

The micrometer extension also retrieves the OperatingSystemMxBean, but has a few different stop-gaps in place:

https://github.com/micrometer-metrics/micrometer/blob/8339d57bef8689beb8d7a18b429a166f6595f2af/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/system/ProcessorMetrics.java#L81

(Note the Micrometer extension is the recommended metrics extension as of Quarkus 1.9).

rmanibus · 2022-08-17T15:34:15Z

Thanks @ebullient , I will give it a try, but since the application is quite large I need to take some time to migrate all the metrics.
Also I was a fan of the @Gauge annotation which does not have an equivalent with micrometer !

gsmet · 2022-08-17T15:38:34Z

@rmanibus I would try to upgrade the JDK first. It looks like a JDK issue to me.

gsmet · 2022-08-17T15:39:17Z

And even if Micrometer is now the way to go, SmallRye Metrics is still expected to work.

rmanibus · 2022-08-17T15:42:13Z

thanks @gsmet , unfortunately I'm stuck to 15 for now due to snowflakedb/snowflake-jdbc#533
I will definitely upgrade once this is resolved. Do you think rollbacking to 11 might make it work ?
I have a lot of text blocks all around so I don't think it would even be possible for me

gsmet · 2022-08-17T15:45:59Z

I have no idea. But I would try the latest 11 and the latest 17 and see if it fixes this particular issue.

ebullient · 2022-08-17T16:42:39Z

While the micrometer extension handles most of the MP Metrics API, you are right that it doesn't deal with some gauge usage patterns..

@gsmet: I was curious if the way micrometer deals with mxbeans would make a difference or not for the jdk it is on. It has a few more guards around mbean inspection than the quarkus smallrye metrics extension does, which is why I asked if the other extension could be tried.

geoand · 2022-08-23T11:13:31Z

I have no idea. But I would try the latest 11 and the latest 17 and see if it fixes this particular issue.

Very much this. I wouldn't stay on JDK 15 for any longer than I had too :)

quarkus-bot · 2022-08-23T11:55:23Z

/cc @ebullient, @jmartisk

rmanibus · 2022-09-01T16:15:52Z

I do not face this issue anymore after upgrading to 2.12.0.Final. I'm not sure what was causing this though.

geoand · 2022-09-01T16:24:17Z

Thanks for the update.

Let's close the issue, but feel free to comment if you encounter it in the future

rmanibus added the kind/bug Something isn't working label Aug 11, 2022

quarkus-bot bot added the triage/needs-triage label Aug 11, 2022

rmanibus changed the title ~~Quarkus App fail to restart: ArrayIndexOutOfBoundsException in CgroupV2Subsystem~~ Quarkus App fail to restart: SmallRyeMetricsRecorder cause ArrayIndexOutOfBoundsException in CgroupV2Subsystem Aug 11, 2022

geoand added area/metrics and removed triage/needs-triage labels Aug 23, 2022

geoand closed this as not planned Won't fix, can't repro, duplicate, stale Sep 1, 2022

geoand added area/arc Issue related to ARC (dependency injection) triage/out-of-date This issue/PR is no longer valid or relevant labels Sep 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Quarkus App fail to restart: SmallRyeMetricsRecorder cause ArrayIndexOutOfBoundsException in CgroupV2Subsystem #27248

Quarkus App fail to restart: SmallRyeMetricsRecorder cause ArrayIndexOutOfBoundsException in CgroupV2Subsystem #27248

rmanibus commented Aug 11, 2022 •

edited

Loading

rmanibus commented Aug 17, 2022 •

edited

Loading

ebullient commented Aug 17, 2022

rmanibus commented Aug 17, 2022 •

edited

Loading

gsmet commented Aug 17, 2022

gsmet commented Aug 17, 2022

rmanibus commented Aug 17, 2022 •

edited

Loading

gsmet commented Aug 17, 2022

ebullient commented Aug 17, 2022

geoand commented Aug 23, 2022

quarkus-bot bot commented Aug 23, 2022

rmanibus commented Sep 1, 2022

geoand commented Sep 1, 2022

Quarkus App fail to restart: SmallRyeMetricsRecorder cause ArrayIndexOutOfBoundsException in CgroupV2Subsystem #27248

Quarkus App fail to restart: SmallRyeMetricsRecorder cause ArrayIndexOutOfBoundsException in CgroupV2Subsystem #27248

Comments

rmanibus commented Aug 11, 2022 • edited Loading

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

GraalVM version (if different from Java)

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

rmanibus commented Aug 17, 2022 • edited Loading

ebullient commented Aug 17, 2022

rmanibus commented Aug 17, 2022 • edited Loading

gsmet commented Aug 17, 2022

gsmet commented Aug 17, 2022

rmanibus commented Aug 17, 2022 • edited Loading

gsmet commented Aug 17, 2022

ebullient commented Aug 17, 2022

geoand commented Aug 23, 2022

quarkus-bot bot commented Aug 23, 2022

rmanibus commented Sep 1, 2022

geoand commented Sep 1, 2022

rmanibus commented Aug 11, 2022 •

edited

Loading

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

rmanibus commented Aug 17, 2022 •

edited

Loading

rmanibus commented Aug 17, 2022 •

edited

Loading

rmanibus commented Aug 17, 2022 •

edited

Loading