Add GC duration histogram #1854

zeitlinger · 2026-02-06T17:23:39Z

Supercedes #1738

introduces a Prometheus histogram metric jvm_gc_duration_seconds to record JVM garbage collection pause durations. existing metrics provide limited visibility into long GC pauses, making it difficult to detect latency spikes. this change leverages the already registered GC notifications (as used in JvmMemoryPoolAllocationMetrics) to capture pause durations without additional instrumentation.

the histogram uses 0.01, 0.1, 1, 10 buckets and includes labels for gc name, action, and cause, enabling detailed monitoring of both short and long GC pauses. this addresses the lack of visibility highlighted in community discussions such as this one. buckets are also defined according to the opentelemetry semantic conventions spec

Example result:

# HELP jvm_gc_duration_seconds JVM GC pause duration histogram.
# TYPE jvm_gc_duration_seconds histogram
jvm_gc_duration_seconds_bucket{action="end of minor GC",cause="G1 Evacuation Pause",gc="G1 Young Generation",le="0.01"} 806
jvm_gc_duration_seconds_bucket{action="end of minor GC",cause="G1 Evacuation Pause",gc="G1 Young Generation",le="0.1"} 806
jvm_gc_duration_seconds_bucket{action="end of minor GC",cause="G1 Evacuation Pause",gc="G1 Young Generation",le="1.0"} 806
jvm_gc_duration_seconds_bucket{action="end of minor GC",cause="G1 Evacuation Pause",gc="G1 Young Generation",le="10.0"} 806
jvm_gc_duration_seconds_bucket{action="end of minor GC",cause="G1 Evacuation Pause",gc="G1 Young Generation",le="+Inf"} 806
jvm_gc_duration_seconds_count{action="end of minor GC",cause="G1 Evacuation Pause",gc="G1 Young Generation"} 806
jvm_gc_duration_seconds_sum{action="end of minor GC",cause="G1 Evacuation Pause",gc="G1 Young Generation"} 0.7360000000000005

Signed-off-by: gniadeck <77535280+gniadeck@users.noreply.github.com>

…OptIn metric property Signed-off-by: gniadeck <77535280+gniadeck@users.noreply.github.com>

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

…o list Replace the boolean property with a comma-separated list of OTel metric names, giving users fine-grained control over which metrics use OpenTelemetry Semantic Conventions. Use "*" to enable all metrics. Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

jaydeluca · 2026-02-06T21:06:31Z

...-jvm/src/main/java/io/prometheus/metrics/instrumentation/jvm/JvmGarbageCollectorMetrics.java

 public class JvmGarbageCollectorMetrics {

  private static final String JVM_GC_COLLECTION_SECONDS = "jvm_gc_collection_seconds";
+  private static final String JVM_GC_DURATION = "jvm.gc.duration";


should we be consistent with the way the other metric is defined?

Suggested change

private static final String JVM_GC_DURATION = "jvm.gc.duration";

private static final String JVM_GC_DURATION = "jvm_gc_duration";

jaydeluca · 2026-02-06T21:12:35Z

...-jvm/src/main/java/io/prometheus/metrics/instrumentation/jvm/JvmGarbageCollectorMetrics.java

+                if (!GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION.equals(
+                    notification.getType())) {
+                  return;
+                }
+
+                GarbageCollectionNotificationInfo info =
+                    GarbageCollectionNotificationInfo.from(
+                        (CompositeData) notification.getUserData());
+
+                observe(gcDurationHistogram, info);


try/catch here to avoid crashing the JVM is the notification listener throws something

Suggested change

if (!GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION.equals(

notification.getType())) {

return;

}

GarbageCollectionNotificationInfo info =

GarbageCollectionNotificationInfo.from(

(CompositeData) notification.getUserData());

observe(gcDurationHistogram, info);

try {

if (!GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION.equals(

notification.getType())) {

return;

}

GarbageCollectionNotificationInfo info =

GarbageCollectionNotificationInfo.from(

(CompositeData) notification.getUserData());

observe(gcDurationHistogram, info);

} catch (Exception e) {

logger.warning(

"Exception while processing garbage collection notification: " + e.getMessage());

}

jaydeluca · 2026-02-06T21:15:20Z

...-jvm/src/main/java/io/prometheus/metrics/instrumentation/jvm/JvmGarbageCollectorMetrics.java

+            .name(JVM_GC_DURATION)
+            .unit(Unit.SECONDS)
+            .help("Duration of JVM garbage collection actions.")
+            .labelNames("jvm.gc.action", "jvm.gc.name", "jvm.gc.cause")


jvm.gc.cause is still experimental, should we hold off on that one until it stabilizes?

jaydeluca · 2026-02-06T21:46:59Z

...-jvm/src/main/java/io/prometheus/metrics/instrumentation/jvm/JvmGarbageCollectorMetrics.java

+    registerNotificationListener(gcDurationHistogram);
+  }
+
+  private void registerNotificationListener(Histogram gcDurationHistogram) {


taking a look at the OTel implementation, they also track the listeners and do a cleanup, not sure if it makes sense for us to implement something like that, by making JvmMetrics an AutoClosable. Looks like it ends up requiring a lot of changes, not sure if its worth it

Another thing they do is check to see if the class exists first , which I think would be good to do here too

jaydeluca · 2026-02-06T21:47:26Z

.../src/test/java/io/prometheus/metrics/instrumentation/jvm/JvmGarbageCollectorMetricsTest.java

  }
+
+  @Test
+  public void testNonOtelMetricsAbsentWhenUseOtelEnabled() {


Suggested change

public void testNonOtelMetricsAbsentWhenUseOtelEnabled() {

void testNonOtelMetricsAbsentWhenUseOtelEnabled() {

jaydeluca · 2026-02-06T21:47:43Z

.../src/test/java/io/prometheus/metrics/instrumentation/jvm/JvmGarbageCollectorMetricsTest.java

+
+  @Test
+  @SuppressWarnings("rawtypes")
+  public void testGCDurationHistogramLabels() throws Exception {


Suggested change

public void testGCDurationHistogramLabels() throws Exception {

void testGCDurationHistogramLabels() throws Exception {

zeitlinger · 2026-02-09T16:21:08Z

Thinking about this more I created #1861 as an alternative that doesn't duplicate the otel instrumentation work.

Would that work for you @gniadeck ?

gniadeck · 2026-02-09T22:24:24Z

@zeitlinger for my use case this would work fine, although i have some doubts about this implementation as a goal solution - in my PR implementing this metric was just a few lines of code, and now i’d need to import (and in the future update) opentelemetry-sdk, opentelemetry-exporter-prometheus, opentelemetry-runtime-telemetry-java-8 (according to the demo), just to get better GC visibility. i haven’t tested the performance, so i’m not sure what the footprint would be, but it could be a no-go for some low-latency projects that choose prometheus for its speed.

i like this approach from an architectural standpoint tho, nice to see that client_java is so easily extensible with otel work :)

zeitlinger · 2026-02-10T09:56:58Z

now i’d need to import...

Yes, for this particular use case it's only a few lines - but you also get other metrics - and JFR support, etc.

I don't think we want to double maintain this if we can avoid it.

(and in the future update) opentelemetry-sdk, opentelemetry-exporter-prometheus, opentelemetry-runtime-telemetry-java-8 (according to the demo)

we could package the otel sdk and exporter in an "otel support" pom - so you'd only have to import opentelemetry-runtime-telemetry-java8 - would that help?

This would also take care of potential conflicts that might arise because the exporter imports this project as well - although it's not a problem in the demo:

[INFO] --- dependency:3.10.0:tree (default-cli) @ example-otel-jvm-runtime-metrics ---
[INFO] io.prometheus:example-otel-jvm-runtime-metrics:jar:1.5.0-SNAPSHOT
[INFO] +- io.prometheus:prometheus-metrics-core:jar:1.5.0-SNAPSHOT:compile
[INFO] |  +- io.prometheus:prometheus-metrics-model:jar:1.5.0-SNAPSHOT:compile
[INFO] |  +- io.prometheus:prometheus-metrics-config:jar:1.5.0-SNAPSHOT:compile
[INFO] |  \- io.prometheus:prometheus-metrics-tracer-initializer:jar:1.5.0-SNAPSHOT:compile
[INFO] |     +- io.prometheus:prometheus-metrics-tracer-common:jar:1.5.0-SNAPSHOT:compile
[INFO] |     +- io.prometheus:prometheus-metrics-tracer-otel:jar:1.5.0-SNAPSHOT:compile
[INFO] |     \- io.prometheus:prometheus-metrics-tracer-otel-agent:jar:1.5.0-SNAPSHOT:compile
[INFO] +- io.prometheus:prometheus-metrics-exporter-httpserver:jar:1.5.0-SNAPSHOT:compile
[INFO] |  \- io.prometheus:prometheus-metrics-exporter-common:jar:1.5.0-SNAPSHOT:compile
[INFO] |     +- io.prometheus:prometheus-metrics-exposition-textformats:jar:1.5.0-SNAPSHOT:compile
[INFO] |     \- io.prometheus:prometheus-metrics-exposition-formats:jar:1.5.0-SNAPSHOT:runtime <--- here
[INFO] +- io.opentelemetry:opentelemetry-sdk:jar:1.58.0:compile
[INFO] |  +- io.opentelemetry:opentelemetry-api:jar:1.58.0:compile
[INFO] |  |  \- io.opentelemetry:opentelemetry-context:jar:1.58.0:compile
[INFO] |  |     \- io.opentelemetry:opentelemetry-common:jar:1.58.0:compile
[INFO] |  +- io.opentelemetry:opentelemetry-sdk-common:jar:1.58.0:compile
[INFO] |  +- io.opentelemetry:opentelemetry-sdk-trace:jar:1.58.0:compile
[INFO] |  +- io.opentelemetry:opentelemetry-sdk-metrics:jar:1.58.0:compile
[INFO] |  \- io.opentelemetry:opentelemetry-sdk-logs:jar:1.58.0:compile
[INFO] +- io.opentelemetry:opentelemetry-exporter-prometheus:jar:1.58.0-alpha:compile
[INFO] |  +- io.opentelemetry:opentelemetry-exporter-common:jar:1.58.0:runtime
[INFO] |  +- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:jar:1.58.0:runtime
[INFO] |  \- io.prometheus:prometheus-metrics-exposition-formats-no-protobuf:jar:1.3.10:runtime  <--- here

i haven’t tested the performance, so i’m not sure what the footprint would be, but it could be a no-go for some low-latency projects that choose prometheus for its speed.

There's always going to be an overhead in OTel because it's an abstraction - but OTel is basically just delegating to this project back to make the actual processing of the metrics.

The OTel SDK is also very performance sensitive - and currently working in that area.

i like this approach from an architectural standpoint tho, nice to see that client_java is so easily extensible with otel work :)

💯

gniadeck and others added 11 commits February 6, 2026 17:06

Add GC pause duration histogram

ff3ece3

Signed-off-by: gniadeck <77535280+gniadeck@users.noreply.github.com>

align gc label name with opentelemetry spec

b2e2af3

Signed-off-by: gniadeck <77535280+gniadeck@users.noreply.github.com>

simplification, spotless fixes

6423922

Signed-off-by: gniadeck <77535280+gniadeck@users.noreply.github.com>

align naming with opentelemetry semantic conventions

683150b

Signed-off-by: gniadeck <77535280+gniadeck@users.noreply.github.com>

Label test and migration to opentelemetry naming

298717f

Signed-off-by: gniadeck <77535280+gniadeck@users.noreply.github.com>

Add useOtelMetrics to MetricsProperties

c5d1b90

Signed-off-by: gniadeck <77535280+gniadeck@users.noreply.github.com>

Add type info to remove compilation warning

94ac2d6

Signed-off-by: gniadeck <77535280+gniadeck@users.noreply.github.com>

apply spotless

2f73e11

Signed-off-by: gniadeck <77535280+gniadeck@users.noreply.github.com>

add unit to jvm gc histogram, improve isOtelMetric handling, add otel…

5d925cc

…OptIn metric property Signed-off-by: gniadeck <77535280+gniadeck@users.noreply.github.com>

remove attribute opt-in

d0e8172

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

zeitlinger self-assigned this Feb 6, 2026

zeitlinger mentioned this pull request Feb 6, 2026

Add GC duration histogram #1738

Closed

lint

20cc588

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

jaydeluca reviewed Feb 6, 2026

View reviewed changes

zeitlinger mentioned this pull request Feb 9, 2026

Add docs and example for OTel JVM runtime metrics #1861

Open

zeitlinger marked this pull request as draft February 9, 2026 16:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GC duration histogram #1854

Add GC duration histogram #1854

Uh oh!

zeitlinger commented Feb 6, 2026

Uh oh!

jaydeluca Feb 6, 2026

Uh oh!

jaydeluca Feb 6, 2026

Uh oh!

jaydeluca Feb 6, 2026

Uh oh!

jaydeluca Feb 6, 2026

Uh oh!

jaydeluca Feb 6, 2026

Uh oh!

jaydeluca Feb 6, 2026

Uh oh!

zeitlinger commented Feb 9, 2026

Uh oh!

gniadeck commented Feb 9, 2026

Uh oh!

zeitlinger commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	private static final String JVM_GC_DURATION = "jvm.gc.duration";
	private static final String JVM_GC_DURATION = "jvm_gc_duration";

	public void testNonOtelMetricsAbsentWhenUseOtelEnabled() {
	void testNonOtelMetricsAbsentWhenUseOtelEnabled() {

	public void testGCDurationHistogramLabels() throws Exception {
	void testGCDurationHistogramLabels() throws Exception {

Add GC duration histogram #1854

Are you sure you want to change the base?

Add GC duration histogram #1854

Uh oh!

Conversation

zeitlinger commented Feb 6, 2026

Uh oh!

jaydeluca Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

jaydeluca Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

jaydeluca Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

jaydeluca Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

jaydeluca Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

jaydeluca Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

zeitlinger commented Feb 9, 2026

Uh oh!

gniadeck commented Feb 9, 2026

Uh oh!

zeitlinger commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants