feat: metricsV2 + oTel + prometheus sample and Grafana dashboard by csviri · Pull Request #3154 · operator-framework/java-operator-sdk

csviri · 2026-02-04T16:40:41Z

Goal of this PR is to provide a OTel + Prometheus + Grafana setup. So we:

verify integration with OTel
Provide a default Grafana dashboard for metrics.
Adds a new operator and E2E test for h OTel + Prometheus + Grafana handling verification. This should be easy to done by a users, so dey can check and validate the dashboard.
new metrics implementations

Notes on new metrics implementation:

only static gauges (lives until the operator lives) - that makes much easir to manage them
no manual counter removal (should not be done)
sanitized names and labels according to best practices
added some metrics counter for last attempt retries.

csviri · 2026-02-09T17:35:19Z

JVM metrics:

csviri · 2026-02-09T17:40:23Z

JODSK metrics:

Added TODOs to improve those, like we should not have controller name as suffix, rather as a tag:

operator_sdk_reconciliations_executions_webpagestandalonedependentsreconciler

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

…torsdk/operator/sample/metrics/MetricsHandlingSampleOperator.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

…tor/api/monitoring/Metrics.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

…operatorsdk/operator/sample/deployment.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

xstefank · 2026-03-03T11:48:53Z

docs/content/en/docs/documentation/observability.md

+| Meter name (Micrometer)                  | Type    | Tags                              | Description                                                          |
+|------------------------------------------|---------|-----------------------------------|----------------------------------------------------------------------|
+| `reconciliations.executions`             | gauge   | `controller.name`                 | Number of reconciler executions currently in progress                |
+| `reconciliations.active`                 | gauge   | `controller.name`                 | Number of resources currently queued for reconciliation              |


is "active" really a good name for this?

xstefank · 2026-03-03T12:43:18Z

...ort/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java

+  private static final String RECONCILIATIONS_STARTED = RECONCILIATIONS + "started" + TOTAL_SUFFIX;
+
+  private static final String RECONCILIATIONS_EXECUTIONS_GAUGE = RECONCILIATIONS + "executions";
+  private static final String RECONCILIATIONS_QUEUE_SIZE_GAUGE = RECONCILIATIONS + "active";


queue would be better name

The thing is that queue is a bit misleading, because this is actually the queue + ongoing reconiliations, that is the reason I added active. (ExecutorService has a queue). Can expand the docs on this

Actually made an adjustment so we have now active and queue where queue is that are submitted to executor service but not started yet, the active is the number of actually running threads with reconiliation logic. This seems to be more intuitive and useful.

- deleted redundant delete events - adjusted metrics now we have queue and active reconciliations (with the intuitive seantics) Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Signed-off-by: Chris Laprun <metacosm@gmail.com>

metacosm · 2026-03-04T08:39:16Z

docs/content/en/docs/migration/v5-3-migration.md

-```
+```
+
+## Metrics interface changes


This would technically be an API break and would require a new major version.

metacosm · 2026-03-04T08:41:00Z

observability/install-observability.sh

+
+# Check if helm is installed, download locally if not
+echo -e "\n${YELLOW}Checking helm installation...${NC}"
+if ! command -v helm &> /dev/null; then


Installing helm should be made optional. People might not want to have helm automatically installed or want to install it themselves some other way.

metacosm · 2026-03-04T08:45:10Z

...ort/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java

+import java.util.function.Consumer;
+import java.util.function.Function;
+
+import org.jspecify.annotations.NonNull;


This should not have been introduced here. Nullability should be reviewed at the SDK level, not simply here for 2 methods.

metacosm · 2026-03-04T08:53:29Z

docs/content/en/docs/documentation/observability.md

+
+\* `namespace` tag is only included when `withNamespaceAsTag()` is enabled.
+
+The execution timer uses explicit SLO boundaries (10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s, 10s, 30s) to ensure


SLO acronym should be used expanded first before.

metacosm · 2026-03-04T08:53:55Z

docs/content/en/docs/documentation/observability.md

+\* `namespace` tag is only included when `withNamespaceAsTag()` is enabled.
+
+The execution timer uses explicit SLO boundaries (10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s, 10s, 30s) to ensure
+compatibility with `histogram_quantile()` queries in Prometheus. This is important when using the OTLP registry, where


OTLP acronym should be used expanded first.

metacosm · 2026-03-04T09:08:45Z

...ort/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java

+      if (resourceEvent.getAction() == ResourceAction.ADDED) {
+        gauges.get(numberOfResourcesRefName(getControllerName(metadata))).incrementAndGet();
+      }
+      var namespace = resourceEvent.getRelatedCustomResourceID().getNamespace().orElse(null);


Shouldn't that be in sync with what's done for MDC, i.e. use a default value instead of null for clustered resources?

metacosm · 2026-03-04T09:12:09Z

...ort/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java

+
+  private void incrementCounter(
+      String counterName, String namespace, Map<String, Object> metadata, Tag... additionalTags) {
+    final var tags = new ArrayList<Tag>(2 + additionalTags.length);


This shouldn't be a List but a Set.

metacosm · 2026-03-04T09:14:34Z

...ort/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java

+          null,
+          metadata,
+          Tag.of(EVENT, event.getClass().getSimpleName()),
+          Tag.of(ACTION, UNKNOWN_ACTION));


Why unknown action here?

metacosm · 2026-03-04T09:17:42Z

...ort/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java

+
+    int retryNumber = retryInfo.map(RetryInfo::getAttemptCount).orElse(0);
+    if (retryNumber > 0) {
+      incrementCounter(RECONCILIATIONS_RETRIES_NUMBER, namespace, metadata);


How does the retry number get propagated? Seems like the counter that is incremented is completely decorrelated from the actual retry number?

Signed-off-by: Chris Laprun <metacosm@gmail.com>

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 4, 2026

This was linked to issues Feb 4, 2026

Provide a default Grafana dashboard for the default metrics #2975

Open

Investigate integration possibilities with Opentelemetry #2135

Open

csviri changed the title ~~OTel+Prometheus sample and Graphana dashboard~~ [WIP] OTel+Prometheus sample and Graphana dashboard Feb 4, 2026

csviri changed the title ~~[WIP] OTel+Prometheus sample and Graphana dashboard~~ [WIP] OTel+Prometheus sample and Grfana dashboard Feb 8, 2026

csviri changed the title ~~[WIP] OTel+Prometheus sample and Grfana dashboard~~ [WIP] OTel+Prometheus sample and Grafana dashboard Feb 8, 2026

csviri force-pushed the otel-metrics-grafana branch from c7e6ca2 to ece63e8 Compare February 8, 2026 15:25

csviri mentioned this pull request Feb 9, 2026

Micrometer metrics enhancement requests #2974

Open

csviri linked an issue Feb 10, 2026 that may be closed by this pull request

Micrometer metrics enhancement requests #2974

Open

csviri mentioned this pull request Feb 10, 2026

[WIP] improve: micrometer metrics improvements #3164

Closed

csviri requested review from metacosm and xstefank February 10, 2026 11:50

csviri changed the title ~~[WIP] OTel+Prometheus sample and Grafana dashboard~~ [WIP] MetricsV2 + OTel+Prometheus sample and Grafana dashboard Feb 11, 2026

csviri force-pushed the next branch from 280b162 to 429c2c5 Compare February 16, 2026 09:11

csviri force-pushed the otel-metrics-grafana branch 2 times, most recently from 88c118c to cf9eb57 Compare February 21, 2026 12:06

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 23, 2026

csviri force-pushed the otel-metrics-grafana branch from 24f992f to 0438611 Compare February 27, 2026 09:52

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 27, 2026

csviri force-pushed the next branch from bb01a06 to 4868613 Compare February 28, 2026 09:55

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 1, 2026

csviri force-pushed the otel-metrics-grafana branch from 9014b2b to fe4ab6a Compare March 1, 2026 09:51

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 1, 2026

csviri changed the title ~~[WIP] MetricsV2 + OTel+Prometheus sample and Grafana dashboard~~ MetricsV2 + OTel+Prometheus sample and Grafana dashboard Mar 1, 2026

csviri self-assigned this Mar 1, 2026

csviri marked this pull request as ready for review March 1, 2026 10:45

Copilot AI review requested due to automatic review settings March 1, 2026 10:45

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 1, 2026

csviri and others added 19 commits March 3, 2026 08:15

wip

3ccd430

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

3fffab4

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

e52bd37

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

documentation update

a751867

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

4fc3069

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

logging

a3d935d

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Update sample-operators/metrics-processing/src/main/java/io/javaopera…

e6ad757

…torsdk/operator/sample/metrics/MetricsHandlingSampleOperator.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Update sample-operators/metrics-processing/pom.xml

5f46c5c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Update sample-operators/metrics-processing/pom.xml

23416f1

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Update operator-framework-core/src/main/java/io/javaoperatorsdk/opera…

08f33db

…tor/api/monitoring/Metrics.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Update observability/install-observability.sh

e5acc60

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

4eda27b

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Update sample-operators/metrics-processing/src/main/resources/io/java…

6282ca4

…operatorsdk/operator/sample/deployment.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

0074f84

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

e2c4751

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

0d1a23c

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

6ceb74d

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

45fc814

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

cf114e3

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

csviri force-pushed the otel-metrics-grafana branch from d60193a to cf114e3 Compare March 3, 2026 07:15

xstefank approved these changes Mar 3, 2026

View reviewed changes

csviri and others added 7 commits March 3, 2026 15:28

Refinements on metrics

a804406

- deleted redundant delete events - adjusted metrics now we have queue and active reconciliations (with the intuitive seantics) Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

fbb67b6

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

docs improvement

42e2649

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

fix: add deprecation information

ebac5e2

Signed-off-by: Chris Laprun <metacosm@gmail.com>

refactor: consistent constant definition

810ba90

Signed-off-by: Chris Laprun <metacosm@gmail.com>

refactor: reuse available methods to help inlining

995ace7

Signed-off-by: Chris Laprun <metacosm@gmail.com>

refactor: avoid creating intermediate collection

4a2ec96

Signed-off-by: Chris Laprun <metacosm@gmail.com>

metacosm requested changes Mar 4, 2026

View reviewed changes

refactor: remove unused constant

05de31a

Signed-off-by: Chris Laprun <metacosm@gmail.com>


		\* `namespace` tag is only included when `withNamespaceAsTag()` is enabled.

		The execution timer uses explicit SLO boundaries (10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s, 10s, 30s) to ensure

Conversation

csviri commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

csviri commented Feb 9, 2026

Uh oh!

csviri commented Feb 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

csviri Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

csviri commented Feb 4, 2026 •

edited

Loading

csviri Mar 3, 2026 •

edited

Loading