CM-1114: Add health probes and richer status conditions

[sebrandon1]

Summary

Adds two improvements to the cert-manager operator:

Health probes

The operator deployment is the only cert-manager component without Kubernetes health probes. Since the library-go controllercmd framework already serves /healthz and /readyz over HTTPS on port 8443 via its GenericAPIServer, this change only requires manifest updates -- no Go code changes.

• Liveness (/healthz): ping, log, post-start hooks
• Readiness (/readyz): same checks plus shutdown, allowing the pod to drain traffic during graceful termination

Richer status conditions

IstioCSR and TrustManager CRs previously reported only Ready and Degraded conditions with generic reason strings (Failed, Ready, Progressing). Users could not determine why a resource was progressing or degraded without reading operator logs.

This change adds:

• A Progressing condition type alongside Ready and Degraded
• Specific reason constants: Reconciling, WaitingForDependencies, ValidationFailed, MultipleInstancesFound
• A ConditionReason field on ReconcileError with a WithConditionReason() chainable setter, allowing controllers to annotate errors with structured reasons
• Updated HandleReconcileResult to manage all three conditions and extract specific reasons from the error chain

Behavioral change: Duplicate-instance errors now report MultipleInstancesFound as the condition reason instead of the previous generic Failed. Validation errors report ValidationFailed.

Jira

• CM-1114 -- Add health probes and richer status conditions (Code Review)

Test Plan

Health probes

Verified probe endpoints respond correctly on a running operator:

$ curl -sk "https://localhost:8443/healthz?verbose"
[+]ping ok
[+]log ok
[+]poststarthook/max-in-flight-filter ok
[+]poststarthook/storage-object-count-tracker-hook ok
healthz check passed

$ curl -sk "https://localhost:8443/readyz?verbose"
[+]ping ok
[+]log ok
[+]poststarthook/max-in-flight-filter ok
[+]poststarthook/storage-object-count-tracker-hook ok
[+]shutdown ok
readyz check passed

Status conditions -- Reconciling

IstioCSR CR during active reconciliation (retrying due to missing namespace):

[
    {
        "type": "Degraded",
        "status": "False",
        "reason": "Ready"
    },
    {
        "type": "Ready",
        "status": "False",
        "reason": "Progressing",
        "message": "reconciliation failed, retrying: ..."
    },
    {
        "type": "Progressing",
        "status": "True",
        "reason": "Reconciling",
        "message": "reconciliation in progress: ..."
    }
]

Status conditions -- MultipleInstancesFound

Second IstioCSR CR rejected as a duplicate:

[
    {
        "type": "Degraded",
        "status": "False",
        "reason": "MultipleInstancesFound"
    },
    {
        "type": "Ready",
        "status": "False",
        "reason": "MultipleInstancesFound",
        "message": "multiple instances of istiocsr exists, ..."
    },
    {
        "type": "Progressing",
        "status": "False",
        "reason": "MultipleInstancesFound",
        "message": "multiple instances of istiocsr exists, ..."
    }
]

Verification

• All unit tests pass (123/123 Ginkgo specs + all Go packages)
• No lint issues from changed files
• Verified on OCP 4.22 cluster -- all three conditions visible with correct reasons
• E2E tests pass

[openshift-ci-robot]

@sebrandon1: This pull request references CNF-23436 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

The operator deployment currently has no health probes, so Kubernetes cannot detect if the operator process is stuck or not yet ready to serve. All cert-manager operands (controller, webhook, cainjector, trust-manager, istio-csr) already have probes configured — the operator itself is the only component missing them.

The library-go controllercmd framework already serves /healthz and /readyz over HTTPS on port 8443 via its GenericAPIServer, so no Go code changes are needed.

• Liveness → /healthz (ping, log, post-start hooks)
• Readiness → /readyz (same checks + shutdown, so the pod drains traffic during graceful termination)

Test plan

Tested locally against an OCP 4.22 cluster:

$ curl -sk "https://localhost:8443/healthz?verbose"
[+]ping ok
[+]log ok
[+]poststarthook/max-in-flight-filter ok
[+]poststarthook/storage-object-count-tracker-hook ok
healthz check passed

$ curl -sk "https://localhost:8443/readyz?verbose"
[+]ping ok
[+]log ok
[+]poststarthook/max-in-flight-filter ok
[+]poststarthook/storage-object-count-tracker-hook ok
[+]shutdown ok
readyz check passed

• Operator deploys and reports ready
• /healthz and /readyz return 200 when operator is healthy
• Pod is restarted by kubelet when liveness probe fails
• Pod is removed from service endpoints during graceful shutdown via readyz shutdown check

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

The PR adds HTTPS liveness and readiness probes. It adds Progressing status handling and explicit condition reasons across shared reconciliation logic, IstioCSR, and TrustManager controllers.

Changes

Reconciliation status and health probes

Layer / File(s)	Summary
Condition and error-reason contracts api/operator/v1alpha1/conditions.go, api/operator/v1alpha1/conditions_test.go, pkg/controller/common/errors.go, pkg/controller/common/errors_test.go	Adds the Progressing condition, reason constants, and condition-reason storage and extraction helpers with tests.
Shared reconciliation condition handling pkg/controller/common/reconcile_result.go, pkg/controller/common/reconcile_result_test.go	Updates successful, recoverable, and irrecoverable outcomes to manage Degraded, Ready, and Progressing, with tests for status updates, no-op updates, requeueing, and error propagation.
Controller-specific reason wiring pkg/controller/istiocsr/*, pkg/controller/trustmanager/*, test/e2e/trustmanager_test.go	Adds validation, dependency, and multiple-instance reasons and updates controller and e2e condition expectations.
Controller-manager health probes config/manager/manager.yaml, bundle/manifests/cert-manager-operator.clusterserviceversion.yaml	Adds HTTPS liveness and readiness checks for /healthz and /readyz on the named https port.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Controller
  participant HandleReconcileResult
  participant ReconcileError
  participant ConditionalStatus
  Controller->>HandleReconcileResult: pass reconciliation result
  HandleReconcileResult->>ReconcileError: extract condition reason
  HandleReconcileResult->>ConditionalStatus: set Degraded, Ready, and Progressing
  ConditionalStatus-->>Controller: persist status when conditions change

Loading

Possibly related PRs

• openshift/cert-manager-operator#427: Related IstioCSR status-condition behavior and test coverage.
• openshift/cert-manager-operator#465: Related reconciliation error and condition-reason handling.

Suggested reviewers: swghosh, trilokgeer

🚥 Pre-merge checks | ✅ 13 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 26.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality	⚠️ Warning	The changed Ginkgo spec uses bounded Eventually waits and cleanup hooks, but its new condition assertions in test/e2e/trustmanager_test.go have no diagnostic messages.	Add meaningful messages to the new or modified g.Expect assertions and to the outer Eventually assertion, such as identifying the expected condition and status.

✅ Passed checks (13 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	No added or modified Ginkgo title declarations were found. The affected E2E Describe, Context, and It titles are static, and unit-test subtest names use static literals.
Microshift Test Compatibility	✅ Passed	The PR adds no new Ginkgo e2e nodes; it only updates assertions in an existing TrustManager test, and the added lines reference no unavailable MicroShift APIs or resources.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The PR adds no new Ginkgo e2e declarations. Its only e2e change updates existing TrustManager condition assertions and adds no multi-node assumption.
Topology-Aware Scheduling Compatibility	✅ Passed	The diff adds only HTTPS probes and status handling. Added lines contain no scheduling constraints; existing replicas, broad arch/OS affinity, and strategy remain unchanged.
Ote Binary Stdout Contract	✅ Passed	PR files add no process-level stdout writes; the only init change registers schemes, and suite logging uses GinkgoWriter. main() and suite setup contain no fmt/log stdout calls.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The e2e diff only updates an existing TrustManager test; it adds no Ginkgo test declarations, IPv4 literals, IP parsing, external URLs, or public-network access.
No-Weak-Crypto	✅ Passed	The PR diff adds probes and status-condition handling only; changed Go files contain no weak algorithms, crypto APIs, custom crypto, or secret/token comparisons.
Container-Privileges	✅ Passed	Changed manifests retain runAsNonRoot=true, privileged=false, allowPrivilegeEscalation=false, and drop ALL; no hostPID, hostNetwork, hostIPC, SYS_ADMIN, or root settings were added.
No-Sensitive-Data-In-Logs	✅ Passed	Changed production logging adds only condition flags/reason constants and namespace/name fields; error logging was pre-existing, and no passwords, tokens, keys, PII, or customer data are introduced.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes both main changes: health probes and richer status conditions.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sebrandon1
Once this PR has been reviewed and has the lgtm label, please assign swghosh for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

🧹 Nitpick comments (1)

config/manager/manager.yaml (1)

114-122: ⚡ Quick win

Tune readiness probe for faster drain on shutdown.

To better align with graceful termination, Line 120 and Line 122 are a bit slow (10s * 3 worst-case before NotReady). Consider faster readiness failure so endpoints stop routing sooner.

Suggested tweak

           readinessProbe:
             httpGet:
               path: /readyz
               port: https
               scheme: HTTPS
             initialDelaySeconds: 5
-            periodSeconds: 10
+            periodSeconds: 5
             timeoutSeconds: 5
-            failureThreshold: 3
+            failureThreshold: 1

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@config/manager/manager.yaml` around lines 114 - 122, The readinessProbe for
the manager (httpGet path "/readyz", scheme HTTPS) is too slow to mark Pod
NotReady during shutdown; adjust readinessProbe settings to fail faster by
lowering periodSeconds (e.g., from 10 to 2–3), reducing failureThreshold (e.g.,
from 3 to 1–2) and/or decreasing timeoutSeconds to ensure the probe transitions
to NotReady quickly so endpoints are drained sooner; update the readinessProbe
block (httpGet path /readyz, initialDelaySeconds, periodSeconds, timeoutSeconds,
failureThreshold) accordingly.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@config/manager/manager.yaml`:
- Around line 114-122: The readinessProbe for the manager (httpGet path
"/readyz", scheme HTTPS) is too slow to mark Pod NotReady during shutdown;
adjust readinessProbe settings to fail faster by lowering periodSeconds (e.g.,
from 10 to 2–3), reducing failureThreshold (e.g., from 3 to 1–2) and/or
decreasing timeoutSeconds to ensure the probe transitions to NotReady quickly so
endpoints are drained sooner; update the readinessProbe block (httpGet path
/readyz, initialDelaySeconds, periodSeconds, timeoutSeconds, failureThreshold)
accordingly.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f1db3899-53e4-40c3-a38c-c2fc93c4f11f

📥 Commits

Reviewing files that changed from the base of the PR and between e2f1df3 and cc2910e.

📒 Files selected for processing (2)

• bundle/manifests/cert-manager-operator.clusterserviceversion.yaml
• config/manager/manager.yaml

[sebrandon1]

/retest

[openshift-ci-robot]

@sebrandon1: This pull request references CM-1114 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Health Probes

The operator deployment currently has no health probes, so Kubernetes cannot detect if the operator process is stuck or not yet ready to serve. All cert-manager operands (controller, webhook, cainjector, trust-manager, istio-csr) already have probes configured — the operator itself is the only component missing them.

The library-go controllercmd framework already serves /healthz and /readyz over HTTPS on port 8443 via its GenericAPIServer, so no Go code changes are needed.

• Liveness → /healthz (ping, log, post-start hooks)
• Readiness → /readyz (same checks + shutdown, so the pod drains traffic during graceful termination)

Richer Status Conditions

IstioCSR and TrustManager CRs currently report only two status conditions (Ready and Degraded) with generic reason constants (Failed, Ready, Progressing). Users can't tell why something is progressing or degraded without reading operator logs.

This adds:

• A dedicated Progressing condition type alongside Ready and Degraded
• Specific reason constants: Reconciling, WaitingForDependencies, ValidationFailed, MultipleInstancesFound
• A ConditionReason field on ReconcileError with WithConditionReason() chainable setter so controllers can annotate errors with specific reasons
• Updated HandleReconcileResult to manage all three conditions and extract specific reasons from errors

Test plan

Health Probes

$ curl -sk "https://localhost:8443/healthz?verbose"
[+]ping ok
[+]log ok
[+]poststarthook/max-in-flight-filter ok
[+]poststarthook/storage-object-count-tracker-hook ok
healthz check passed

$ curl -sk "https://localhost:8443/readyz?verbose"
[+]ping ok
[+]log ok
[+]poststarthook/max-in-flight-filter ok
[+]poststarthook/storage-object-count-tracker-hook ok
[+]shutdown ok
readyz check passed

Status Conditions — Reconciling

IstioCSR CR during active reconciliation (retrying due to missing namespace):

[
   {
       "lastTransitionTime": "2026-06-08T21:44:45Z",
       "message": "",
       "reason": "Ready",
       "status": "False",
       "type": "Degraded"
   },
   {
       "lastTransitionTime": "2026-06-08T21:44:45Z",
       "message": "reconciliation failed, retrying: failed to create istio-system/cert-manager-istio-csr role resource: ...",
       "reason": "Progressing",
       "status": "False",
       "type": "Ready"
   },
   {
       "lastTransitionTime": "2026-06-08T21:44:45Z",
       "message": "reconciliation in progress: failed to create istio-system/cert-manager-istio-csr role resource: ...",
       "reason": "Reconciling",
       "status": "True",
       "type": "Progressing"
   }
]

Status Conditions — MultipleInstancesFound

Second IstioCSR CR rejected as a duplicate:

[
   {
       "lastTransitionTime": "2026-06-08T21:45:06Z",
       "message": "",
       "reason": "MultipleInstancesFound",
       "status": "False",
       "type": "Degraded"
   },
   {
       "lastTransitionTime": "2026-06-08T21:45:06Z",
       "message": "multiple instances of istiocsr exists, cert-manager-operator/default will not be processed",
       "reason": "MultipleInstancesFound",
       "status": "False",
       "type": "Ready"
   },
   {
       "lastTransitionTime": "2026-06-08T21:45:06Z",
       "message": "multiple instances of istiocsr exists, cert-manager-operator/default will not be processed",
       "reason": "MultipleInstancesFound",
       "status": "False",
       "type": "Progressing"
   }
]

• All unit tests pass (123/123 Ginkgo specs + all Go packages)
• No lint issues from changed files
• Verified on OCP 4.22 cluster — all three conditions visible with correct reasons
• E2E tests pass

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[sebrandon1]

@coderabbitai resume

[coderabbitai]

✅ Action performed

Reviews resumed.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/controller/common/errors.go`:
- Around line 143-146: Update GetConditionReason to verify rerr is non-nil after
errors.As succeeds before accessing ConditionReason, returning the existing
fallback for a typed-nil *ReconcileError; add a regression test covering an
error interface containing a nil *ReconcileError.

In `@pkg/controller/trustmanager/install_trustmanager.go`:
- Around line 21-23: Update the error handling around validateTrustNamespace in
the trust-manager installation flow to distinguish a missing trust namespace
from lookup/API failures. Wrap only the not-found result with
ReasonWaitingForDependencies; propagate other validation errors without that
reason so transient failures remain retryable.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bac91557-f9e0-4aaf-8ed7-9e5aedab1f3c

📥 Commits

Reviewing files that changed from the base of the PR and between d9d40bd and b484ca2.

📒 Files selected for processing (13)

• api/operator/v1alpha1/conditions.go
• api/operator/v1alpha1/conditions_test.go
• bundle/manifests/cert-manager-operator.clusterserviceversion.yaml
• config/manager/manager.yaml
• pkg/controller/common/errors.go
• pkg/controller/common/errors_test.go
• pkg/controller/common/reconcile_result.go
• pkg/controller/istiocsr/controller_test.go
• pkg/controller/istiocsr/install_istiocsr.go
• pkg/controller/istiocsr/utils.go
• pkg/controller/trustmanager/controller_test.go
• pkg/controller/trustmanager/install_trustmanager.go
• test/e2e/trustmanager_test.go

🚧 Files skipped from review as they are similar to previous changes (10)

• pkg/controller/istiocsr/install_istiocsr.go
• bundle/manifests/cert-manager-operator.clusterserviceversion.yaml
• config/manager/manager.yaml
• pkg/controller/istiocsr/utils.go
• pkg/controller/common/errors_test.go
• api/operator/v1alpha1/conditions.go
• pkg/controller/trustmanager/controller_test.go
• api/operator/v1alpha1/conditions_test.go
• test/e2e/trustmanager_test.go
• pkg/controller/istiocsr/controller_test.go

[sebrandon1]

/rebase

[sebrandon1]

/retest

[openshift-ci]

@sebrandon1: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-operator	f76b729	link	true	/test e2e-operator

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.