For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

Вебхуки

В этом документе описана функциональность webhook'ов в Dynamo Operator, включая validation webhooks, управление сертификатами и устранение неполадок.

Обзор

Dynamo Operator использует Kubernetes admission webhooks, чтобы выполнять проверку и изменение пользовательских ресурсов в реальном времени. Сейчас оператор реализует validation webhooks, которые гарантируют, что некорректные конфигурации сразу отклоняются на уровне API server, давая более быстрый отклик по сравнению с проверкой на уровне controller.

Все типы webhook'ов (validating, mutating, conversion и т. д.) используют один и тот же webhook server и ту же TLS certificate infrastructure, поэтому управление сертификатами остается единообразным для всех операций webhook.

Ключевые возможности

✅ Всегда включены - Webhook'и являются обязательным компонентом оператора
✅ Общая инфраструктура сертификатов - все типы webhook'ов используют одни и те же TLS-сертификаты
✅ Автоматическое создание и ротация сертификатов - встроенный cert-controller, ручное управление не требуется
✅ Интеграция с cert-manager - опциональная интеграция для custom PKI или политик сертификатов организации
✅ Контроль неизменяемости - критические поля защищены правилами CEL validation

Текущие типы webhook'ов

Validating Webhooks: проверяют спецификации пользовательских ресурсов до сохранения
- DynamoComponentDeployment validation
- DynamoGraphDeployment validation
- DynamoModel validation
- DynamoGraphDeploymentRequest validation
Mutating Webhooks: применяют значения по умолчанию к ресурсам при создании
- DynamoGraphDeployment defaulting

Примечание: все типы webhook'ов используют ту же инфраструктуру сертификатов, которая описана в этом документе.

Архитектура

┌─────────────────────────────────────────────────────────────────┐
│                         API Server                               │
│  1. User submits CR (kubectl apply)                             │
│  2. API server calls MutatingWebhookConfiguration               │
└────────────────────────┬────────────────────────────────────────┘
                         │ HTTPS (TLS required)
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                  Webhook Server (in Operator Pod)                │
│  3. Applies defaults (e.g., operator version annotation)        │
│  4. Returns mutated CR                                          │
└────────────────────────┬────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                         API Server                               │
│  5. API server calls ValidatingWebhookConfiguration             │
└────────────────────────┬────────────────────────────────────────┘
                         │ HTTPS (TLS required)
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                  Webhook Server (in Operator Pod)                │
│  6. Validates CR against business rules                         │
│  7. Returns admit/deny decision + warnings                      │
└────────────────────────┬────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                      API Server                                  │
│  8. If admitted: Persist CR to etcd                             │
│  9. If denied: Return error to user                             │
└─────────────────────────────────────────────────────────────────┘

Поток admission

Mutating webhooks: применяют значения по умолчанию и преобразования перед проверкой
Validating webhooks: проверяют (возможно измененный) CR на соответствие бизнес-правилам
CEL validation: встроенные в Kubernetes проверки неизменяемости (всегда активны)

Обновление с версий, где `webhook.enabled: false`

Параметр Helm webhook.enabled удален. Теперь webhook'и являются обязательным компонентом оператора и всегда активны. Если раньше вы запускали систему с webhook.enabled: false, перед обновлением выполните следующие шаги:

Удалите webhook.enabled из любых файлов custom values. Helm проигнорирует неизвестный ключ, но его лучше убрать, чтобы не создавать путаницу.
Убедитесь, что порт 9443 доступен от Kubernetes API server до pod'а оператора. Если у вас есть правила NetworkPolicy или firewall, ограничивающие трафик, добавьте ingress-правило, разрешающее API server подключаться к webhook server на порт 9443.
Убедитесь, что TLS-сертификаты webhook доступны. По умолчанию встроенный cert-controller оператора автоматически создает и ротирует self-signed сертификаты при запуске - ничего делать не нужно. Если вы используете cert-manager или внешне управляемые сертификаты, проверьте, что конфигурация готова до обновления.

Конфигурация

Варианты управления сертификатами

The operator supports three certificate management modes:

Mode	Description	Use Case
Automatic (Default)	Operator's built-in cert-controller generates and rotates certificates	All environments (recommended)
cert-manager	Integrate with cert-manager for certificate lifecycle management	Clusters with cert-manager and custom PKI requirements
External	Bring your own certificates	Environments with externally managed PKI

Расширенная конфигурация

Complete Configuration Reference

dynamo-operator:
  webhook:
    # Certificate management (optional, to use cert-manager instead of built-in)
    certManager:
      enabled: false
      issuerRef:
        kind: Issuer
        name: selfsigned-issuer

    # Certificate secret configuration
    certificateSecret:
      name: webhook-server-cert
      external: false           # Set to true for externally managed certificates

    # Webhook behavior configuration
    failurePolicy: Fail        # Fail (reject on error) or Ignore (allow on error)
    timeoutSeconds: 10         # Webhook timeout

    # Namespace filtering (advanced)
    namespaceSelector: {}      # Kubernetes label selector for namespaces

Failure Policy

# Fail: Reject resources if webhook is unavailable (recommended for production)
webhook:
  failurePolicy: Fail

# Ignore: Allow resources if webhook is unavailable (use with caution)
webhook:
  failurePolicy: Ignore

Recommendation: Use Fail in production to ensure validation is always enforced. Only use Ignore if you need high availability and can tolerate occasional invalid resources.

Namespace Filtering

Control which namespaces are validated (applies to cluster-wide operator only):

# Only validate resources in namespaces with specific labels
webhook:
  namespaceSelector:
    matchLabels:
      dynamo-validation: enabled

# Or exclude specific namespaces
webhook:
  namespaceSelector:
    matchExpressions:
    - key: dynamo-validation
      operator: NotIn
      values: ["disabled"]

Note: For namespace-restricted operators (deprecated), the namespace selector is automatically set to validate only the operator's namespace. This configuration is ignored in namespace-restricted mode.

Certificate Management

Automatic Certificates (Default)

Zero configuration required! The operator's built-in cert-controller generates and rotates certificates automatically at startup.

How It Works

Operator starts: The CertManager checks for an existing certificate Secret (configured via webhook.certificateSecret.name, default: webhook-server-cert). If missing or invalid, it generates a self-signed Root CA and server certificate and writes them to the Secret.
CA bundle injection: The CABundleInjector reads ca.crt from the Secret and patches both the ValidatingWebhookConfiguration and MutatingWebhookConfiguration with the base64-encoded CA bundle.
Certificate rotation: The cert-controller monitors certificate validity and regenerates certificates before they expire.
Webhook server starts: The webhook server only begins serving after certificates are confirmed ready, preventing startup races.

Certificate Validity

Root CA: 10 years
Server Certificate: 10 years (same as Root CA)
Automatic rotation: The cert-controller monitors validity and regenerates before expiration

Smart Certificate Management

The cert-controller is intelligent about certificate lifecycle:

✅ Checks existing certificates at startup before generating new ones
✅ Skips generation if valid certificates already exist in the Secret
✅ Regenerates only when needed (missing, expiring soon, or incorrect SANs)

This means:

Fast operator restarts (no unnecessary cert generation)
No dependency on Helm hooks or external Jobs
Certificates persist across pod restarts (stored in Secret)

Manual Certificate Rotation

If you need to rotate certificates manually:

# Delete the certificate secret -- the operator will regenerate it on restart
kubectl delete secret <release>-webhook-server-cert -n <namespace>

# Restart the operator pod to trigger regeneration
kubectl rollout restart deployment/<release>-dynamo-operator -n <namespace>

cert-manager Integration

For clusters with cert-manager installed, you can enable automated certificate lifecycle management.

Prerequisites

cert-manager installed (v1.0+)
CA issuer configured (e.g., selfsigned-issuer)

Configuration

dynamo-operator:
  webhook:
    certManager:
      enabled: true
      issuerRef:
        kind: Issuer              # Or ClusterIssuer
        name: selfsigned-issuer   # Your issuer name

How It Works

Helm creates Certificate resource: Requests TLS certificate from cert-manager
cert-manager generates certificate: Based on configured issuer
cert-manager stores in Secret: <release>-webhook-server-cert
cert-manager ca-injector: Automatically injects CA bundle into ValidatingWebhookConfiguration
Operator pod: Mounts certificate secret and serves webhook

When to Use cert-manager

✅ Custom validity periods: Configure certificate lifetime to match organizational policy
✅ Integration with existing PKI: Use your organization's certificate infrastructure
✅ Centralized certificate management: Manage all cluster certificates through cert-manager

Certificate Rotation

With cert-manager, certificate rotation is fully automated:

Leaf certificate rotation (default: every year)
- cert-manager auto-renews before expiration
- controller-runtime auto-reloads new certificate
- No pod restart required
- No caBundle update required (same Root CA)
Root CA rotation (every 10 years)
- cert-manager rotates Root CA
- ca-injector auto-updates caBundle in ValidatingWebhookConfiguration
- No manual intervention required

Example: Self-Signed Issuer

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: dynamo-system
spec:
  selfSigned: {}
---
# Enable in platform values.yaml
dynamo-operator:
  webhook:
    certManager:
      enabled: true
      issuerRef:
        kind: Issuer
        name: selfsigned-issuer

External Certificates

Bring your own certificates for custom PKI requirements.

Steps

Create certificate secret manually:

kubectl create secret tls <release>-webhook-server-cert \
  --cert=tls.crt \
  --key=tls.key \
  -n <namespace>

# Also add ca.crt to the secret
kubectl patch secret <release>-webhook-server-cert -n <namespace> \
  --type='json' \
  -p='[{"op": "add", "path": "/data/ca.crt", "value": "'$(base64 -w0 < ca.crt)'"}]'

Configure operator to use external secret:

dynamo-operator:
  webhook:
    certificateSecret:
      external: true
    caBundle: <base64-encoded-ca-cert>  # Must manually specify

Deploy operator:

helm install dynamo-platform . -n <namespace> -f values.yaml

Certificate Requirements

Secret name: Must match webhook.certificateSecret.name (default: webhook-server-cert)
Secret keys: tls.crt, tls.key, ca.crt
Certificate SAN: Must include <service-name>.<namespace>.svc
- Example: dynamo-platform-dynamo-operator-webhook-service.dynamo-system.svc

Multi-Operator Deployments (DEPRECATED)

DEPRECATED: Namespace-restricted mode and multi-operator deployments are deprecated and will be removed in a future release. Use a single cluster-wide operator instead.

The operator supports running both cluster-wide and namespace-restricted instances simultaneously using a lease-based coordination mechanism.

Scenario

Cluster:
├─ Operator A (cluster-wide, namespace: platform-system)
│  └─ Validates all namespaces EXCEPT team-a
└─ Operator B (namespace-restricted, namespace: team-a)
   └─ Validates only team-a namespace

How It Works

Namespace-restricted operator creates a Lease in its namespace
Cluster-wide operator watches for Leases named dynamo-operator-ns-lock
Cluster-wide operator skips validation for namespaces with active Leases
Namespace-restricted operator validates resources in its namespace

Lease Configuration

The lease mechanism is automatically configured based on deployment mode:

# Cluster-wide operator (default)
namespaceRestriction:
  enabled: false
# → Watches for leases in all namespaces
# → Skips validation for namespaces with active leases

# Namespace-restricted operator
namespaceRestriction:
  enabled: true
  namespace: team-a
# → Creates lease in team-a namespace
# → Does NOT check for leases (no cluster permissions)

Deployment Example

# 1. Deploy cluster-wide operator
helm install platform-operator dynamo-platform \
  -n platform-system \
  --set namespaceRestriction.enabled=false

# 2. Deploy namespace-restricted operator for team-a
helm install team-a-operator dynamo-platform \
  -n team-a \
  --set namespaceRestriction.enabled=true \
  --set namespaceRestriction.namespace=team-a

ValidatingWebhookConfiguration Naming

The webhook configuration name reflects the deployment mode:

Cluster-wide: <release>-validating
Namespace-restricted: <release>-validating-<namespace>

Example:

# Cluster-wide
platform-operator-validating

# Namespace-restricted (team-a)
team-a-operator-validating-team-a

This allows multiple webhook configurations to coexist without conflicts.

Lease Health

If the namespace-restricted operator is deleted or becomes unhealthy:

Lease expires after leaseDuration + gracePeriod (default: ~30 seconds)
Cluster-wide operator automatically resumes validation for that namespace

Troubleshooting

Webhook Not Called

Symptoms:

Invalid resources are accepted
No validation errors in logs

Checks:

Verify webhook configuration exists:

kubectl get validatingwebhookconfiguration | grep dynamo

Check webhook configuration:

kubectl get validatingwebhookconfiguration <name> -o yaml
# Verify:
# - caBundle is present and non-empty
# - clientConfig.service points to correct service
# - webhooks[].namespaceSelector matches your namespace

Verify webhook service exists:

kubectl get service -n <namespace> | grep webhook

Check operator logs for webhook startup:

kubectl logs -n <namespace> deployment/<release>-dynamo-operator | grep webhook
# Should see: "Registering validation webhooks"
# Should see: "Starting webhook server"

Connection Refused Errors

Symptoms:

Error from server (InternalError): Internal error occurred: failed calling webhook:
Post "https://...webhook-service...:443/validate-...": dial tcp ...:443: connect: connection refused

Checks:

Verify operator pod is running:

kubectl get pods -n <namespace> -l app.kubernetes.io/name=dynamo-operator

Check webhook server is listening:

# Port-forward to pod
kubectl port-forward -n <namespace> pod/<operator-pod> 9443:9443

# In another terminal, test connection
curl -k https://localhost:9443/validate-nvidia-com-v1alpha1-dynamocomponentdeployment
# Should NOT get "connection refused"

Verify webhook port in deployment:

kubectl get deployment -n <namespace> <release>-dynamo-operator -o yaml | grep -A5 "containerPort: 9443"

Check for webhook initialization errors:

kubectl logs -n <namespace> deployment/<release>-dynamo-operator | grep -i error

Certificate Errors

Symptoms:

Error from server (InternalError): Internal error occurred: failed calling webhook:
x509: certificate signed by unknown authority

Checks:

Verify caBundle is present:

kubectl get validatingwebhookconfiguration <name> -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d
# Should output a valid PEM certificate

Verify certificate secret exists:

kubectl get secret -n <namespace> <release>-webhook-server-cert

Check certificate validity:

kubectl get secret -n <namespace> <release>-webhook-server-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text
# Check:
# - Not expired
# - SAN includes: <service-name>.<namespace>.svc

Check operator logs for CA injection errors:

kubectl logs -n <namespace> deployment/<release>-dynamo-operator | grep -i "cert\|ca.*bundle\|inject"

Certificate Controller Errors

Symptoms:

Operator logs show cert-controller errors
Certificate Secret is not created
CA bundle is not injected into webhook configurations

Checks:

Check cert-controller logs:

kubectl logs -n <namespace> deployment/<release>-dynamo-operator | grep -i "cert-manager\|cert-rotation\|cert-controller"

Verify RBAC permissions:

# The operator needs permissions to manage Secrets, ValidatingWebhookConfigurations,
# MutatingWebhookConfigurations, and CustomResourceDefinitions
kubectl auth can-i create secrets -n <namespace> --as=system:serviceaccount:<namespace>:<release>-dynamo-operator
kubectl auth can-i patch validatingwebhookconfigurations --as=system:serviceaccount:<namespace>:<release>-dynamo-operator

Check if the certificate Secret was created:

kubectl get secret -n <namespace> <release>-webhook-server-cert

Force certificate regeneration:

# Delete the certificate secret and restart the operator
kubectl delete secret <release>-webhook-server-cert -n <namespace>
kubectl rollout restart deployment/<release>-dynamo-operator -n <namespace>

Validation Errors Not Clear

Symptoms:

Webhook rejects resource but error message is unclear

Solution:

Check operator logs for detailed validation errors:

kubectl logs -n <namespace> deployment/<release>-dynamo-operator | grep "validate create\|validate update"

Webhook logs include:

Resource name and namespace
Validation errors with context
Warnings for immutable field changes

Stuck Deleting Resources

Symptoms:

Resource stuck in "Terminating" state
Webhook blocks finalizer removal

Solution:

The webhook automatically skips validation for resources being deleted. If stuck:

Check if webhook is blocking:

kubectl describe <resource-type> <name> -n <namespace>
# Look for events mentioning webhook errors

Temporarily work around the webhook:

# Option 1: Set failurePolicy to Ignore
kubectl patch validatingwebhookconfiguration <name> \
  --type='json' \
  -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"}]'

# Option 2 (last resort): Delete ValidatingWebhookConfiguration
kubectl delete validatingwebhookconfiguration <name>

Delete resource again:

kubectl delete <resource-type> <name> -n <namespace>

Restore webhook configuration:

helm upgrade <release> dynamo-platform -n <namespace>

Best Practices

Production Deployments

✅ Use failurePolicy: Fail (default) to ensure validation is enforced
✅ Monitor webhook latency - Validation adds ~10-50ms per resource operation
✅ Automatic certificates work well for production - The built-in cert-controller handles generation and rotation; use cert-manager only if you need integration with organizational PKI
✅ Test webhook configuration in staging before production

Development Deployments

✅ Use failurePolicy: Ignore if webhook availability is problematic during development
✅ Keep automatic certificates (zero configuration, built into the operator)

Multi-Tenant Deployments

✅ Deploy one cluster-wide operator for platform-wide validation
~~Deploy namespace-restricted operators for tenant-specific namespaces~~ (DEPRECATED - use cluster-wide mode instead)

Additional Resources

Support

For issues or questions:

Check Troubleshooting section
Review operator logs: kubectl logs -n <namespace> deployment/<release>-dynamo-operator
Open an issue on GitHub

Обзор​

Ключевые возможности​

Текущие типы webhook'ов​

Архитектура​

Поток admission​

Обновление с версий, где webhook.enabled: false​

Конфигурация​

Варианты управления сертификатами​

Расширенная конфигурация​

Complete Configuration Reference​

Failure Policy​

Namespace Filtering​

Certificate Management​

Automatic Certificates (Default)​

How It Works​

Certificate Validity​

Smart Certificate Management​

Manual Certificate Rotation​

cert-manager Integration​

Prerequisites​

Configuration​

How It Works​

When to Use cert-manager​

Certificate Rotation​

Example: Self-Signed Issuer​

External Certificates​

Steps​

Certificate Requirements​

Multi-Operator Deployments (DEPRECATED)​

Scenario​

How It Works​

Lease Configuration​

Deployment Example​

ValidatingWebhookConfiguration Naming​

Lease Health​

Troubleshooting​

Webhook Not Called​

Connection Refused Errors​

Certificate Errors​

Certificate Controller Errors​

Validation Errors Not Clear​

Stuck Deleting Resources​

Best Practices​

Production Deployments​

Development Deployments​

Multi-Tenant Deployments​

Additional Resources​

Support​

Обзор

Ключевые возможности

Текущие типы webhook'ов

Архитектура

Поток admission

Обновление с версий, где `webhook.enabled: false`

Конфигурация

Варианты управления сертификатами

Расширенная конфигурация

Complete Configuration Reference

Failure Policy

Namespace Filtering

Certificate Management

Automatic Certificates (Default)

How It Works

Certificate Validity

Smart Certificate Management

Manual Certificate Rotation

cert-manager Integration

Prerequisites

Configuration

How It Works

When to Use cert-manager

Certificate Rotation

Example: Self-Signed Issuer

External Certificates

Steps

Certificate Requirements

Multi-Operator Deployments (DEPRECATED)

Scenario

How It Works

Lease Configuration

Deployment Example

ValidatingWebhookConfiguration Naming

Lease Health

Troubleshooting

Webhook Not Called

Connection Refused Errors

Certificate Errors

Certificate Controller Errors

Validation Errors Not Clear

Stuck Deleting Resources

Best Practices

Production Deployments

Development Deployments

Multi-Tenant Deployments

Additional Resources

Support