Securing Production Debugging in Kubernetes

Summary

This architecture replaces broad, long-lived production access—such as cluster-admin privileges and shared bastion hosts—with a secure, just-in-time (JIT) debugging workflow. It utilizes an access gateway and short-lived, identity-bound credentials to ensure that production access is temporary, auditable, and strictly scoped.

Key Points

• Use Kubernetes RBAC to grant permissions to Groups or ServiceAccounts rather than individual users to facilitate centralized identity management.
• Implement short-lived authentication via OIDC-based exec credential helpers or the Kubernetes CertificateSigningRequest (CSR) API.
• Deploy an access broker to enforce granular session policies, such as restricting specific commands within an exec session, which standard RBAC cannot do.
• Utilize a just-in-time access gateway, deployed as an on-demand pod, to act as an SSH-style "front door" for cluster access.
• Leverage OpenSSH user certificates and hardware-backed private keys (e.g., YubiKey) to ensure credentials are tied to a specific identity and cannot be easily forged.
• Enable automated auditing by capturing all session activity through both gateway logs and Kubernetes API audit logs.

Technical Details

The architecture relies on Kubernetes RBAC as the primary authorization engine, but introduces an access broker to provide fine-grained control. While RBAC can authorize actions like pods/exec or pods/portforward, it lacks the capability to restrict specific commands executed within a session. The broker manages this by applying policies (maintained via code review in JSON or XML) that define permitted commands and whether requests require manual approval.

Authentication can be implemented through two primary patterns:
1. OIDC Tokens: Using the client.authentication.k8s.io/v1 API, a cred-helper can be configured in the kubeconfig to refresh short-lived tokens automatically with a defined TTL.
2. X.509 Client Certificates: Engineers generate a private key (ideally on a hardware token) and a CSR. This CSR is submitted to the certificates.k8s.io/v1 API with a specific expirationSeconds value (e.g., 1800 seconds) to ensure the certificate expires quickly.

The access gateway further restricts the blast radius by mapping authenticated identities to specific RoleBindings. These can be namespace-scoped (e.g., jit-debug roles limited to a single namespace) or cluster-scoped (e.g., jit-cluster-read for viewing nodes and namespaces), ensuring that even if a session is established, it cannot access resources outside the approved scope.

Impact / Why It Matters

Implementing this workflow eliminates the security risks associated with long-lived SSH keys and shared administrative accounts. It provides developers with a scalable way to perform production debugging while maintaining a strict principle of least privilege and a comprehensive audit trail.