Debugging software in a containerised environment can be challenging. The software itself is generally stripped-down, in a minimal runtime environment with none of the usual auxiliary tools available. It runs with minimal permissions, and is generally not directly visible or accessible from sessions that have greater privileges.
Thankfully on Kubernetes, updates to the ephemeral containers feature and the addition of the --profile and --target options to kubectl debug have greatly simplified this process. There is now a fairly easy means of creating a privileged process context that shares a pid namespace with a specific target container. Older workarounds involving launching entirely separate pods, container-breaking and jumping through the node pid namespace are generally no longer required.
This article explores why it's necessary to use ephemeral containers, and how to do so in a typical Kubernetes environment.
TL;DR: To jump to the quick answer for gdb, go to section "Working gdb command" below.
Put the debugger in the target container
The most obvious way to debug a process in a container is to put the debugger, or an agent for it, in the same container as the debugee, but this is not generally simple as the target container:
- is typically very minimal or OS-less, with no auxiliary commands or libraries included;
- often has a read-only filesystem with no writeable locations;
- keeps container image size as small as possible; and
- generally runs with minimal privileges and capabilities
... meaning there will be no debugger install present, no way to install one, and insufficient permissions to run and attach one if one did exist on disk.
Sometimes it's feasible to embed a minimal debugger remote agent (such as gdb-server) in the container. It may also be possible to use built-in features of the programming language runtime to attach a remote debugger, but these are typically disabled for security reasons in production.
Build and deploy a debug version of the container
Much like development workflows may generate a separate debug build of a program, it is not uncommon to produce debug builds of containers. These builds will generally contain a debug-friendly build of the target program with full debug symbols and will include many more utilities and tools.
In cases where an issue is easily reproduced and/or it's acceptable to ship debug containers to a customer and run it with elevated privileges this approach may be acceptable.
But in many cases it's wholly unsuitable: the problem occurs only intermittently and must be debugged in-situ without disturbing the running program; the image may contain proprietary source code or debug data that can be used to reverse-engineer such code; the debug image may be too large; customer compliance and security requirements may prohibit delivery of custom images; etc.
Inject an ephemeral container for debugging
If debugging in place isn't feasible, it's reasonable to assume that a Kubernetes ephemeral container could be used to inject a more privileged sidecar container into the pod, with visibility into the target container's process namespace.
Ephemeral containers can be added to Pods after pod creation and may have greater privileges than the base Pod, so they can be useful for debugging. Pods are otherwise immutable, so ephemeral containers provide an important workaround when in-place inspection and analysis becomes necessary.
The kubectl debug command's --target option sets a targetContainerName on the ephemeralContainer's spec (not well documented at present). This causes the new ephemeral container to see processes from the target container without needing to break out into the node.
For example, to inject an ephemeral container running Ubuntu 24.04 into pod/target-pod so it can see pids from `target-container:
kubectl debug -it \
--image "ubuntu:24.04" --profile sysadmin \
-n default pod/target-pod --target target-container \
-- /bin/bashWorking around pod security context with ephemeral containers
However, if the target Pod has a .spec.securityContext set, you might find that your ephemeral container's processes are not as privileged as expected. If the Pod spec has:
spec:
securityContext:
fsGroup: 65534
runAsGroup: 65534
runAsNonRoot: true
runAsUser: 65534... then the ephemeral container will inherit these, resulting in a non-root process without the desired capabilities flags.
The kubectl debug command is missing any arguments to override the security context at container level; it assumes that privileged: true is sufficient. But a custom debug container template like:
# debug-container-template.yaml
imagePullPolicy: Always
securityContext:
privileged: true
runAsNonRoot: false
runAsUser: 0
runAsGroup: 0
allowPrivilegeEscalation: true... will solve the issue when passed to kubectl debug with the additional argument --custom debug-template.yaml
E.g.
kubectl debug -it \
--image "ubuntu:24.04" --profile sysadmin \
-n default pod/target-pod --target target-container \
--custom debug-template.yaml \
-- /bin/bashwill result in a privileged session:
# getpcaps $$
1037418: =epAttach a debugger from within the container
These instructions assume you're using gdb, but similar principles apply for other common debuggers like lldb and dlv.
I will be working on a postgres container managed by CloudnativePG (CNPG) in these examples, using an injected privileged ephemeral container. It will be launched with --target postgres to allow my debug container to share a pid namespace with the postgres container. For convenience I'm using an image pre-populated with gdb and other common tools.
In this case the target pid will be identified by starting a new psql session and running SELECT pg_backend_pid(); in it, then running SELECT pg_sleep(3600); to give it something to do, and leaving it running.
Simply attaching gdb by pid alone is likely to produce unhelpful results because the process is in a different mount namespace, so gdb won't find the process executable; e.g.:
# gdb -q -p 893620
Attaching to process 893620
No executable file now.
warning: Could not load vsyscall page because no executable was specified
0x00007f8117f4783a in ?? ()
(gdb) bt
#0 0x00007f8117f4783a in ?? ()
#1 0x000000000096cd6b in ?? ()
#2 0x0000000030ad7b38 in ?? ()
#3 0x00007ffced121a60 in ?? ()
...You'll need to explicitly specify the executable, even though it's always right there in /proc/$pid/exe. Just append the executable path /proc/$pid/exe:
# gdb -q -p 893620 /proc/893620/exe
Reading symbols from /proc/893620/exe...
Reading symbols from .gnu_debugdata for /proc/893620/exe...
(No debugging symbols found in .gnu_debugdata for /proc/893620/exe)
Attaching to program: /proc/893620/exe, process 893620
warning: Could not load shared library symbols for 53 libraries, e.g. /lib64/libzstd.so.1.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
0x00007f8117f4783a in ?? ()
(gdb) bt
#0 0x00007f8117f4783a in ?? ()
#1 0x000000000096cd6b in WaitEventSetWait ()
#2 0x0000000000a4708c in pg_sleep ()
#3 0x0000000000b18170 in FunctionCallInvokeCheckSPL ()
#4 0x000000000077e1bf in ExecInterpExpr.lto_priv.0 ()
#5 0x00000000007b7890 in ExecResult ()
#6 0x00000000007828eb in ExecProcNodeRowNum ()
#7 0x000000000078150a in standard_ExecutorRun ()
#8 0x00007f8112bb3b6d in ?? ()
#9 0x0000000030cc0758 in ?? ()
#10 0x00007ffced121d60 in ?? ()
#11 0x0000000000000000 in ?? ()Better, but:
- This is a binary without debug symbols so debug capabilities are limited; and
gdbcan't resolve addresses in libraries because it's not looking in the right place for the libraries due to the mount namespace
We need to tell gdb to also resolve library paths relative to the target executable. Add the option -iex 'set sysroot /proc/893620/root' to the command. You'll now get a lot of output, but still see a warning:
warning: File "/proc/893620/root/lib64/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/target/lib64:/target/usr/lib64".so lets do that, but wildcard it as `
-iex 'add-auto-load-safe-path /proc/893620/root/lib64/*'which results in a basically working gdb command.
Working gdb command
To attach to pid 893620 in the same pid namespace but a different mount namespace:
target_pid=893620;
gdb -q -p "${target_pid}" -iex "set sysroot /proc/${target_pid}/root" \
-iex "add-auto-load-safe-path /proc/${target_pid}/root/lib64/*" \
-iex 'set print symbol-loading off' \
"/proc/${target_pid}/exe"with output
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/proc/893620/root/lib64/libthread_db.so.1".
0x00007f8117f4783a in epoll_wait () from /proc/893620/root/lib64/libc.so.6
(gdb) bt
#0 0x00007f8117f4783a in epoll_wait () from /proc/893620/root/lib64/libc.so.6
#1 0x000000000096cd6b in WaitEventSetWait ()
#2 0x0000000000a4708c in pg_sleep ()
#3 0x0000000000b18170 in FunctionCallInvokeCheckSPL ()
#4 0x000000000077e1bf in ExecInterpExpr.lto_priv.0 ()
#5 0x00000000007b7890 in ExecResult ()
#6 0x00000000007828eb in ExecProcNodeRowNum ()
#7 0x000000000078150a in standard_ExecutorRun ()
#8 0x00007f8112bb3b6d in pgsm_ExecutorRun () from /proc/893620/root/usr/edb/as17/lib/edb_stat_monitor.so
#9 0x00007f8112ba60a9 in ews_ExecutorRun () from /proc/893620/root/usr/edb/as17/lib/edb_wait_states.so
#10 0x00007f8112b96170 in pgss_ExecutorRun () from /proc/893620/root/usr/edb/as17/lib/pg_stat_statements.so
#11 0x00007f810d56ec49 in pgqs_ExecutorRun () from /proc/893620/root/usr/edb/as17/lib/query_advisor.so
#12 0x000000000099cdb1 in PortalRunSelect ()
#13 0x000000000099ec06 in PortalRun ()
#14 0x0000000000993983 in exec_simple_query ()
#15 0x0000000000997540 in PostgresMain ()
#16 0x0000000000998525 in BackendMain ()
#17 0x00000000008eb590 in postmaster_child_launch.part ()
#18 0x00000000008f2609 in ServerLoop ()
#19 0x00000000008f33ee in PostmasterMain ()
#20 0x0000000000547cea in main ()Simple actions like backtraces (without arguments), function entry breakpoints, thread inspection etc will now work.
Missing debuginfo limits debugger capabilities
This remains limited by the lack of debuginfo. The target container has a read-only root file system and may have its package manager database deleted.
Messages like these (when not invoking gdb with -iex 'set print symbol-loading off') indicate limited debug capabilities:
(No debugging symbols found in .gnu_debugdata for /proc/893620/exe)Simple function breakpoints will work, function-level backtraces, thread inspection and raw memory inspection - but not a lot else.
For example, I can't step through a function, set line-level breakpoints, or view variables on the stack. If I cancel my pg_sleep() , run this in gdb:
(gdb) break exec_simple_query
Breakpoint 1 at 0x993600
(gdb) c
Continuing.and re-run my pg_sleep , I'll hit the breakpoint but not be able to do a lot with it:
Breakpoint 1, 0x0000000000993600 in exec_simple_query ()
(gdb) info locals
No symbol table info available.
(gdb) step
Single stepping until exit from function exec_simple_query,
which has no line number information.
...To fix this, it's necessary to inject debuginfo for the target process and the libraries it links to into your debug container and tell gdb how to find that.
If your organisation runs a debuginfod server and your gdb is built with debuginfod support, this should be straightforward: just configure gdb to connect to your debuginfod (and perhaps a debuginfod for the distro your container is based on, if any) and let it download the required symbols.
You should run a debuginfod.
If your organisation does not have a debuginfod, a lot more hoop-jumping may be required to install debug symbols, if external symbols are even available.
That's beyond the scope of this article, but is a topic I'd like to tackle in a subsequent one.
What if I can't use an ephemeral container?
There are more complex workarounds available if use of ephemeral containers is prevented, e.g. by an enforcing pod security policy or a container runtime that does not support targetContainerName.
I will explore how to achieve a similar effect with a separate kubectl debug Pod run on the same Node as the target container in a later article.
(The --target / targetContainerName feature is also silently ignored if the Pod has hostPID: true , but in this case there is no cross-namespace debugging since the container pid runs in the host's pid namespace directly).
Diving deeper
In some cases it's necessary to debug between processes in different pid and mount namespaces. This is trickier due to issues like a mismatched /proc file system and the pid seen by the debug target differing from the pids seen by the debugger. gdb may log a warning like
warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable. Connect to gdbserver inside the container.I will write about this, and workarounds for it, in future articles. In the meantime, if you find this article because of the above error, t his gdb feature request may be of interest. I included a detailed explanation and workaround steps there.
End notes
(1) chroot technically didn't create a new mount namespace, it just tried to restrict a process to seeing only a subtree of the single global mount namespace. With limited success.
(a) I know the syntax highlighting on this article is broken, and am working on getting it fixed.