bwrap has impact on the shell when finished

[immae]

It’s a bit difficult to explain the issue, so I’m giving a script below to reproduce it. It arises with docker but also "other container tools" (my first occurence occured with machinectl, but docker allows a more minimal example)

#!/usr/bin/env bash

if ! command -v bwrap || ! command -v docker; then
  echo "bubblewrap and docker are needed for this example"
  exit 1
fi

# Make sure we don't leave some running containers behind
container_id=$(docker run -d -it alpine)
trap "docker stop $container_id" EXIT

# This command works
echo "This docker command works"
docker exec -it "$container_id" ls -a

echo "Please exit the shell below to continue"
bwrap --unshare-pid --ro-bind-try /bin /bin --ro-bind-try /lib64 /lib64 --ro-bind-try /lib /lib --ro-bind-try /usr /usr --ro-bind-try /nix /nix -- /bin/sh

echo "This docker command doesn't work"
# Bash "backgrounds" the command instead of running docker. We are
# obliged to run "fg" to let it continue
docker exec -it "$container_id" ls -a

When I run this script, I get the following output:

host $ ./example.sh 
This docker command works
(some ls output)
Please exit the shell below to continue
$ 
This docker command doesn't work

[1]+  Stopped                 ./example.sh
host $ fg
./example.sh
(some ls output)
7e75ced3ecdf227d1eaad68d6dfc3e44587ae483ec124e09937628238a4d6560

The issue only happens when the "--unshare-pid" flag is present on bwrap (the rest of the flags is just to get a MVE). I could reproduce the issue on two different kinds of systems (debian and nixos). "Normal" (non-container related) commands work just fine. Also, we need to get an interactive shell with bwrap, otherwise the issue doesn’t arise either.

I searched for some information regarding that kind of behavior in vain, any pointer to an explanation / solution would be appreciated

[immae]

actually the script below is even more minimalist and doesn’t require docker:

#!/usr/bin/env bash

echo "Please exit the shell below to continue"
bwrap --unshare-pid --ro-bind-try /bin /bin --ro-bind-try /lib64 /lib64 --ro-bind-try /lib /lib --ro-bind-try /usr /usr --ro-bind-try /nix /nix -- /bin/sh

echo "This command doesn't work"
bwrap --ro-bind-try /bin /bin --ro-bind-try /lib64 /lib64 --ro-bind-try /lib /lib --ro-bind-try /usr /usr --ro-bind-try /nix /nix -- /bin/sh

(note that the second invocation doesn't use --unshare-pid. And it’s necessary to run as a shell script, it doesn’t happen in an interactive shell)

And I tried on a third system (which happened to have a zsh shell) and I got this line which could be a hint:
/bin/sh: 0: Cannot set tty process group (No such process)

[GalaxySnail]

I can reproduce this issue. Run the following script:

#!/bin/bash
echo "--- ps before 1st bwrap ---"
ps -o pid,ppid,pgid,sid,tpgid,args

( sleep 1; echo "--- ps while 1st bwrap running ---"; ps -o pid,ppid,pgid,sid,tpgid,args ) &
bwrap --unshare-pid --ro-bind /usr /usr --symlink usr/bin /bin --symlink usr/lib /lib --symlink usr/lib64 /lib64 --ro-bind /etc /etc --dev /dev --proc /proc --tmpfs /tmp --setenv PS1 'bwrap1$ ' -- /bin/sh

echo "--- ps after 1st bwrap ---"
ps -o pid,ppid,pgid,sid,tpgid,args

( sleep 1; echo "--- ps while 2nd bwrap running ---"; ps -o pid,ppid,pgid,sid,tpgid,args ) &
bwrap --ro-bind /usr /usr --symlink usr/bin /bin --symlink usr/lib /lib --symlink usr/lib64 /lib64 --ro-bind /etc /etc --dev /dev --proc /proc --tmpfs /tmp --setenv PS1 'bwrap2$ ' -- /bin/sh

echo "--- ps after 2nd bwrap ---"
ps -o pid,ppid,pgid,sid,tpgid,args

The result is:

[galaxysnail@myhostname ~]$ bash /tmp/tmp.WGMbeOTIcP/foo.sh
--- ps before 1st bwrap ---
    PID    PPID    PGID     SID   TPGID COMMAND
   4866    4865    4866    4866    4960 bash -l
   4960    4866    4960    4866    4960 bash /tmp/tmp.WGMbeOTIcP/foo.sh
   4961    4960    4960    4866    4960 ps -o pid,ppid,pgid,sid,tpgid,args
bwrap1$ --- ps while 1st bwrap running ---
    PID    PPID    PGID     SID   TPGID COMMAND
   4866    4865    4866    4866    4966 bash -l
   4960    4866    4960    4866    4966 bash /tmp/tmp.WGMbeOTIcP/foo.sh
   4962    4960    4960    4866    4966 ps -o pid,ppid,pgid,sid,tpgid,args
   4964    4960    4960    4866    4966 bwrap --unshare-pid --ro-bind /usr /usr --symlink
   4965    4964    4960    4866    4966 bwrap --unshare-pid --ro-bind /usr /usr --symlink
   4966    4965    4966    4866    4966 /bin/sh

bwrap1$ ps -o pid,ppid,pgid,sid,tpgid,args
    PID    PPID    PGID     SID   TPGID COMMAND
      1       0       0       0       3 bwrap --unshare-pid --ro-bind /usr /usr --symlink
      2       1       2       0       3 /bin/sh
      3       2       3       0       3 ps -o pid,ppid,pgid,sid,tpgid,args
bwrap1$ exit
/bin/sh: 3: Cannot set tty process group (No such process)
--- ps after 1st bwrap ---
    PID    PPID    PGID     SID   TPGID COMMAND
   4866    4865    4866    4866    4966 bash -l
   4960    4866    4960    4866    4966 bash /tmp/tmp.WGMbeOTIcP/foo.sh
   4968    4960    4960    4866    4966 ps -o pid,ppid,pgid,sid,tpgid,args

[1]+  Stopped                    bash /tmp/tmp.WGMbeOTIcP/foo.sh

[galaxysnail@myhostname ~]$ ps -o pid,ppid,pgid,sid,tpgid,args
    PID    PPID    PGID     SID   TPGID COMMAND
   4866    4865    4866    4866    4983 bash -l
   4960    4866    4960    4866    4983 bash /tmp/tmp.WGMbeOTIcP/foo.sh
   4969    4960    4960    4866    4983 bash /tmp/tmp.WGMbeOTIcP/foo.sh
   4970    4960    4960    4866    4983 bwrap --ro-bind /usr /usr --symlink usr/bin /bin
   4971    4969    4960    4866    4983 sleep 1
   4972    4970    4960    4866    4983 /bin/sh
   4983    4866    4983    4866    4983 ps -o pid,ppid,pgid,sid,tpgid,args

[galaxysnail@myhostname ~]$ fg
bash /tmp/tmp.WGMbeOTIcP/foo.sh
bwrap2$ --- ps while 2nd bwrap running ---
    PID    PPID    PGID     SID   TPGID COMMAND
   4866    4865    4866    4866    4972 bash -l
   4960    4866    4960    4866    4972 bash /tmp/tmp.WGMbeOTIcP/foo.sh
   4969    4960    4960    4866    4972 ps -o pid,ppid,pgid,sid,tpgid,args
   4970    4960    4960    4866    4972 bwrap --ro-bind /usr /usr --symlink usr/bin /bin
   4972    4970    4972    4866    4972 /bin/sh

bwrap2$ exit
--- ps after 2nd bwrap ---
    PID    PPID    PGID     SID   TPGID COMMAND
   4866    4865    4866    4866    4960 bash -l
   4960    4866    4960    4866    4960 bash /tmp/tmp.WGMbeOTIcP/foo.sh
   4993    4960    4960    4866    4960 ps -o pid,ppid,pgid,sid,tpgid,args

Notice the TPGID. It looks like /bin/sh in bwrap changes the foreground process group, but fails to restore it because it's running in a pidns. After the first bwrap exits, the TPGID (4966) points to a process group that doesn't exist, and bash foo.sh is running in a background process group (4960). If I replace /bin/sh with something that doesn't support job control (e.g., python), the TPGID doesn't change.

[rusty-snake]

... and --unshare-pid --new-session fixes it.

Or starting it with a new tty/pty systemd-pty-forward bwrap --unshare-pid.

[rusty-snake]

I searched for some information regarding that kind of behavior in vain, any pointer to an explanation / solution would be appreciated

In addition to @GalaxySnail explanations about PGIDs. What happens from a "sandbox" PoV is that you grant access to the current tty/pty by passing STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO to the sandbox and don't use --add-seccomp-fd to restrict available ioctls on them.

[immae]

Thanks both for your explanations, I think I understood. IIUC, the sandbox grants access to the pty and bash does its bash job with it (with some ioctl commands), and when it leave it leaves the pty in whatever state it left it (something about process group id), and this state is broken for every tool that tries to do something related to job control (not sure what it means) following that, like docker, bwrap, sh.

The systemd-pty-forward hint works very well, so I can leave this that.

However, since bash "interactive" is able to restore the process group id (because the issue doesn’t happen when we run the commands in a normal interactive session), isn’t there a built-in way to do it in bash instead?

Additional question: shouldn’t it be bwrap’s job to restore this process group id at the end? rationale for this question: bwrap "knows" what the PGID is before running the command, so it should be able to restore it to the correct value if it changed?

[GalaxySnail]

and this state is broken for every tool that tries to do something related to job control (not sure what it means) following that, like docker, bwrap, sh.

Actually, every tool that reads from the controlling terminal in a background process group will receive SIGTTIN and will be stopped by default. See signal(7).

However, since bash "interactive" is able to restore the process group id (because the issue doesn’t happen when we run the commands in a normal interactive session), isn’t there a built-in way to do it in bash instead?

Interactive shells that support job control will try to restore the foreground process group using tcsetpgrp(3) at exit, but it fails if the pgid is from out of the current pidns.

$ strace -f -e ioctl -o >(grep PGRP - >&2) busybox sh
6734  ioctl(10, TIOCGPGRP, [6732])      = 0
6734  ioctl(10, TIOCSPGRP, [6734])      = 0
~ $ /bin/sleep 100
6735  ioctl(10, TIOCSPGRP, [6735])      = 0
^Z
6734  ioctl(10, TIOCSPGRP, [6734])      = 0
[1]+  Stopped                    /bin/sleep 100
~ $ fg
/bin/sleep 100
6734  ioctl(10, TIOCSPGRP, [6735])      = 0
^C
6734  ioctl(10, TIOCSPGRP, [6734])      = 0
~ $ exit
6734  ioctl(10, TIOCSPGRP, [6732])      = 0

$ bwrap --unshare-pid --ro-bind /usr /usr --symlink usr/bin /bin --symlink usr/lib /lib --symlink usr/lib64 /lib64 --ro-bind /etc /etc --dev /dev --proc /proc --tmpfs /tmp --setenv PS1 'bwrap$ ' \
  -- strace -f -e ioctl -o >(grep PGRP - >&2) busybox sh
4     ioctl(10, TIOCGPGRP, [0])         = 0
4     ioctl(10, TIOCSPGRP, [4])         = 0
bwrap$ /bin/sleep 100
5     ioctl(10, TIOCSPGRP, [5])         = 0
^Z
4     ioctl(10, TIOCSPGRP, [4])         = 0
[1]+  Stopped                    /bin/sleep 100
bwrap$ fg
/bin/sleep 100
4     ioctl(10, TIOCSPGRP, [5])         = 0
^C
4     ioctl(10, TIOCSPGRP, [4])         = 0
bwrap$ exit
4     ioctl(10, TIOCSPGRP, [0])         = -1 ESRCH (No such process)

I've also tested dash, bash, and zsh --no-rcs. All of them show similar results as busybox sh (except that you need /bin/ps and /bin/sleep in busybox instead of its builtin ps and sleep).

Additional question: shouldn’t it be bwrap’s job to restore this process group id at the end? rationale for this question: bwrap "knows" what the PGID is before running the command, so it should be able to restore it to the correct value if it changed?

I'm not sure whether it should be bwrap's job, as that would require bwrap to implement some job control mechanism. Simply setting the foreground process group once doesn't work, as shown in #735. The glibc's documentation has a section on this topic: 29.5 Implementing a Job Control Shell, but I haven't read it yet.