spec: minimal set of os-specific device nodes

[vcaputo]

For the apps to be able to have reasonable expectations of what to find at places like /dev and /proc, the spec should probably explicitly state what the execution environment must provide.

For linux we could just conform to http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/, though there's quite a bit going on there, some of which probably isn't necessary in the app's root. Perhaps we should instead require LSB conformance: http://refspecs.linuxfoundation.org/LSB_2.0.0/LSB-Core/LSB-Core/execenvfhs.html

What about the other operating systems?

Is it considered beyond the scope of the spec?

[alban]

Yes I think the spec should specific that. There is another list of mount points and device nodes:
https://github.com/docker/libcontainer/blob/master/SPEC.md#filesystem

[jonboulle]

Hijacking this issue!

Per out-of-band discussion: we reached a consensus that of the various device-file related possibilities this approach probably makes the most sense.

To restate: within the spec, we define for each OS a minimum set of device files that need to be provided by the executor (i.e. made available in an app's chroot filesystem). So on Linux, this might look like for example the LSB standard mentioned above.

@mpasternacki would you be able to help out on the BSD side for this?

[mpasternacki]

On FreeBSD side (jails are FreeBSD-specific, not a generic BSD mechanism), I would need to check out the standard, but here is what I gathered from practice:

Side Note: for FreeBSD guests, it doesn't seem to make much sense not to have whole basesystem (base.txz) inside the container's chroot. Because of how the system is developed (userspace basesystem in lockstep with kernel), much of the software - including pieces of basesystem and stdlib - relies on basesystem being present: directory structure, binaries, files, etc. Trying to get things working without a basesystem is going to be difficult, and I wouldn't expect any of the system maintainers to help or support that. Given that there is no disk space gained by not having all images include the basesystem (because of ZFS copy-on-write clones), and that the basesystem can be considered secure (if there is a security issue in the basesystem, I have bigger problems outside the jails), I just go with what the full basesystem as the supported basis of FreeBSD guests.

FreeBSD, though, uses a devfs (https://www.freebsd.org/cgi/man.cgi?query=devfs&sektion=5 https://www.freebsd.org/cgi/man.cgi?query=devfs&sektion=8&apropos=0&manpath=FreeBSD+10.1-RELEASE) to manage device nodes. Supplied devfs.conf includes a ruleset number 4 that is supposed to be used inside jails. As a default, I use devfs_ruleset="4"; in generated jail.conf; as almost any jail.conf setting, it can be overridden with an annotation, though. What I see in a new container with base system is:

root@39f840dc-360d-473d-a9cd-15613086eb82:~ # ls -l /dev
total 1
dr-xr-xr-x  2 root  wheel      512 Feb 12 11:50 fd
crw-rw-rw-  1 root  wheel     0x1f Feb 12 11:51 null
dr-xr-xr-x  2 root  wheel      512 Feb 12 11:50 pts
crw-r--r--  1 root  wheel      0x8 Feb 11 13:28 random
lrwxr-xr-x  1 root  wheel        4 Feb 12 11:50 stderr -> fd/2
lrwxr-xr-x  1 root  wheel        4 Feb 12 11:50 stdin -> fd/0
lrwxr-xr-x  1 root  wheel        4 Feb 12 11:50 stdout -> fd/1
lrwxr-xr-x  1 root  wheel        6 Feb 12 11:50 urandom -> random
crw-rw-rw-  1 root  wheel     0x20 Feb 11 13:28 zero
crw-rw-rw-  1 root  operator  0xf4 Feb 11 13:28 zfs

For FreeBSD, I'd defer to system-supplied /etc/defaults/devfs.rules ruleset 4 rather than specify a particular set of nodes. Devfs would also make it difficult to explicitly support deviceFiles entries: it would require either skipping devfs entirely and explicitly creating device nodes, or writing a custom ruleset (which are, as far as I understand, global). The notion of deviceFiles is also very Unix-centric; if an application needs particular device files, shouldn't the image tarball include them?

[vcaputo]

The notion of deviceFiles is also very Unix-centric; if an application needs particular device files, shouldn't the image tarball include them?

One problem with expecting the ACI to bring it's own device nodes is it must bring (predict) the correct major:minor numbers. If the devices are only named and left for the ACE it's more robust, the ACE can simply lookup the names in /dev to find the major:minor numbers.

[alban]

In Rocket, systemd-nspawn already mounts a few filesystems in the container namespace:
/proc as procfs, /dev as tmpfs etc:
http://cgit.freedesktop.org/systemd/systemd/tree/src/nspawn/nspawn.c#n799

But they are not easily visible by the apps because they are "protected" by the chroot.

$ sudo nsenter -t $(pidof /usr/lib/systemd/systemd)  -m /bin/bash -c 'test -c /dev/null' && echo exists
exists
$