NixOS VM tests fail on aarch64-darwin: Determinate linux-builder cannot allocate PTYs

[gotha]

Summary

On aarch64-darwin hosts, running the flake's NixOS VM checks (nix build .#checks.x86_64-linux.<test> / .#checks.aarch64-linux.<test>) against the Determinate Nix linux-builder fails before any VM boots. The Python test-driver that orchestrates each nixosTest cannot allocate a pseudo-terminal, which vde_switch (the virtual-LAN helper) requires to attach the guest's serial console.

This blocks the entire checks.*-linux.* attribute set from running locally on macOS, even though all tests pass on native Linux and in CI.

Environment

• Host: aarch64-darwin (Apple Silicon)
• Nix: Determinate Nix 3.15.2 (nix 2.33.1)
• Builder in use: Determinate's bundled linux-builder (external-builders = [{"program":"/usr/local/bin/determinate-nixd","systems":["aarch64-linux","x86_64-linux"]}])

Reproduction

Minimal upstream nixosTest (no agentbox code needed):

nix build --impure --no-link --max-jobs 1 --expr '
  let pkgs = (builtins.getFlake "github:NixOS/nixpkgs/nixos-25.11").legacyPackages.x86_64-linux;
  in pkgs.testers.nixosTest {
    name = "pty-smoke";
    nodes.machine = { ... }: { };
    testScript = "machine.succeed(\"true\")";
  }'

Fails with:

OSError: out of pty devices
  at pty.openpty() inside the test-driver
  vde_switch fails to start -> no guest networking -> test aborts

Root Cause

The Determinate linux-builder sandbox does not expose a working devpts mount to builds. pty.openpty() (used by the NixOS test-driver to wire each node's console and by vde_switch for its management socket) returns ENOENT/out of pty devices because no PTY multiplexer is available inside the build environment. Every nixosTest derivation ends up running its test-driver inside that sandbox, so the failure is universal, not test-specific.

Impact

• All 12 flake checks (boot, project-*, host-shares, tools-*) are unrunnable locally on macOS via the default builder.
• Developers on Apple Silicon cannot iterate on VM tests without a workaround.
• Emulated aarch64-linux (same Determinate builder, different system) is affected identically.

Temporary Mitigation

Route nixos-test–tagged derivations to a self-hosted remote Linux builder over ssh-ng://, while leaving the rest of the Linux builds on Determinate. The remote host has a real /dev/pts, so PTY allocation succeeds and all 12 checks pass.

Prerequisites

• A reachable x86_64-linux machine with a working Nix installation where your user is in trusted-users.
• SSH connectivity from your Mac to that host (jump host/bastion is fine — configure via ~/.ssh/config).

Steps

1. Provision a root-owned SSH identity so the Nix daemon (running as root) can authenticate:

   sudo install -d -m 700 -o root -g wheel /var/root/.ssh
   sudo install -m 600 -o root -g wheel ~/.ssh/id_rsa /var/root/.ssh/nix_builder_key

2. Write a root-level SSH config pinning the key (adjust host names):

   sudo tee /var/root/.ssh/config >/dev/null <<'EOF'
   Host bastion
     HostName bastion
     IdentityFile /var/root/.ssh/nix_builder_key
     IdentitiesOnly yes
     UserKnownHostsFile /var/root/.ssh/known_hosts
     StrictHostKeyChecking accept-new
   
   Host linux-builder
     HostName <remote-ip-or-hostname>
     ProxyJump bastion
     IdentityFile /var/root/.ssh/nix_builder_key
     IdentitiesOnly yes
     UserKnownHostsFile /var/root/.ssh/known_hosts
     StrictHostKeyChecking accept-new
   EOF
   sudo chmod 600 /var/root/.ssh/config
   sudo touch /var/root/.ssh/known_hosts && sudo chmod 600 /var/root/.ssh/known_hosts

3. Register the builder in /etc/nix/machines scoped to nixos-test only, so normal Linux builds continue to use Determinate:

   sudo tee -a /etc/nix/machines >/dev/null <<'EOF'
   ssh-ng://<user>@linux-builder x86_64-linux /var/root/.ssh/nix_builder_key 8 100 nixos-test,kvm,big-parallel,benchmark nixos-test -
   EOF

   The 7th field (nixos-test) is mandatoryFeatures — this builder is only eligible for derivations that require it, leaving everything else on Determinate.

4. Enable substituter use on the remote via Determinate's user-writable config (editing /etc/nix/nix.conf directly is wiped on daemon restart):

   echo 'builders-use-substitutes = true' | sudo tee -a /etc/nix/nix.custom.conf

5. Restart the daemon and verify:

   sudo launchctl kickstart -k system/systems.determinate.nix-daemon
   nix show-config | grep -E '^(builders |builders-use)'
   sudo -H nix store info --store ssh-ng://<user>@linux-builder   # expect Trusted: 1

6. Run the tests:

   nix build .#checks.x86_64-linux.boot   # should build on the remote and copy result back

Acceptance for Permanent Fix

• Track upstream Determinate issue for PTY support in the linux-builder sandbox.
• Once fixed, the workaround above can be removed (delete the /etc/nix/machines entry; /etc/nix/nix.custom.conf override is harmless to keep).
• Alternative long-term options to evaluate: (a) shipping a flake.nix dev-shell that sets up a nix-darwin linux-builder VM with proper devpts, (b) documenting CI-only execution for these checks.

References

• Determinate Nix linux-builder: https://docs.determinate.systems/determinate-nix#linux-builder
• Upstream NixOS test driver PTY usage: nixpkgs/nixos/lib/test-driver/test_driver/vlan.py
• ssh-ng:// remote builder docs: https://nix.dev/manual/nix/latest/advanced-topics/distributed-builds