DNS not working in Docker Compose

[boostpaal]

DNS lookups fail in Docker Compose 2.3.

docker-compose.yml

version: '2.3'
services:
  gvisor_test:
    command: node /home/test.js
    image: node:8-alpine
    runtime: runsc
    volumes:
      - /home/ubuntu/compose/test.js:/home/test.js

test.js

const http = require('http')
http.get('http://www.google.com', res => console.log(res))

Error:

$ docker-compose up
Starting compose_gvisor_test_1 ... done
Attaching to compose_gvisor_test_1
gvisor_test_1  | events.js:183
gvisor_test_1  |       throw er; // Unhandled 'error' event
gvisor_test_1  |       ^
gvisor_test_1  | 
gvisor_test_1  | Error: getaddrinfo EAI_AGAIN www.google.com:80
gvisor_test_1  |     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:67:26)
compose_gvisor_test_1 exited with code 1

$ uname -a
Linux ubuntu-2 4.15.0-36-generic #39~16.04.1-Ubuntu SMP Tue Sep 25 08:59:23 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client:
Version:           18.09.0
API version:       1.39
Go version:        go1.10.4
Git commit:        4d60db4
Built:             Wed Nov  7 00:48:57 2018
OS/Arch:           linux/amd64
Experimental:      false

Server: Docker Engine - Community
Engine:
 Version:          18.09.0
 API version:      1.39 (minimum version 1.12)
 Go version:       go1.10.4
 Git commit:       4d60db4
 Built:            Wed Nov  7 00:16:44 2018
 OS/Arch:          linux/amd64
 Experimental:     false

I've tried using the use-vc and other resolve options in Compose (http://man7.org/linux/man-pages/man5/resolv.conf.5.html) to force TCP over UDP, with no luck:

version: '2.3'
services:
  gvisor_test:
    command: node /home/test.js
    image: node:8-alpine
    runtime: runsc
    volumes:
      - /home/ubuntu/compose/test.js:/home/test.js
    dns_opt:
      - use-vc

[iangudger]

Can you post strace logs?
https://github.com/google/gvisor#debugging

[boostpaal]

https://gist.github.com/boostpaal/e8a03abdbd3c40ea8b633ec7b2b83873

[iangudger]

It appears that you are trying to connect to a DNS server on 127.0.0.11:53, but that there isn't a DNS server in the sandbox running on that port. When it comes to networking, the sandbox behaves like a VM. localhost is the sandbox's localhost. You might be able to talk to the DNS server on the host by using the host's IP address.

Another thing you can try is enabling network passthrough. This makes networking behave more like a normal container at the cost of security. I am not sure if this will help as I think the sandbox runs in a network namespace with its own loopback on the host.

[boostpaal]

Network passthrough works but it's not a viable option for security reasons. I seems like Compose's native DNS server is the root of the problem. There might be a missing DNS feature that gVisor requires. I think submitting an issue to Compose would be the right way to go from here.

[boostpaal]

The issue can be resolved with a workaround.

Changing the DNS server in Compose is not enough:

version: '2.3'
services:
  apiconbug:
    restart: always
    image: apiconbug
    tty: true
    working_dir: /home/alpine/apiconbug
    runtime: runsc
    dns:
        - 8.8.8.8

However, resolv.conf inside the container would still be:

nameserver 127.0.0.11

which would render the same problem.

When running Compose with runsc, you get two networks:

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
c9fcf0c8d7dc        bridge              bridge              local
29cf0bbb8e49        nodebug_default     bridge              local

The nodebug_default network is created by Compose when it has runsc as runtime, The Options object is empty:

$ docker network inspect 29cf0bbb8e49
[
    {
        "Name": "apiconbug_default",
        "Id": "a3fcbc8dbf8c8500ff9eee3f4ca7328f7b1dd181ad601168ccbe48be4fee99ac",
        "Created": "2018-11-30T09:50:39.643998227+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},  // <--- EMPTY
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "apiconbug",
            "com.docker.compose.version": "1.22.0"
        }
    }
]

The bridge network has these Options:

$ docker network inspect c9fcf0c8d7dc
[
    {
        "Name": "bridge",
        ...
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        ...
    }
]

We then switched to using the bridge network:

version: '2.3'
services:
  apiconbug:
    restart: always
    image: apiconbug
    tty: true
    working_dir: /home/alpine/apiconbug
    runtime: runsc
    dns:
        - 8.8.8.8
    network_mode: "bridge"

and voila, resolv.conf:

nameserver 8.8.8.8

DNS lookups are now OK.

Would be nice to know which of these options resolve the problem, what the problem actually is, and how to configure these options in Compose.

We could possibly have changed resolv.conf directly, but this file might be changed or overwritten by Docker and is not a viable solution.

This opens access to the hosts' local network, so be sure you know what you are exposing there.

[Anjali05]

I am getting error in apt update inside the ubuntu image when I run it with gvisor, with runc it work fine. Can someone tell me how to fix it?
content of the conf file inside the container:

root@d846d6c54be4:/# cat /etc/resolv.conf
search utah.cloudlab.us
nameserver 10.0.0.2
nameserver 8.8.8.8

error:
root@d846d6c54be4:/# apt update
Err:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu bionic-security InRelease
Temporary failure resolving 'security.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
root@d846d6c54be4:/#