Document that using a firewall can cause tests to fail in non-obvious ways mentioning `host.docker.internal` (and how to work around)

[reivilibre]

I've been using the ufw firewall on my machine 'for security purposes'. Turns out this was blocking connections from the containers to host.docker.internal and causing some Synapse tests to fail.

Frustratingly, Synapse's error messages were pretty hopeless at giving any details (ConnectingCancelledError: HostnameAddress(hostname=b'host.docker.internal', port=33145) is not particularly insightful :/).

Nevertheless, I imagine this is not an uncommon setup and so it would be nice to bring attention to this as part of the instructions for running Complement?

I didn't know what host.docker.internal is; I now see that it that it should resolve to the address of the host (although it didn't resolve in an ad-hoc container, so I'm not sure).

I'll take a crack at this; ideally once I've found the right rules to work around it rather than disabling the whole thing.

[DMRobertson]

Rich wrote:

I seem to have this:

rav@fred:~$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
Anywhere on br-+           ALLOW       Anywhere                  
Anywhere on docker+        ALLOW       Anywhere                  
Anywhere (v6) on br-+      ALLOW       Anywhere (v6)             
Anywhere (v6) on docker+   ALLOW       Anywhere (v6)  

which I think I must have done somehow, to fix this
but I can never remember the right incantations for the ufw commandline

I'm not using ufw; I'm relying on systemd's firewalld. No idea as of yet.

[DMRobertson]

Applying the change described here made things suddenly work for me. I've no idea if it's appropriate to make this change now. It seems to be included in Docker 20.10, which is what happens to be running on my system.

complement/internal/docker/deployer.go

Lines 179 to 184 in 1986a59

	if runtime.GOOS == "linux" {
	// By default docker for linux does not expose this, so do it now.
	// When https://github.com/moby/moby/pull/40007 lands in Docker 20, we should
	// change this to be `host.docker.internal:host-gateway`
	extraHosts = []string{HostnameRunningComplement + ":172.17.0.1"}
	}

[squahtx]

ftr the firewall configuration I ended up with was:

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
...
172.17.0.1                 ALLOW       Anywhere
...
``

[richvdh]

@DMRobertson:

Applying the change described here made things suddenly work for me. I've no idea if it's appropriate to make this change now. It seems to be included in Docker 20.10, which is what happens to be running on my system.

complement/internal/docker/deployer.go

Lines 179 to 184 in 1986a59

	if runtime.GOOS == "linux" {
	// By default docker for linux does not expose this, so do it now.
	// When https://github.com/moby/moby/pull/40007 lands in Docker 20, we should
	// change this to be `host.docker.internal:host-gateway`
	extraHosts = []string{HostnameRunningComplement + ":172.17.0.1"}
	}

I'd only expect this to make a difference if for some reason linux had assigned a different address to your docker0 virtual network interface - otherwise host.docker.internal:host-gateway is the same as HostnameRunningComplement + ":172.17.0.1". (The comment predates #21 which replaced the string host.docker.internal with the global variable HostnameRunningComplement throughout the codebase, but appears to have forgotten this comment. It's unclear that we actually do want to use HostnameRunningComplement here, and rather this should actually be a hardcoded host.docker.internal.)

What does sudo ip addr ls dev docker0 show for you?

[DMRobertson]

@richvdh:

What does sudo ip addr ls dev docker0 show for you?

$ sudo ip addr ls dev docker0
126: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:77:fd:98:bf brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:77ff:fefd:98bf/64 scope link 
       valid_lft forever preferred_lft forever

[richvdh]

well, that would do it. Is something else listening on 172.17.0.1 ?

[DMRobertson]

Not AFAICS:

$ sudo lsof -i -P -n | \grep LISTEN
systemd-r    1555 systemd-resolve   11u  IPv4    22860      0t0  TCP *:5355 (LISTEN)
systemd-r    1555 systemd-resolve   13u  IPv6    22863      0t0  TCP *:5355 (LISTEN)
systemd-r    1555 systemd-resolve   17u  IPv4    22866      0t0  TCP 127.0.0.53:53 (LISTEN)
systemd-r    1555 systemd-resolve   19u  IPv4    22868      0t0  TCP 127.0.0.54:53 (LISTEN)
dnsmasq      2235         dnsmasq    6u  IPv4    19453      0t0  TCP 192.168.122.1:53 (LISTEN)
java       970680             dmr   35u  IPv6 16467276      0t0  TCP 127.0.0.1:6943 (LISTEN)
java       970680             dmr   54u  IPv6 16457363      0t0  TCP 127.0.0.1:63343 (LISTEN)
clamd      972136        clamscan    4u  IPv4 16510987      0t0  TCP 127.0.0.1:3310 (LISTEN)

[richvdh]

sorry, when I say "listen", I mean "is that address assigned to any interfaces". Is 172.17.0 listed anywhere in sudo ip addr ls ?

[richvdh]

(this is somewhat moot: clearly you do have a different address, and the fix is indeed to change the extraHosts mapping to host.docker.internal:host-gateway as the comment suggests. However, it is actually a separate issue to requests being blocked by the firewall, even if the results are the same.)

[DMRobertson]

sorry, when I say "listen", I mean "is that address assigned to any interfaces". Is 172.17.0 listed anywhere in sudo ip addr ls ?

Ahh. Not that I can see. I can see my VPN device on 172.20.0.62, but I don't think that's relevant(?)

$ sudo ip addr ls | \grep 172 -B2 -A3
6: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 172.20.0.62 peer 172.20.0.61/32 scope global noprefixroute tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::5ffb:7df:3008:f20a/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

(this is somewhat moot: clearly you do have a different address, and the fix is indeed to change the extraHosts mapping to host.docker.internal:host-gateway as the comment suggests. However, it is actually a separate issue to requests being blocked by the firewall, even if the results are the same.)

Agreed on all counts, thanks. I'll PR this change separately.