Hetzner Node Set-Up

Sets up some nodes in the Hetzner cloud for demo purposes, using the Hetzner Cloud provider for Terraform.

Prerequisites

• Terraform installed
• Hetzner Cloud API token obtained and exported
• Ansible installed (for provisioning the box with Docker, Java etc.)

brew install terraform
brew install ansible
export TF_VAR_hcloud_token=<YOUR TOKEN>

Set-Up

The repo is split into terraform/ (infrastructure) and ansible/ (provisioning). Run this once after a fresh check out:

terraform -chdir=terraform init

Custom SSH port and local values

The boxes listen on a custom, non-standard SSH port, and are created with an SSH key from your Hetzner project. These local values are kept out of this (public) repo in two gitignored files, created once from the committed templates:

cp terraform/local.auto.tfvars.example terraform/local.auto.tfvars   # ssh_port, ssh_key_name, ssh_private_key_path, servers
cp ansible/vars.local.yml.example       ansible/vars.local.yml         # ssh_port, private key path

Fill them in; ssh_port must match in both files. The port is used in three places:

• the Hetzner firewall rule (Terraform var.ssh_port);
• the boxes' sshd — Terraform renders terraform/cloud-init.yaml as user_data, which sets the Port, labels it for SELinux, and opens firewalld;
• the Ansible connection — ansible/group_vars/all.yml sets ansible_port from ssh_port.

ssh_key_name is the name of the SSH key in your Hetzner project. Terraform registers it for the server and injects its public half into cloud-init, which on first boot creates the build user, installs the key, disables root and password login, and moves sshd to the custom port. So the box is fully set up before Ansible ever connects — and Ansible only ever connects as build. servers is the map of boxes to create (see Running).

The ansible/hosts inventory is generated by Terraform from the created servers (template terraform/hosts.tftpl), so it is gitignored and never edited by hand. The port and key come from ansible/group_vars/all.yml and ansible/vars.local.yml, so the inventory needs only the host IPs.

Running

Create the servers. This also (re)generates the Ansible inventory ansible/hosts:

terraform -chdir=terraform apply -var="firewall_source_ip=$(dig +short txt ch whoami.cloudflare @1.0.0.1 | tr -d '"')"

To run more than one box, add entries to the servers map in terraform/local.auto.tfvars:

servers = {
  control = { server_type = "ccx43" }
  worker  = { server_type = "cx22" }
}

Provisioning Docker etc.

cloud-init already created build and hardened SSH at boot, and terraform apply generated the inventory — so provisioning is a single, idempotent playbook run from the ansible/ directory (it first verifies the SSH hardening, then installs packages):

cd ansible
ansible-playbook -i hosts playbook.yml

This is safe to re-run, and safe to run against a mix of existing and freshly added boxes — every box self-configures at boot, so there is no per-host ordering to worry about.

Using AWS (manual EC2, no Terraform)

The same Ansible setup works against an EC2 instance launched by hand — only the Hetzner bits (cloud-init bootstrap + the Terraform-generated inventory) need reproducing. aws/user-data.yaml is a ready-to-paste copy of the cloud-init that creates the build user and hardens sshd, so the playbook runs unchanged.

1. Key pair. Ansible authenticates as build with the key from ansible/vars.local.yml (ansible_ssh_private_key_file). cloud-init installs its public half on the box, so the AWS key pair chosen in the launch wizard is irrelevant (it only lands on the default fedora/ec2-user account). Reuse an existing key or generate one:

   ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_aws

2. Prepare user-data. Copy aws/user-data.yaml and substitute the two placeholders — __SSH_PORT__ (your sshd port, matching ssh_port in vars.local.yml) and __BUILD_SSH_PUBLIC_KEY__ (contents of the .pub from step 1). Keep the real port out of git: save the filled copy as aws/user-data.local.yaml (gitignored) or just paste it without saving.

3. Launch the instance in the EC2 console:

   • AMI: a Fedora-family image (Fedora Cloud / Rocky / Alma) — the roles use dnf and cloud-init uses firewalld + SELinux. Avoid Ubuntu.
   • Type: e.g. m7i.2xlarge (8 vCPU / 32 GB, fixed performance, no burst).
   • Root volume: gp3, ~30–60 GB — cheapest tier, with 3,000 IOPS / 125 MB/s baseline regardless of size (gp2 ties IOPS to capacity; skip it). If disk I/O is part of the benchmark, don't use the EBS root — pick a *d instance type with local NVMe (e.g. m7id.2xlarge) instead.
   • Advanced details → User data: paste the filled user-data from step 2.
   • Security group: add an inbound rule for TCP on your custom SSH port (not 22) from your IP — AWS filters ingress here, before the host firewall.

4. Point Ansible at the box. Edit ansible/hosts (no longer Terraform-managed in this flow):

   control ansible_host=<EC2 public IP or DNS>
   

5. Provision as usual from ansible/:

   ansible-playbook -i hosts playbook.yml

Troubleshooting SSH

Can't connect after the box was running fine before? Two AWS-specific gotchas, both of which present as a connection timeout (not "refused"):

• The public IP changed after a stop/start. Stopping and starting an instance (what you do to save money — a plain reboot keeps the IP) assigns a new public IP, since there's no Elastic IP attached. The old IP in ansible/hosts is now dead. Grab the new one from the console, or:

  aws ec2 describe-instances --query 'Reservations[].Instances[].PublicIpAddress'

  then update control ansible_host=... in ansible/hosts. To avoid this for good, allocate an Elastic IP and associate it — it survives stop/start and is free while the instance runs.

• The security-group source IP doesn't match yours. The console's "My IP" button fills in the IP as your browser appears to AWS, which is often not the address your SSH client egresses from — common when the console captured your IPv6 but ssh went out over IPv4, or a VPN/proxy is on one but not the other. Don't trust the button; read the real value from the same machine you SSH from and paste that /32:

  curl -4 ifconfig.me      # IPv4 — add a /32 rule for this
  curl -6 ifconfig.me      # IPv6 — add a /128 rule if you SSH over v6

Quick triage by error: Connection timed out → wrong target IP or SG blocking your client; Connection refused → reached the host, wrong port/sshd; Permission denied → networking is fine, it's a key/user problem.

Shell access

terraform apply also generates ansible/ssh_config, with port, user, key, and IP filled in per box, so to get a shell just run (from the repo root):

ssh -F ansible/ssh_config control

The same config works for scp — the control alias resolves the host, user, port, and key, so no need to repeat -P/-i:

# local -> remote
scp -F ansible/ssh_config ./localfile control:/remote/path/

# remote -> local
scp -F ansible/ssh_config control:/remote/path/file ./

Add -r for directories.

Keeping commands alive across disconnects

A command started in a plain SSH session is killed when the connection drops: the remote shell gets SIGHUP and passes it on to its jobs. For long-running benchmarks or profiling runs, start them inside tmux (installed by the base role) so they survive a dropped connection, a closed laptop, or a lost VPN.

The whole loop:

# 1. SSH into the box, then start a session:
tmux

# 2. Run whatever you want — it's a normal shell now:
./my-long-command

# 3. Detach, leaving it running: press Ctrl-b, release, then press d

Now you can disconnect freely; the command keeps running on the server. Later, SSH back in and reattach exactly where you left off, with live output:

tmux attach

The commands you'll actually use:

tmux              # start a new session
tmux attach       # reattach after reconnecting
tmux ls           # list running sessions
Ctrl-b  d         # detach (leave it running)

The one thing to internalise: Ctrl-b is a prefix — press and release it, then press the command key (so detach is Ctrl-b then d, not held together). If tmux attach says "no sessions," nothing is running — it either finished or was never started. You can ignore tmux's window/pane features entirely; none of them are needed just to keep a command alive.

To see earlier output after reattaching, enter copy mode with Ctrl-b [ and scroll up (q to exit). But the scrollback buffer is capped (2000 lines by default), so for the full stdout of a long run, capture it to a file instead — this survives a buffer overflow and the session being killed, not just a disconnect:

./my-long-command 2>&1 | tee run.log    # on screen now, full record in run.log