Skip to content

Commit d06b90b

Browse files
drewdrew
committed
wip
1 parent d5fdd01 commit d06b90b

7 files changed

Lines changed: 441 additions & 39 deletions

File tree

‎crates/openshell-vm/README.md‎

Lines changed: 235 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,235 @@
1+
# openshell-vm
2+
3+
> Status: Experimental and work in progress (WIP). VM support is under active development and may change.
4+
5+
MicroVM runtime for OpenShell, powered by [libkrun](https://github.com/containers/libkrun). Boots a lightweight ARM64 Linux VM on macOS (Apple Hypervisor.framework) or Linux (KVM) running a single-node k3s cluster with the OpenShell control plane.
6+
7+
## Quick Start
8+
9+
Build and run the VM in one command:
10+
11+
```bash
12+
mise run vm
13+
```
14+
15+
This will:
16+
17+
1. Compress runtime artifacts (libkrun, libkrunfw, gvproxy, rootfs)
18+
2. Build the `openshell-vm` binary with embedded runtime
19+
3. Codesign it (macOS)
20+
4. Build the rootfs if needed
21+
5. Boot the VM
22+
23+
## Prerequisites
24+
25+
- **macOS (Apple Silicon)** or **Linux (aarch64 with KVM)**
26+
- Rust toolchain
27+
- [mise](https://mise.jdx.dev/) task runner
28+
- Docker (for rootfs builds)
29+
30+
### macOS-Specific
31+
32+
The binary must be codesigned with the Hypervisor.framework entitlement. The `mise run vm` flow handles this automatically. To codesign manually:
33+
34+
```bash
35+
codesign --entitlements crates/openshell-vm/entitlements.plist --force -s - target/debug/openshell-vm
36+
```
37+
38+
## Build
39+
40+
### Embedded Binary (Recommended)
41+
42+
Produces a single self-extracting binary with all runtime artifacts baked in:
43+
44+
```bash
45+
mise run vm:build:embedded
46+
```
47+
48+
On first run, the binary extracts its runtime to `~/.local/share/openshell/vm-runtime/<version>/`.
49+
50+
### Quick Rebuild (Skip Rootfs)
51+
52+
If you already have a cached rootfs tarball and just want to rebuild the binary:
53+
54+
```bash
55+
mise run vm:build:embedded:quick
56+
```
57+
58+
### Force Full Rebuild
59+
60+
Rebuilds everything including the rootfs:
61+
62+
```bash
63+
mise run vm:build
64+
```
65+
66+
## Run
67+
68+
### Default (Gateway Mode)
69+
70+
Boots the full OpenShell gateway --- k3s + openshell-server + openshell-sandbox:
71+
72+
```bash
73+
mise run vm
74+
```
75+
76+
Or run the binary directly:
77+
78+
```bash
79+
./target/debug/openshell-vm
80+
```
81+
82+
### Custom Process
83+
84+
Run an arbitrary process inside a fresh VM instead of k3s:
85+
86+
```bash
87+
./target/debug/openshell-vm --exec /bin/sh --vcpus 2 --mem 2048
88+
```
89+
90+
### Execute in a Running VM
91+
92+
Attach to a running VM and run a command:
93+
94+
```bash
95+
./target/debug/openshell-vm exec -- ls /
96+
./target/debug/openshell-vm exec -- sh # interactive shell
97+
```
98+
99+
### Named Instances
100+
101+
Run multiple isolated VM instances side-by-side:
102+
103+
```bash
104+
./target/debug/openshell-vm --name dev
105+
./target/debug/openshell-vm --name staging
106+
```
107+
108+
Each instance gets its own rootfs clone under `~/.local/share/openshell/openshell-vm/instances/<name>/`.
109+
110+
## CLI Reference
111+
112+
```
113+
openshell-vm [OPTIONS] [COMMAND]
114+
115+
Options:
116+
--rootfs <PATH> Path to aarch64 Linux rootfs directory
117+
--name <NAME> Named VM instance (auto-clones rootfs)
118+
--exec <PATH> Run a custom process instead of k3s
119+
--args <ARGS>... Arguments to the executable
120+
--env <KEY=VALUE>... Environment variables
121+
--workdir <DIR> Working directory inside the VM [default: /]
122+
-p, --port <H:G>... Port mappings (host_port:guest_port)
123+
--vcpus <N> Virtual CPUs [default: 4 gateway, 2 exec]
124+
--mem <MiB> RAM in MiB [default: 8192 gateway, 2048 exec]
125+
--krun-log-level <0-5> libkrun log level [default: 1]
126+
--net <BACKEND> Networking: gvproxy, tsi, none [default: gvproxy]
127+
--reset Wipe runtime state before booting
128+
129+
Subcommands:
130+
exec Execute a command inside a running VM
131+
```
132+
133+
## Rootfs
134+
135+
The rootfs is an aarch64 Ubuntu filesystem containing k3s, pre-loaded container images, and the OpenShell binaries.
136+
137+
### Full Rootfs (~2GB+)
138+
139+
Pre-initialized k3s cluster state for fast boot (~3-5s):
140+
141+
```bash
142+
mise run vm:build:rootfs-tarball
143+
```
144+
145+
### Minimal Rootfs (~200-300MB)
146+
147+
Just k3s + supervisor, cold starts in ~30-60s:
148+
149+
```bash
150+
mise run vm:build:rootfs-tarball:minimal
151+
```
152+
153+
## Custom Kernel (libkrunfw)
154+
155+
The stock libkrunfw (e.g. from Homebrew) lacks bridge, netfilter, and conntrack support needed for pod networking. OpenShell builds a custom libkrunfw with these enabled.
156+
157+
Build it:
158+
159+
```bash
160+
mise run vm:runtime:build-libkrunfw
161+
```
162+
163+
See [`runtime/README.md`](runtime/README.md) for details on the kernel config and troubleshooting.
164+
165+
## Architecture
166+
167+
```
168+
Host (macOS / Linux)
169+
openshell-vm binary
170+
├── Embedded runtime (libkrun, libkrunfw, gvproxy, rootfs.tar.zst)
171+
├── FFI: loads libkrun at runtime via dlopen
172+
├── gvproxy: virtio-net networking (real eth0 + DHCP)
173+
├── virtio-fs: shares rootfs with guest
174+
└── vsock: host-to-guest command execution (port 10777)
175+
176+
Guest VM (aarch64 Linux)
177+
PID 1: openshell-vm-init.sh
178+
├── Mounts filesystems, configures networking
179+
├── Sets up bridge CNI, generates PKI
180+
└── Execs k3s server
181+
├── openshell-server (gateway control plane)
182+
└── openshell-sandbox (pod supervisor)
183+
```
184+
185+
## Environment Variables
186+
187+
| Variable | When | Purpose |
188+
|----------|------|---------|
189+
| `OPENSHELL_VM_RUNTIME_COMPRESSED_DIR` | Build time | Path to compressed runtime artifacts |
190+
| `OPENSHELL_VM_RUNTIME_DIR` | Runtime | Override the runtime bundle directory |
191+
| `OPENSHELL_VM_DIAG=1` | Runtime | Enable diagnostic output inside the VM |
192+
193+
## mise Tasks Reference
194+
195+
| Task | Description |
196+
|------|-------------|
197+
| `vm` | Build and run the VM |
198+
| `vm:build` | Force full rebuild including rootfs |
199+
| `vm:build:embedded` | Build single binary with embedded runtime |
200+
| `vm:build:embedded:quick` | Build using cached rootfs tarball |
201+
| `vm:build:rootfs-tarball` | Build full rootfs tarball |
202+
| `vm:build:rootfs-tarball:minimal` | Build minimal rootfs tarball |
203+
| `vm:runtime:compress` | Compress runtime artifacts for embedding |
204+
| `vm:runtime:build-libkrunfw` | Build custom libkrunfw |
205+
| `vm:runtime:build-libkrun` | Build libkrun from source (Linux) |
206+
| `vm:runtime:build-libkrun-macos` | Build libkrun from source (macOS) |
207+
| `vm:check-capabilities` | Check VM kernel capabilities |
208+
209+
## Testing
210+
211+
Integration tests require a built rootfs and macOS ARM64 with libkrun:
212+
213+
```bash
214+
cargo test -p openshell-vm -- --ignored
215+
```
216+
217+
Individual tests:
218+
219+
```bash
220+
# Full gateway boot test (boots VM, waits for gRPC on port 30051)
221+
cargo test -p openshell-vm gateway_boots -- --ignored
222+
223+
# Run a command inside the VM
224+
cargo test -p openshell-vm gateway_exec_runs -- --ignored
225+
226+
# Exec into a running VM
227+
cargo test -p openshell-vm gateway_exec_attaches -- --ignored
228+
```
229+
230+
Verify kernel capabilities inside a running VM:
231+
232+
```bash
233+
./target/debug/openshell-vm exec -- /srv/check-vm-capabilities.sh
234+
./target/debug/openshell-vm exec -- /srv/check-vm-capabilities.sh --json
235+
```

‎crates/openshell-vm/pins.env‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,10 @@
1515
# 2. Run the relevant build script to verify.
1616
# 3. Commit pins.env alongside any script changes.
1717

18-
# ── k3s binary (arm64) ─────────────────────────────────────────────────
18+
# ── k3s binary ─────────────────────────────────────────────────────────
1919
K3S_VERSION="${K3S_VERSION:-v1.35.2+k3s1}"
2020
K3S_ARM64_SHA256="${K3S_ARM64_SHA256:-228809a7ef47d25c1bdbe746944931ec2fd2edf842b9cf50f1dd4f9ec2505b0e}"
21+
K3S_AMD64_SHA256="${K3S_AMD64_SHA256:-3ae8e35a62ac83e8e197c117858a564134057a7b8703cf73e67ce60d19f4a22b}"
2122

2223
# ── Base Docker image (digest-pinned) ──────────────────────────────────
2324
# Tag: nvcr.io/nvidia/base/ubuntu:noble-20251013

‎crates/openshell-vm/runtime/build-custom-libkrunfw.sh‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -149,7 +149,14 @@ else
149149
cat "$FRAGMENT" >> .config
150150
fi
151151
152-
make ARCH=arm64 olddefconfig
152+
# Detect the kernel ARCH value from the host (or krunvm guest) architecture.
153+
case "$(uname -m)" in
154+
aarch64) KARCH="arm64" ;;
155+
x86_64) KARCH="x86_64" ;;
156+
*) KARCH="$(uname -m)" ;;
157+
esac
158+
echo " Kernel ARCH: ${KARCH}"
159+
make ARCH="${KARCH}" olddefconfig
153160
154161
# Verify critical configs are set
155162
REQUIRED=(

0 commit comments

Comments
 (0)