VFIO Resource Manager

A comprehensive VFIO device management and resource allocation system for high-performance networking on Linux. Manages network devices across kernel drivers, VFIO passthrough, and DPDK userspace drivers with automatic capability detection, PTP clock synchronization, and queue-level resource sharing.

Architecture

This project consists of three components:

• vfiod: System daemon that manages device modes, DPDK initialization, and resource allocation
• vfioctl: User-facing CLI tool for device management and configuration
• libvfio: Core library with device discovery, capability detection, and IPC

Features

Device Mode Management

Support for three operational modes per device:

• Kernel Mode: Standard Linux kernel drivers (mlx5_core, i40e, ice, etc.)

  • Full networking stack support
  • Standard socket APIs
  • AF_XDP support for kernel bypass

• VFIO Mode: Bound to vfio-pci for VM passthrough

  • Direct device access via VFIO
  • IOMMU protection
  • Virtual PTP clock creation

• DPDK Mode: Bound to vfio-pci with DPDK initialization

  • Userspace packet processing
  • Hardware timestamping via DPDK
  • PTP clock synchronization (ptpsyncd integration)

Automatic Capability Detection

• Hardware Capabilities: Link speed, queue counts, PTP support, RDMA
• Driver-Specific Features: Bifurcated mode (Mellanox), hardware timestamps
• Dynamic Recalculation: Capabilities refresh on driver changes
• WiFi Support: Generation detection (WiFi 6E, WiFi 6, WiFi 5)

Resource Management (Coming Soon)

• Reservations: Pre-allocation by orchestrator
• Queue-Level Sharing: Multiple processes sharing device queues
• DPDK Primary Process: Shared by all workers (ptpsyncd, latency workers)
• Automatic Cleanup: Process tracking and resource reclamation

PTP Clock Integration

• Physical Clocks: /dev/ptp* for kernel-mode devices
• Virtual Clocks: Automatic creation for VFIO/DPDK devices
• ptpsyncd Integration: Hardware clock sync for DPDK devices
• Timestamp Offloading: Hardware TX/RX timestamps

Quick Start

Installation

# Build all components
cd /home/serle/projects/rust/vfio
cargo build --release

# Install binaries
sudo cp target/release/vfiod /usr/local/bin/
sudo cp target/release/vfioctl /usr/local/bin/

# Install systemd service
sudo cp vfiod/vfiod.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable vfiod
sudo systemctl start vfiod

Configuration

Configuration file: /etc/vfiod/devices.toml

[dpdk]
hugepages_2mb = 512  # 1GB of hugepages

[[devices]]
pci_address = "0000:01:00.0"
mode = "kernel"
vendor_device = "15b3:101f"
description = "Mellanox ConnectX-6 Lx"

[[devices]]
pci_address = "0000:21:00.0"
mode = "dpdk"
vendor_device = "8086:158b"
description = "Intel XXV710 for 25GbE SFP28"

Basic Usage

# List all network devices
vfioctl list

# List only DPDK devices
vfioctl list --vfio

# Show configuration
vfioctl config

# Set device mode
vfioctl set-mode 0000:01:00.0 dpdk

# Apply configuration (switch devices to configured modes)
vfioctl apply-config

Device Modes

Kernel Mode

Devices use standard kernel drivers for normal networking:

# Set device to kernel mode
vfioctl set-mode 0000:01:00.0 kernel

# Apply the change
vfioctl apply-config

The daemon will:

1. Unbind from vfio-pci (if needed)
2. Bind to original kernel driver (mlx5_core, i40e, etc.)
3. Recalculate capabilities for kernel driver
4. Discover network interface name

VFIO Mode

For VM passthrough without DPDK:

# Set device to VFIO mode
vfioctl set-mode 0000:01:00.0 vfio
vfioctl apply-config

The daemon will:

1. Unbind from kernel driver
2. Bind to vfio-pci
3. Set VFIO device permissions (0o666)
4. Create virtual PTP clock (if supported)

DPDK Mode

For userspace packet processing with DPDK:

# Set device to DPDK mode
vfioctl set-mode 0000:01:00.0 dpdk
vfioctl apply-config

The daemon will:

1. Bind device to vfio-pci (same as VFIO mode)
2. Initialize DPDK primary process
3. Setup hugepages
4. Configure device for DPDK
5. Enable hardware timestamping
6. Create virtual PTP clock for ptpsyncd

Supported Hardware

Network Interface Cards

• Intel: X710 (10G), XXV710 (25G), E810 (100G)

  • Driver: i40e, ice
  • Full PTP support
  • 64 RX/TX queues

• Mellanox/NVIDIA: ConnectX-4/5/6

  • Driver: mlx5_core
  • Bifurcated mode support
  • RDMA/ibverbs
  • 63 RX/TX queues

• Broadcom: NetXtreme-E series

  • Driver: bnxt_en

• Solarflare: SFC series

  • Driver: sfc
  • EF_VI, OpenOnload support

Wireless Cards

• Intel WiFi 6E: AX210, AX211, AX411
• Intel WiFi 6: AX200, AX201
• Intel WiFi 5: AC 9560, AC 9260, AC 8265

Architecture Details

Component Interaction

┌─────────────┐
│   vfioctl   │ ← User commands
└──────┬──────┘
       │ IPC (Unix socket)
       ↓
┌─────────────┐
│    vfiod    │ ← System daemon
│             │   - Device discovery
│             │   - Mode management
│             │   - DPDK primary
└──────┬──────┘
       │
       ↓
┌─────────────┐
│   libvfio   │ ← Core library
│             │   - Capability detection
│             │   - Configuration
└─────────────┘

DPDK Integration

When DPDK mode is active:

1. Primary Process: vfiod runs DPDK EAL in primary mode
2. Shared Memory: Hugepages shared across all processes
3. Port Initialization: Devices configured for hardware timestamps
4. Secondary Processes: ptpsyncd attaches as DPDK secondary
5. Virtual PTP: Clock device created for synchronization

Capability Detection

Detection happens at:

• Startup: All devices discovered and analyzed
• Driver Change: Automatic recalculation on bind/unbind
• Manual Refresh: Via apply-config command

Detection methods:

• Link Speed: ethtool → interface speed → PCI → device database
• Queue Counts: sysfs for kernel, DPDK API for vfio
• PTP Support: /sys/class/ptp enumeration
• Hardware Timestamps: ethtool feature flags
• RDMA: /sys/class/infiniband detection

Configuration File Format

DPDK Section

[dpdk]
hugepages_2mb = 512  # Number of 2MB hugepages (default: 512 = 1GB)
                     # Shared by vfiod primary and all secondary processes
                     # Increase for many workers or large packet buffers

Device Sections

[[devices]]
pci_address = "0000:01:00.0"     # PCI address
mode = "kernel"                   # kernel, vfio, or dpdk
vendor_device = "15b3:101f"       # vendor:device ID
description = "Mellanox ConnectX-6 Lx (auto-discovered)"

The configuration file is automatically updated when:

• New devices are discovered
• Modes are changed via vfioctl set-mode
• vfioctl apply-config is run

Requirements

System Requirements

• Linux kernel 5.0+ with VFIO support
• IOMMU enabled (VT-d for Intel, AMD-Vi for AMD)
  • BIOS: Enable VT-d/AMD-Vi
  • Kernel: intel_iommu=on or amd_iommu=on
• Hugepages configured (for DPDK mode)

Build Dependencies

• Rust 1.83+ (edition 2024)
• DPDK 23.11+ development packages
• pkg-config
• libdpdk-dev

Runtime Dependencies

• vfio-pci kernel module
• DPDK libraries (for DPDK mode)
• ethtool (for capability detection)

Development

Building

# Debug build
cargo build

# Release build
cargo build --release

# Build only specific component
cargo build -p vfiod
cargo build -p vfioctl
cargo build -p libvfio

Testing

# Run all tests
cargo test

# Run specific crate tests
cargo test -p libvfio

# Run with output
cargo test -- --nocapture

Documentation

# Generate and open documentation
cargo doc --open

# Document private items
cargo doc --document-private-items --open

Integration with Other Projects

ptpsyncd

The PTP synchronization daemon uses vfiod for:

• Device capability discovery
• DPDK secondary process attachment
• Virtual PTP clock access
• Hardware timestamp configuration

See /home/serle/projects/rust/ptp for ptpsyncd documentation.

latency-orchestrator

The latency testing orchestrator uses vfiod for:

• Device reservation and allocation
• Queue-level resource sharing
• DPDK device initialization
• Worker process coordination

See /home/serle/projects/rust/latency-orchestrator for orchestrator documentation.

Troubleshooting

Device binding fails

# Check IOMMU is enabled
dmesg | grep -i iommu

# Check vfio-pci module is loaded
lsmod | grep vfio

# Check device IOMMU group
ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group

DPDK initialization fails

# Check hugepages
cat /proc/meminfo | grep Huge

# Check DPDK library installation
pkg-config --libs libdpdk

# View vfiod logs
journalctl -u vfiod -f

Speed detection shows 0 or wrong value

The system tries multiple detection methods:

1. Current link speed (requires link up)
2. ethtool maximum supported speed (works link down)
3. PCI link speed (rough approximation)
4. Device database (final fallback)

If detection fails, check:

# Check ethtool output
ethtool <interface>

# Check if link is up
ip link show <interface>

License

MIT OR Apache-2.0

Contributing

See /home/serle/projects/rust/TODO.md for project status and planned features.