Skip to content
/ ptp Public

PTP clock synchronization system for kernel-mode and DPDK/VFIO network devices

1 star 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

serle/ptp

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
configs
configs
 
 
docs
docs
 
 
libptp
libptp
 
 
ptpctl
ptpctl
 
 
ptpsyncd
ptpsyncd
 
 
scripts
scripts
 
 
.gitignore
.gitignore
 
 
Cargo.toml
Cargo.toml
 
 
README.md
README.md
 
 

Repository files navigation

PTP Clock Synchronization System

A high-performance PTP (Precision Time Protocol) synchronization daemon for Linux supporting both kernel-mode and DPDK/VFIO network devices. Provides drop-in replacement for ptp4l + phc2sys with unified management, automatic capability detection, and seamless integration with vfiod.

Architecture

This project consists of three components:

  • ptpsyncd: System daemon that synchronizes PTP hardware clocks
  • ptpctl: User-facing CLI tool for monitoring and control
  • libptp: Core library with types, servo controllers, and IPC protocol

Features

Complete PTP Stack Replacement

  • Drop-in Replacement: Replace ptp4l + phc2sys with single daemon
  • Network Mode: Sync to network grandmaster (PTP IEEE 1588)
  • Local Mode: Sync all devices to system time (CLOCK_REALTIME)
  • Unified Management: Single configuration file, one monitoring interface
  • Resilient: Continues syncing working devices if one fails

Multi-Backend Support

  • Kernel Backend: POSIX clock ioctls for kernel-mode devices

    • Full frequency and phase adjustment
    • PI servo controller for optimal tracking
    • Devices: /dev/ptp0, /dev/ptp1, etc.

  • DPDK Backend: DPDK timesync API for VFIO devices

    • Secondary process attachment (vfiod runs primary)
    • Virtual PTP clock support
    • Runtime capability detection
    • Phase-only or frequency adjustment based on driver

Automatic Discovery

  • Device Detection: Scans /sys/class/ptp for all PTP devices
  • Capability Detection: Runtime detection of freq vs phase-only
  • Backend Selection: Automatic kernel vs DPDK backend
  • Hot Reload: Rediscovers devices on SIGHUP

Integration with vfiod

  • Seamless Integration: Starts after vfiod, discovers all devices
  • Mode Awareness: Works with kernel, VFIO, and DPDK modes
  • Virtual Clocks: Uses virtual PTP clocks for VFIO devices
  • DPDK Secondary: Attaches to vfiod's DPDK primary process
  • IPC Pattern: Same stateless IPC pattern as vfiod

Quick Start

Installation

# Build all components
cd /home/serle/projects/rust/ptp
cargo build --release

# Install binaries
sudo cp target/release/ptpsyncd /usr/local/bin/
sudo cp target/release/ptpctl /usr/local/bin/

# Install systemd service
sudo cp ptpsyncd/ptpsyncd.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable ptpsyncd
sudo systemctl start ptpsyncd

Configuration

Configuration file: /etc/ptpsyncd/config.toml

[sync]
# Synchronization method
method = "local"              # "local" or "network"
interval_ms = 1000            # Sync interval (1000ms = 1Hz)
sync_threshold_ns = 100       # Threshold for considering "in sync"

# For network mode: specify which device syncs to network grandmaster
# network_master_device = "/dev/ptp0"

[servo]
# PI controller gains (only for frequency-capable backends)
kp = 0.7                      # Proportional gain
ki = 0.3                      # Integral gain

[discovery]
auto_discover = true          # Automatically discover all PTP devices
# exclude = ["ptp1", "ptp5"] # Optional: exclude specific devices

Basic Usage

# Check daemon status
ptpctl status

# List all managed PTP devices
ptpctl list

# Detailed device information
ptpctl list --detailed

# Reload configuration
sudo systemctl reload ptpsyncd
# or: sudo pkill -HUP ptpsyncd
# or: ptpctl reload

Synchronization Modes

Local Mode (Default)

All devices sync to system time (CLOCK_REALTIME):

[sync]
method = "local"

Use cases:

  • Local testing and development
  • Non-PTP networks
  • Ensuring all devices have consistent timestamps
  • Multi-interface performance testing

How it works:

  1. Read system time (CLOCK_REALTIME)
  2. Read each device's hardware clock
  3. Calculate offset and adjust device clock
  4. PI servo minimizes drift over time

Network Mode

One device syncs to network grandmaster, all others sync to it:

[sync]
method = "network"
network_master_device = "/dev/ptp0"

Use cases:

  • Production PTP deployments
  • Replacing ptp4l + phc2sys
  • Multi-device synchronization to PTP grandmaster
  • Distributed systems requiring network-wide time sync

How it works:

  1. Master device synced to network grandmaster (via ptp4l or hardware PTP)
  2. All other devices sync to master device
  3. Single sync tree: Grandmaster → Master device → All other devices

Deployment:

# Option 1: Use ptp4l on master device
sudo ptp4l -i enp1s0f0np0 -s -m

# Option 2: Hardware PTP (device syncs automatically)
# No additional setup required

# ptpsyncd syncs all other devices to master
sudo systemctl start ptpsyncd

Device Support

Network Interface Controllers

ptpsyncd works with all PTP-capable NICs, including:

  • Intel: X710, XXV710, E810

    • Kernel mode: i40e, ice drivers
    • DPDK mode: Full frequency adjustment support
    • PTP capabilities: Hardware TX/RX timestamps

  • Mellanox/NVIDIA: ConnectX-4/5/6

    • Kernel mode: mlx5_core driver
    • DPDK mode: Typically phase-only
    • PTP capabilities: Full hardware timestamping

  • Broadcom: NetXtreme-E series

    • Kernel mode: bnxt_en driver
    • PTP support varies by model

  • Solarflare: SFC series

    • Kernel mode: sfc driver
    • Enhanced PTP support

Device Modes

ptpsyncd automatically adapts to device mode:

Mode Driver PTP Device Backend Servo
Kernel mlx5_core, i40e /dev/ptp0 Kernel PI
VFIO vfio-pci /dev/ptp8 (virtual) Kernel PI
DPDK vfio-pci DPDK API DPDK PI or Phase-only

Note: VFIO mode uses virtual PTP clocks created by vfiod.

Architecture Details

Component Interaction

┌──────────────────────────────────────────────────────────┐
│                    ptpsyncd Daemon                        │
├──────────────────────────────────────────────────────────┤
│  Configuration (/etc/ptpsyncd/config.toml)               │
│                                                           │
│  ┌─────────────────────────────────────────────┐        │
│  │         Synchronization Coordinator          │        │
│  │                                               │        │
│  │  Master Clock (CLOCK_REALTIME or /dev/ptp0)  │        │
│  │            ↓                                  │        │
│  │  Device /dev/ptp4: KernelBackend + PI Servo │        │
│  │  Device /dev/ptp8: KernelBackend + PI Servo │        │
│  │  Device port0: DpdkBackend + PI Servo        │        │
│  │  Device port1: DpdkBackend + PhaseServo      │        │
│  │                                               │        │
│  │  Each device runs async servo loop (1000ms)  │        │
│  └─────────────────────────────────────────────┘        │
│                                                           │
│  IPC Server (/var/run/ptpsyncd.sock)                     │
│    └─ Handlers: Status, List, Reload, Version           │
└──────────────────────────────────────────────────────────┘
                          ↑
                          │ IPC (Unix socket)
                          │
                    ┌─────┴──────┐
                    │   ptpctl   │ ← User commands
                    │ (stateless) │
                    └────────────┘

Backend Abstraction

trait PtpBackend {
    /// Read current time from hardware clock
    fn read_time(&self) -> Result<i64>;

    /// Adjust clock frequency (only if supported)
    fn adjust_frequency(&mut self, ppb: i64) -> Result<()>;

    /// Adjust clock phase
    fn adjust_time(&mut self, delta_ns: i64) -> Result<()>;

    /// Query backend capabilities
    fn capabilities(&self) -> BackendCapabilities;
}

Implementations:

  • KernelBackend: Uses POSIX clock ioctls (clock_gettime, clock_adjtime)
  • DpdkBackend: Uses DPDK timesync API (rte_eth_timesync_*)

Servo Controllers

Automatically selected based on backend capabilities:

PI Servo (Proportional-Integral):

  • For frequency-capable backends
  • Uses both frequency and phase adjustments
  • Optimal for long-term stability
  • Integral term corrects for oscillator drift

Phase-Only Servo:

  • For backends without frequency adjustment
  • Uses only phase adjustments
  • Suitable for testing and short-term sync
  • Less stable over long periods

IPC Stateless Pattern

ptpsyncd follows the same IPC pattern as vfiod:

  • Daemon: Owns all state, manages hardware, runs continuously
  • CLI: Stateless IPC client, no hardware access, no persistent state
  • Benefits:
    • Remote monitoring possible
    • Clean separation of concerns
    • Easy systemd integration
    • No root required for monitoring

All ptpctl commands are IPC queries to the running daemon.

Integration with vfiod

Boot Sequence

System Boot
     │
     ↓
vfiod.service starts
     │ - Discovers all network devices
     │ - Applies configured modes (kernel/vfio/dpdk)
     │ - Initializes DPDK primary (if needed)
     │ - Creates virtual PTP clocks
     ↓
ptpsyncd.service starts (After=vfiod.service)
     │ - Discovers all PTP devices
     │ - Attaches as DPDK secondary (if needed)
     │ - Starts synchronization loops
     ↓
System Ready

Runtime Interaction

# User changes device mode
vfioctl set-mode 0000:21:00.0 dpdk
vfioctl apply-config
  ↓
# vfiod binds device to vfio-pci, initializes DPDK
  ↓
# ptpsyncd automatically rediscovers (sees DPDK device)
  ↓
ptpctl status
# Shows device now using DPDK backend

DPDK Process Model

┌─────────────────────────────────────────┐
│       vfiod (DPDK Primary Process)       │
│  - Initializes DPDK EAL                  │
│  - Configures devices                    │
│  - Creates shared hugepage memory        │
└─────────────────────────────────────────┘
                  │
                  │ Shared hugepages
                  │
┌─────────────────┴───────────────────────┐
│    ptpsyncd (DPDK Secondary Process)    │
│  - Attaches to primary process          │
│  - Accesses devices via DPDK API        │
│  - Reads timestamps, adjusts clocks     │
└─────────────────────────────────────────┘

Key points:

  • vfiod runs DPDK primary with file-prefix "vfiod"
  • ptpsyncd attaches as secondary with same file-prefix
  • Hugepages configured in /etc/vfiod/devices.toml
  • Both processes share same DPDK memory pool

Requirements

System Requirements

  • Linux kernel 4.15+ with PTP support
  • For DPDK devices: vfiod with DPDK mode enabled
  • For network mode: ptp4l or hardware PTP on master device

Build Dependencies

  • Rust 1.83+ (edition 2024)
  • DPDK 23.11+ development packages (for DPDK backend)
  • pkg-config
  • libdpdk-dev

Runtime Dependencies

  • For kernel backend: PTP-capable network driver
  • For DPDK backend: vfiod running with DPDK primary
  • For network mode: ptp4l or hardware PTP

Development

Building

# Debug build
cargo build

# Release build
cargo build --release

# Build specific component
cargo build -p ptpsyncd
cargo build -p ptpctl
cargo build -p libptp

Testing

# Run all tests
cargo test

# Run specific crate tests
cargo test -p libptp

# Run with output
cargo test -- --nocapture

Documentation

# Generate and open documentation
cargo doc --open

# Document private items
cargo doc --document-private-items --open

Use Cases

1. Complete PTP Stack Replacement

Replace ptp4l + phc2sys with ptpsyncd:

Before:

# Run ptp4l on each device
ptp4l -i enp1s0f0np0 &
ptp4l -i enp33s0f0np0 &

# Run phc2sys to sync to system
phc2sys -s enp1s0f0np0 -c CLOCK_REALTIME &
phc2sys -s enp33s0f0np0 -c CLOCK_REALTIME &

After:

# Single daemon, single config
sudo systemctl start ptpsyncd

# Monitor all devices
ptpctl status

2. Multi-Interface Performance Testing

Synchronize multiple NICs for fair testing:

# Ensure all test NICs have synchronized timestamps
ptpctl status
# Shows offset for each device

# Run tests knowing timestamps are synchronized
./latency-test --interface enp1s0f0np0 &
./latency-test --interface enp33s0f0np0 &

3. DPDK + Kernel Hybrid Deployments

Mix DPDK and kernel devices:

# /etc/vfiod/devices.toml
[[devices]]
pci_address = "0000:01:00.0"
mode = "dpdk"  # DPDK device

[[devices]]
pci_address = "0000:21:00.0"
mode = "kernel"  # Kernel device
# ptpsyncd syncs both automatically
ptpctl list
# Shows kernel backend for 01:00.0
# Shows DPDK backend for 21:00.0

4. Distributed System Time Synchronization

Sync all servers in cluster to network grandmaster:

# On each server:
# 1. Run ptp4l on one interface (or use hardware PTP)
ptp4l -i enp1s0f0np0 -s -m

# 2. ptpsyncd syncs all other devices to it
[sync]
method = "network"
network_master_device = "/dev/ptp0"

CLI Reference

ptpctl status

Show daemon and per-device sync status:

$ ptpctl status

ptpsyncd Status
================================================================================
Status: Running
Sync Method: local
Sync Interval: 1000ms
Devices Managed: 4

Device Sync Status
================================================================================
Device: /dev/ptp0  Interface: enp1s0f0np0  Backend: kernel
  Offset: +12 ns  Freq Adj: -45 ppb  State: LOCKED
  Last Sync: 0.5s ago

Device: port0  Interface: (DPDK)  Backend: dpdk
  Offset: -8 ns  Freq Adj: +32 ppb  State: LOCKED
  Last Sync: 0.3s ago

ptpctl list

List all managed devices:

$ ptpctl list

Managed PTP Devices
================================================================================
/dev/ptp0 (enp1s0f0np0) - Kernel backend
/dev/ptp4 (enp33s0f0np0) - Kernel backend
port0 (DPDK) - DPDK backend
port1 (DPDK) - DPDK backend

ptpctl list --detailed

Show detailed device information:

$ ptpctl list --detailed

Device: /dev/ptp0
  Interface: enp1s0f0np0
  Backend: kernel
  Capabilities: frequency adjustment, phase adjustment
  Servo: PI (kp=0.7, ki=0.3)
  PCI: 0000:01:00.0

Troubleshooting

ptpsyncd fails to start

# Check vfiod is running (required for DPDK devices)
systemctl status vfiod

# Check configuration
cat /etc/ptpsyncd/config.toml

# View logs
journalctl -u ptpsyncd -f

Device not discovered

# Check PTP devices exist
ls -l /sys/class/ptp/

# Check device has PTP capability
ethtool -T <interface>

# Force rediscovery
sudo systemctl reload ptpsyncd

Large offsets reported

# Check if device is actually syncing
ptpctl status
# Look for "State: LOCKED"

# For network mode: ensure master device is synced
sudo ptp4l -i enp1s0f0np0 -s -m

# Adjust servo gains if needed
[servo]
kp = 0.5  # Lower for more damping
ki = 0.2

DPDK backend fails

# Check vfiod DPDK primary is running
ps aux | grep vfiod

# Check hugepages
cat /proc/meminfo | grep Huge

# Check device is in DPDK mode
vfioctl list --vfio

# View DPDK logs
journalctl -u vfiod -f

Performance

Synchronization Accuracy

Typical performance with hardware timestamping:

  • Kernel mode: ±50ns RMS offset
  • DPDK mode (frequency adjustment): ±100ns RMS offset
  • DPDK mode (phase-only): ±500ns RMS offset

System Impact

  • CPU: <0.1% per device (1Hz sync rate)
  • Memory: ~10MB base + DPDK shared memory
  • Network: Minimal (only for network mode master)

License

MIT OR Apache-2.0

Contributing

See /home/serle/projects/rust/TODO.md for project status and planned features.

About

PTP clock synchronization system for kernel-mode and DPDK/VFIO network devices

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages