Distributed system for orchestrating multiple LLM providers with real-time web monitoring and duplicate elimination
igentai is a high-performance distributed Rust system that orchestrates multiple LLM providers (OpenAI, Anthropic, Gemini) to explore topics and generate unique attributes. The system coordinates parallel content generation while eliminating duplicates across all sources, providing a real-time web interface for monitoring and control.
┌─────────┐
│ LLM 1 │
└────┬────┘
│
┌─────────┐ ┌────▼────┐ ┌─────────┐
│ LLM 2 ├────────►│ │◄────────┤ Web UI │
└─────────┘ │Orchestr-│ └─────────┘
│ ator │
┌─────────┐ │ │ ┌──────────┐
│ LLM N ├────────►│ ├────────►│Attribute │
└─────────┘ └─────────┘ │ File │
└──────────┘
The system consists of:
- Orchestrator: Central coordinator managing producer processes and eliminating duplicates
- Producers: Individual processes calling LLM APIs to generate attributes
- WebServer: Real-time web interface for monitoring and control
- Tester: End-to-end testing framework using distributed tracing
- The OpenAI API key does not work, so I have not yet been able to test against a real LLM.
- The websocket stops updating the browser after a while, although the orchestrator continues to send messages. Working on fixing this issue.
# Build all components
./scripts/build.shFor real LLM providers, set up your API keys:
# Copy the example environment file
cp example.env .env
# Edit .env and add your API keys (at minimum, OPENAI_API_KEY)
# You can also set keys as environment variables
export OPENAI_API_KEY="your-key-here"The system supports multiple routing strategies for load balancing across providers with specific models using a unified configuration format:
# Single provider with model (default)
ROUTING_STRATEGY=backoff
ROUTING_CONFIG=openai:gpt-4o-mini
# Round-robin across multiple providers with their models
ROUTING_STRATEGY=roundrobin
ROUTING_CONFIG=openai:gpt-4o-mini,anthropic:claude-3-sonnet,gemini:gemini-pro
# Priority order (try cheapest first)
ROUTING_STRATEGY=priority
ROUTING_CONFIG=gemini:gemini-pro,openai:gpt-4o-mini,anthropic:claude-3-sonnet
# Weighted distribution with models and weights
ROUTING_STRATEGY=weighted
ROUTING_CONFIG=openai:gpt-4o-mini:0.5,anthropic:claude-3-sonnet:0.3,gemini:gemini-pro:0.2The environment variables align exactly with CLI arguments:
# Environment variables
ROUTING_STRATEGY=roundrobin
ROUTING_CONFIG=openai:gpt-4o-mini,anthropic:claude-3-sonnet
# Equivalent CLI arguments
--routing-strategy roundrobin --routing-config "openai:gpt-4o-mini,anthropic:claude-3-sonnet"Choose between two modes:
# Start with web interface at http://localhost:8080
./scripts/start.sh
# With debug logging
./scripts/start.sh --log-level debug
# Open your browser to http://localhost:8080
# Stop with Ctrl+C or:
./scripts/stop.sh# Direct command line usage - no web interface
cargo run --bin orchestrator -- --topic "Paris attractions" --producers 3 --routing-strategy backoff --routing-config "openai:gpt-4o-mini"
# With test provider (no API keys needed)
cargo run --bin orchestrator -- --topic "Paris attractions" --routing-strategy backoff --routing-config "random:random" --iterations 5
# Build first, then use the binary directly
./scripts/build.sh
./target/release/orchestrator --topic "Paris attractions" --producers 3 --routing-strategy backoff --routing-config "openai:gpt-4o-mini"- Best for: Interactive exploration, monitoring, real-time control
- Features: Live web dashboard, real-time metrics, visual progress tracking
- Access: Open http://localhost:8080 in your browser
- Control: Start/stop generation through web interface
- Output: Files saved to
./output/<topic>/plus web display
- Best for: Automated workflows, batch processing, scripting
- Features: Direct command-line control, immediate execution, script integration
- Control: All configuration through command-line flags
- Output: Files saved to
./output/<topic>/(or custom directory) - Completion: Automatically stops when iterations complete or manually with Ctrl+C
The following are the most commonly used command-line options. For a complete list of all available options with detailed descriptions and defaults, use --help:
./target/release/orchestrator [OPTIONS]
Key Options:
--topic <TOPIC> Topic for generation (enables CLI mode)
--producers <N> Number of producer processes (default: 5)
--iterations <N> Max iterations per producer (default: unlimited)
--request-size <N> Items requested per API call (default: 60)
--routing-strategy <STRATEGY> Load balancing: backoff, roundrobin, priority, weighted
--routing-config <CONFIG> Provider:model configuration for routing (e.g., "openai:gpt-4o-mini")
--output <DIR> Output directory (default: ./output/<topic>)
--log-level <LEVEL> Logging detail: info, debug, trace (default: info)
--trace-ep <URL> Distributed tracing endpoint for observability
--webserver-addr <ADDR> Web interface bind address (default: 127.0.0.1:6000)
--producer-addr <ADDR> Producer communication bind address (default: 127.0.0.1:6001)
--help Display all available options with full descriptions# Quick test with random provider
./target/release/orchestrator --topic "Paris attractions" --routing-strategy backoff --routing-config "random:random" --iterations 3 --producers 2# Basic generation with default backoff routing
./target/release/orchestrator --topic "Renewable Energy" --producers 3 --routing-strategy backoff --routing-config "openai:gpt-4o-mini"
# Limited iterations for cost control
./target/release/orchestrator --topic "AI Ethics" --producers 5 --routing-strategy backoff --routing-config "openai:gpt-4o-mini" --iterations 50
# Custom routing strategy with specific models
./target/release/orchestrator --topic "Space Technology" --routing-strategy roundrobin --routing-config "openai:gpt-4o-mini,anthropic:claude-3-sonnet"
# Multiple providers with weighted distribution and custom request size
./target/release/orchestrator --topic "Climate Change" --routing-strategy weighted --routing-config "openai:gpt-4o-mini:0.6,gemini:gemini-pro:0.4" --request-size 80
# With distributed tracing for observability
./target/release/orchestrator --topic "Machine Learning" --producers 4 --trace-ep "http://localhost:4317" --log-level debug# Start web interface
./scripts/start.sh
# Then open http://localhost:8080 and:
# 1. Enter a topic in the web form
# 2. Set producer count and iterations
# 3. Click "Start" to begin generation
# 4. Watch real-time progress and resultsThe system creates structured output in the specified directory:
output.txt- Plain text list of unique attributesoutput.json- Structured JSON with metadatametadata.json- Generation statistics and settings
The system includes comprehensive testing capabilities:
# Run end-to-end tests with random provider
cargo run --bin tester
# Test web mode (starts server for manual testing)
cargo run --bin tester -- --scenario webserver
# Keep test environment running for debugging
cargo run --bin tester -- --keep-runningThe testing framework uses distributed tracing to validate real system behavior across all components, providing more reliable validation than traditional mocking approaches.
- Rust: 1.70+ (included in sandbox)
- Memory: 2GB+ recommended for multiple producers
- Network: Internet access for LLM API calls
- API Keys: At least OpenAI for real generation (optional for testing)
"OpenAI API key should start with 'sk-'"
- Set a valid API key in
.envfile or environment variable - Or use
--provider randomfor testing without API keys
"Connection refused" or port errors
- Stop existing processes:
./scripts/stop.sh - Check no other services are using port 8080
Low generation speed
- Increase producer count:
--producers N - Check API key rate limits
- Monitor with
--log-level debug
Run with --log-level debug for detailed diagnostic information.
This project is licensed under the MIT License - see the LICENSE file for details.
Serle Shuman - serle.shuman@gmail.com