Skip to content

Bug: Functional tests broken since named-channels merge — orchestrator discovery hangs in CI #92

@renecannao

Description

@renecannao

Summary

Since the named-channels merge (#77, #78, #79), all functional tests are broken. Orchestrator starts in the Docker container but cannot save discovered instances to the SQLite backend, causing /api/clusters to return empty results.

Symptoms

  • Orchestrator starts successfully and logs "Starting HTTP listener on :3099"
  • Discover API calls resolve hostnames (mysql1 → mysql1, sees replica IPs 172.30.0.11, 172.30.0.12)
  • But /api/clusters returns empty ([]) after 60+ seconds
  • Only ~30 lines of orchestrator log in 8+ minutes — the continuous discovery loop appears to stop after the first iteration
  • No error messages logged

Last working commit

70065f9d (Merge PR #76, issue74-web-ui-refresh) — functional tests passed in 3m42s

First broken commit

e4d5b2ae (Multi-cluster support: manage MySQL and PostgreSQL simultaneously, #73)

Investigation so far

  • Schema migration for provider_type column works correctly on SQLite (verified locally)
  • Orchestrator self-elects on local machine (no Docker) without issues
  • The discover API handler calls ReadTopologyInstance which connects to MySQL successfully (hostname resolution + replica IPs appear in logs)
  • But instance data never appears in backend DB
  • The continuous discovery loop seems to stop/block after the first cycle
  • Possibly related to SQLite concurrency with the new detectProviderType queries running alongside the write path

To reproduce

go build -o bin/orchestrator ./go/cmd/orchestrator
cd tests/functional
docker compose up -d mysql1 mysql2 mysql3 proxysql
bash setup-replication.sh
docker compose up -d orchestrator
# Wait 30s, then:
curl http://localhost:3099/api/clusters  # Returns []
docker compose logs orchestrator  # Shows only ~30 lines

Impact

All functional CI tests fail on master and all PR branches.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions