Add a stable machine-readable summary artifact for generated scenario reports

[shreyaskommuri]

Hi CloudAI maintainers,

While building a small closed-loop experiment wrapper around CloudAI, I noticed that generated report artifacts seem to be intentionally flexible and workload-specific. That makes sense for rich workload reports, but it also makes automation harder because there does not appear to be one stable machine-readable artifact that external tools can rely on after a run.

From what I saw:

• cloudai generate-report writes reports under the scenario/result directory.
• The README documents a general HTML report as <scenario>.html.
• DSE scenarios can produce <scenario>-dse-report.html and a best config TOML.
• Workloads may produce their own artifacts, for example SGLang uses sglang-bench.jsonl, AI Dynamo uses CSV report files, and other workloads have their own filenames/fields.
• Custom reporters may also generate additional artifacts.

This is useful for humans, but external automation has to know workload-specific filenames and metric field names, or fall back to scraping logs/HTML.

Why this matters

Closed-loop tools, CI jobs, dashboards, and experiment trackers need a predictable way to answer a few basic questions after a CloudAI run:

• Did the scenario/test run complete successfully?
• Where is the result directory?
• Which artifacts were generated?
• Which config files were used?
• What metrics were produced?
• Which workload/test run produced each metric?

Without a stable machine-readable summary, integrations have to duplicate CloudAI-specific parsing logic for each workload.

Suggested improvement

Add a small JSON artifact, for example cloudai-summary.json, emitted during report generation. It could live next to the existing scenario reports and include fields like:

{
  "scenario": "example-scenario",
  "status": "completed",
  "result_dir": "...",
  "reports": [
    {
      "name": "status",
      "path": "example-scenario.html",
      "format": "html"
    }
  ],
  "test_runs": [
    {
      "name": "sglang-bench",
      "status": "completed",
      "output_path": "...",
      "artifacts": [
        {
          "path": "sglang-bench.jsonl",
          "format": "jsonl"
        }
      ],
      "metrics": {
        "request_throughput": 42.5,
        "mean_ttft_ms": 18.2
      }
    }
  ]
}

This would not need to replace the existing HTML reports or workload-specific artifacts. It would just provide a stable automation entry point for tools that want to consume CloudAI results programmatically.

Possible implementation direction

One lightweight path might be to add a scenario-level summary reporter that gathers known report paths, test run output paths, statuses, and any metrics already exposed by each workload's ReportGenerationStrategy / metric reporter. Even a minimal first version with scenario metadata, status, output paths, and artifact paths would be useful; normalized metrics could be expanded later.

Context

I ran into this while trying to ingest CloudAI SGLang results from an external closed-loop config tuning tool. The SGLang JSONL output was parseable, but the integration had to know the exact filename and fields instead of reading a single CloudAI-level summary artifact.