Skip to content

Use multi-run median for lighthouse performance score#105

Merged
tehw0lf merged 1 commit into
mainfrom
feature/lighthouse-multi-run-median
Jul 13, 2026
Merged

Use multi-run median for lighthouse performance score#105
tehw0lf merged 1 commit into
mainfrom
feature/lighthouse-multi-run-median

Conversation

@tehw0lf

@tehw0lf tehw0lf commented Jul 13, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds a num_runs input (default 3) to the reusable lighthouse-scan.yml workflow, passed through as numberOfRuns to the treosh lighthouse-ci-action.
  • Lighthouse now reports the median performance score across multiple runs instead of a single sample, reducing false-positive CI failures caused by GitHub-hosted runner CPU jitter.
  • No consumer repos require changes — the new input is optional and defaults to 3 runs.

Test plan

  • actionlint .github/workflows/lighthouse-scan.yml passes
  • Trigger the weekly lighthouse scan in a consumer repo (e.g. tehw0lf/tehw0lf) and confirm the median score is more stable across runs

https://claude.ai/code/session_01YXxxzAa9Mk67SjBqHwrmqm

Summary by CodeRabbit

  • Chores
    • Added a configurable number of Lighthouse audit runs, defaulting to three.
    • Lighthouse results now use the median across runs to provide more consistent performance measurements.

Single-run performance scores are noisy due to GitHub-hosted runner
CPU jitter, causing flaky CI failures unrelated to actual regressions.
numberOfRuns now defaults to 3 so lighthouse-ci reports the median
score instead of a single sample.

Claude-Session: https://claude.ai/code/session_01YXxxzAa9Mk67SjBqHwrmqm
@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The reusable Lighthouse workflow adds an optional num_runs input with a default of 3 and passes it to Lighthouse CI as numberOfRuns.

Changes

Lighthouse audit configuration

Layer / File(s) Summary
Configure Lighthouse run count
.github/workflows/lighthouse-scan.yml
Adds the num_runs workflow input and passes it to the Lighthouse CI action as numberOfRuns.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Possibly related PRs

  • tehw0lf/workflows#89: Updates the same reusable Lighthouse workflow with configurable run-count behavior.

Suggested reviewers: dependabot[bot]

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly reflects the main change: using multiple Lighthouse runs with median scoring to reduce flakiness.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/lighthouse-multi-run-median

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/lighthouse-scan.yml:
- Line 116: Update the treosh/lighthouse-ci-action configuration to use the runs
input instead of numberOfRuns, while preserving the existing inputs.num_runs
value so the workflow’s requested run count is applied.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5262fd2a-3547-44ad-823b-825501f500fb

📥 Commits

Reviewing files that changed from the base of the PR and between d74d445 and d45004a.

📒 Files selected for processing (1)
  • .github/workflows/lighthouse-scan.yml

with:
urls: ${{ steps.urls.outputs.urls }}
configPath: .lighthouserc.json
numberOfRuns: ${{ inputs.num_runs }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

ref="3e7e23fb74242897f95c0ba9cabad3d0227b9b18"
curl -fsSL "https://github.com/treosh/lighthouse-ci-action/${ref}/action.yml" |
  grep -E '(^| )runs:|numberOfRuns'

Repository: tehw0lf/workflows

Length of output: 169


Use runs here.
numberOfRuns isn’t an input accepted by treosh/lighthouse-ci-action, so the workflow input is ignored and the run count won’t change.

Proposed fix
-          numberOfRuns: ${{ inputs.num_runs }}
+          runs: ${{ inputs.num_runs }}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
numberOfRuns: ${{ inputs.num_runs }}
runs: ${{ inputs.num_runs }}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/lighthouse-scan.yml at line 116, Update the
treosh/lighthouse-ci-action configuration to use the runs input instead of
numberOfRuns, while preserving the existing inputs.num_runs value so the
workflow’s requested run count is applied.

@tehw0lf tehw0lf merged commit dc8df35 into main Jul 13, 2026
3 checks passed
@tehw0lf tehw0lf deleted the feature/lighthouse-multi-run-median branch July 13, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant