✨ Add Vitest browser mode plugin (drop-in replacement for vitest visual testing)

[Robdel12]

Vitest Browser Mode Integration

This PR adds a Vitest v4 browser mode plugin that completely replaces Vitest's native visual testing with Vizzly.

Implementation

The plugin works by:

1. Extending expect API with a custom toMatchScreenshot matcher via a browser-side setup file
2. Disabling Vitest's native screenshot system with screenshotFailures: false
3. Making direct HTTP POST requests from browser to Vizzly server
4. Returning comparison results in TDD mode, always passing in cloud mode

API

Users just add the plugin - no test changes required:

// vitest.config.js
import { vizzlyPlugin } from '@vizzly-testing/vitest'

export default defineConfig({
  plugins: [vizzlyPlugin()],
  test: {
    browser: {
      enabled: true,
      name: 'chromium',
      provider: 'playwright'
    }
  }
})

// Tests use standard Vitest API
await expect(page).toMatchScreenshot('screenshot.png', {
  properties: { theme: 'dark' },
  threshold: 5
})

Files Changed

• clients/vitest/src/setup.js (NEW) - Custom matcher implementation in browser context
• clients/vitest/src/index.js - Plugin that injects setup file and passes env vars
• clients/vitest/vitest.e2e.config.js (NEW) - E2E test configuration
• clients/vitest/tests/ - Updated tests for new implementation
• clients/vitest/README.md - Updated docs
• .github/workflows/ci.yml - Removed unnecessary --update flag
• src/services/test-runner.js - Removed wrong --update flag logic

Testing

• ✅ Unit tests: 9/9 passed
• ✅ E2E tests: 2/2 passed in both TDD and cloud modes
• ✅ CI workflow updated and working
• ✅ No Vitest test artifacts created (native system fully disabled)

[Robdel12]

@claude can you rereview this PR?

[Robdel12]

@claude rereview this PR -- the approach is completely different from the first two reviews now.

[Robdel12]

@claude rereview this PR

[claude]

Claude finished @Robdel12's task —— View job

---

Re-reviewing PR: Vitest Browser Mode Plugin

• Read and understand the new implementation approach
• Review plugin architecture and integration
• Review browser-side setup and matcher implementation
• Review test coverage and E2E tests
• Review documentation and configuration
• Provide comprehensive feedback

---

Review Summary

This PR successfully implements a drop-in replacement for Vitest's native visual testing. The approach has evolved significantly from earlier reviews, and the current implementation is much cleaner and more aligned with Vitest's philosophy. Overall, this is excellent work with a well-thought-out architecture.

Architecture & Design ✅

Strengths:

1. Clean separation of concerns: Plugin configuration (index.js) vs. browser-side matcher (setup.js)
2. Proper Vite plugin lifecycle: Uses config() hook to inject setup files and env vars via define
3. Smart auto-discovery: Searches parent directories for .vizzly/server.json - great DX
4. File path support: The new image-input-detector.js utility elegantly handles both base64 and file paths
5. Dual-mode behavior: TDD mode returns comparison results (pass/fail), cloud mode always passes - this is correct

Architecture observation:
The plugin uses Vite's define feature to inject __VIZZLY_SERVER_URL__ and __VIZZLY_BUILD_ID__ as compile-time constants. This is a standard Vite pattern and works well for browser contexts.

Implementation Quality 🎯

Plugin (clients/vitest/src/index.js) - Very Good

// Line 93: Uses import.meta.dirname with __dirname fallback
resolve(import.meta.dirname || __dirname, 'setup.js')

✅ Good ESM compatibility handling

Minor suggestion: The auto-discovery logic (lines 69-87) duplicates code from src/client/index.js:50-77. Consider extracting to a shared utility if you plan to maintain consistency.

Custom Matcher (clients/vitest/src/setup.js) - Excellent

The matcher implementation is well-designed with several highlights:

Strengths:

1. Proper error handling (lines 124-141): Differentiates between network errors, server errors, and screenshot errors with helpful messages
2. Threshold handling (lines 103-109): Correctly passes screenshots within threshold
3. TDD vs Cloud mode (lines 74-80): Clean conditional logic
4. Good UX: Returns dashboard links in error messages

Observation on line 10:

let serverUrl = typeof __VIZZLY_SERVER_URL__ !== 'undefined' ? __VIZZLY_SERVER_URL__ : '';

This is correct for define usage - the constants are replaced at compile time, so they may literally be undefined in the code.

Question (not a blocker): Lines 26-33 handle both element.locator (Playwright locator) and page object. Should we document what types are expected? The TypeScript definitions show only Assertion.toMatchScreenshot but don't specify what this context is passed.

Server-Side Changes - Well Done

src/utils/image-input-detector.js (clients/vitest/src/index.js:71):

• Comprehensive test coverage (265 lines of tests!)
• Clear documentation with examples
• Handles edge cases (data URIs, Windows paths, file:// URIs)
• Strategy is sound: check file paths FIRST (more specific), then base64 (more permissive)

src/server/handlers/api-handler.js (lines 69-99):

• Properly resolves file paths and reads them
• Good error messages for missing files
• Security: Uses resolve() to prevent path traversal (though input is from localhost tests, still good practice)

src/server/handlers/tdd-handler.js (lines 1-100+):

• Same file path detection logic applied consistently
• PNG dimension detection remains robust (bytes 16-23 per PNG spec)

TypeScript Definitions ✅

clients/vitest/src/index.d.ts:

declare module 'vitest' {
  interface Assertion {
    toMatchScreenshot(
      name?: string,
      options?: VizzlyScreenshotOptions
    ): Promise<void>;
  }
}

✅ Proper module augmentation - this extends Vitest's types correctly

Completeness: Types match the implementation well. The VizzlyScreenshotOptions interface documents all supported fields.

Test Coverage ✅

Unit Tests (clients/vitest/tests/vitest-plugin.spec.js):

• 9 test cases covering plugin registration, config injection, TypeScript declarations, package structure
• Tests verify setup file injection and env var passing via define

E2E Tests (clients/vitest/tests/e2e/example.test.js):

• 2 real browser tests using the plugin
• Tests both basic usage and properties/threshold options
• Actually renders HTML and takes screenshots - excellent dogfooding

New Utility Tests (tests/unit/image-input-detector.spec.js):

• 265 lines of comprehensive tests
• Covers base64 detection, file path detection (Unix, Windows, relative, absolute), edge cases

Coverage verdict: Excellent coverage across unit, integration, and E2E levels.

CI/CD Integration ✅

.github/workflows/ci.yml (lines 340-401):

• New test-vitest-client job runs on blacksmith runners (matching other integration tests)
• Tests across Node 20, 22, 24 - good version coverage
• Runs unit tests (npm run test:unit) and E2E tests via vizzly run in cloud mode
• Properly passes secrets and git metadata
• Integrated into ci-check job for pass/fail gating

Path filter (lines 243-256):

• Only runs when clients/vitest/** or workflow files change - efficient CI usage

Documentation 📚

clients/vitest/README.md:

• Comprehensive (280 lines)
• Clear installation, quick start, API reference
• Explains TDD vs Cloud modes
• Multi-variant testing examples
• TypeScript support documented
• Troubleshooting section
• ✅ Excellent user-facing documentation

Code comments: Both index.js and setup.js have detailed JSDoc comments explaining the approach.

Potential Issues & Questions

1. Minor: Dependency on pngjs

clients/vitest/package.json:57 lists pngjs as a dependency, but I don't see it imported in src/index.js or src/setup.js. Is this actually needed for the Vitest client?

Impact: Low - extra dependency in node_modules, but minimal size
Recommendation: Remove if unused, or document why it's needed

2. Question: Screenshot path resolution

In setup.js:26-33, Vitest browser mode returns a file path from screenshot(). The path is sent directly to the server.

Question: Are these paths always absolute? Or could they be relative to the Vitest working directory? If relative, does the server resolve them correctly?

Looking at api-handler.js:75:

const filePath = resolve(image.replace('file://', ''));

If image is a relative path like ./screenshots/test.png, resolve() will resolve it relative to the server's CWD, which should be the repo root. This should work, but I'd verify that Vitest browser mode always returns paths that resolve correctly from the server's perspective.

Impact: Medium - could cause "file not found" errors if paths don't resolve
Recommendation: Add a comment documenting expected path format, or add defensive logic

3. Minor: Error message formatting

setup.js:129:

error.message.includes('HTTP 500') || error.message.includes('HTTP 502')

This checks if the error message includes "HTTP 500", but earlier (line 69) you throw:

throw new Error(`HTTP ${response.status}: ${response.statusText}`);

So the format is "HTTP 500: Internal Server Error", which would match. But if error messages ever change format, this could break. Consider checking error.status or error.code if available.

Impact: Low - error handling is best-effort anyway
Recommendation: Consider storing status code on error object for more robust checking

4. Documentation: First-run experience

The README mentions (line 96):

New screenshots will fail the first time they run - this is expected!

This is good to document, but users might find it surprising. Consider adding a note in the custom matcher message for 'new' status to make this even clearer (you already do this on line 91, which is great).

✅ Already handled well

Security Review 🔒

Path traversal: api-handler.js:75 uses resolve() which prevents ../../../etc/passwd attacks
Input validation: Image input detector properly validates base64 and paths
Injection: No user input is interpolated into shell commands or eval

✅ No security concerns

Consistency with Vizzly Architecture ✅

Follows patterns:

• Uses service container approach (implicit via server handlers)
• Matches Ruby/Storybook client pattern (thin client, server does heavy lifting)
• TDD/Cloud mode behavior is consistent with other integrations
• Screenshot signature matching aligns with cloud product

CLAUDE.md compliance:

• ✅ Service-oriented design
• ✅ Client SDK philosophy (thin client, server-side processing)
• ✅ Dogfooding (E2E tests use Vizzly itself)
• ✅ Plugin architecture (this could even be a Vizzly CLI plugin, but as a Vite plugin it's appropriate)

Final Recommendations

Must Address (before merge):

None - the implementation is production-ready

Should Address (nice to have):

1. Clarify pngjs dependency - remove if unused
2. Document screenshot path format - add comment about expected path format from Vitest
3. Consider extracting auto-discovery logic - shared utility between plugin and client

Could Address (future work):

1. Support for custom comparators/diffing algorithms (probably out of scope)
2. Progress indicators for long screenshot batches (low priority)

Changelog Review ✅

clients/vitest/CHANGELOG.md:

• Documents initial release (0.1.0)
• Lists all features appropriately
• Mentions browser mode requirement

Overall Assessment

Code Quality: ⭐⭐⭐⭐⭐ (5/5)

• Clean, well-documented, properly tested

Architecture: ⭐⭐⭐⭐⭐ (5/5)

• Excellent separation of concerns, follows Vite/Vitest patterns

Documentation: ⭐⭐⭐⭐⭐ (5/5)

• Comprehensive README, good code comments, clear examples

Testing: ⭐⭐⭐⭐⭐ (5/5)

• Unit, integration, and E2E tests, great coverage

Integration: ⭐⭐⭐⭐⭐ (5/5)

• CI properly configured, follows monorepo patterns

Verdict: ✅ APPROVE

This PR is ready to merge. The implementation is clean, well-tested, thoroughly documented, and follows best practices. The approach of completely replacing Vitest's native visual testing via expect.extend() is the right call - it provides a seamless DX with zero API changes for users.

The file path support is a great addition that makes this work elegantly with Vitest browser mode's screenshot API. The dual-mode behavior (TDD pass/fail vs Cloud always-pass) is architecturally correct.

Great work! 🎉

---

•