Add validation event

[Jdubrick]

Description

• For safety shields it updates the event type from token to validation to differentiate between LLM query response and validation response

Type of change

• Refactor
• New feature
• Bug fix
• CVE fix
• Optimization
• Documentation Update
• Configuration Update
• Bump-up service version
• Bump-up dependent library
• Bump-up library or tool used for development (does not change the final image)
• CI configuration change
• Konflux configuration change
• Unit tests improvement
• Integration tests improvement
• End to end tests improvement

Related Tickets & Documents

• Related Issue #
• Closes #

Checklist before requesting a review

• I have performed a self-review of my code.
• PR has passed all pre-merge test jobs.
• If it is a core feature, I have added thorough tests.

Testing

• Please provide detailed steps to perform tests related to this code change.
• How were the fix/results from this change verified? Please provide relevant screenshots or results.

Summary by CodeRabbit

• New Features

  • Added validation event tracking to distinguish validation operations from other event types.
  • Enhanced error metrics tracking for validation processes.

• Tests

  • Updated test expectations to reflect new validation event type handling.

[coderabbitai]

Walkthrough

The pull request introduces a new event type constant LLM_VALIDATION_EVENT set to "validation" and updates shield handling logic to emit validation events instead of token events. A metrics counter for validation errors is added to track violations.

Changes

Cohort / File(s)	Summary
Event Type & Metrics Tracking src/app/endpoints/streaming_query.py	Added LLM_VALIDATION_EVENT constant with value "validation". Modified shield handling to emit validation events instead of token events for both no-violation and violation cases. Added metrics increment llm_calls_validation_errors_total.inc() when a violation occurs.
Test Expectations tests/unit/app/endpoints/test_streaming_query.py	Updated two test functions (test_stream_build_event_shield_call_step_complete_no_violation and test_stream_build_event_shield_call_step_complete_with_violation) to expect "validation" event type instead of "token" in shield_call step results.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

• Straightforward constant addition and event type substitution with consistent pattern
• Metrics tracking logic is simple and localized
• Test updates are parallel and repetitive in nature

Poem

🐰 A validation hop, not a token bounce,
Shield events now speak with proper pronounce,
Errors tracked clear, metrics in place,
LLM calls keep pace,
One new constant makes all things prance! ✨

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title "Add validation event" directly and specifically refers to the primary changes in this pull request. The changeset introduces a new event type constant (LLM_VALIDATION_EVENT) and modifies shield handling to emit validation events instead of token events, fundamentally establishing a validation event mechanism. The title is concise, clear, and avoids noise while accurately summarizing the core functionality being added. It communicates the main intent effectively without requiring additional context.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 36d0a94 and f8b5f56.

📒 Files selected for processing (2)

• src/app/endpoints/streaming_query.py (3 hunks)
• tests/unit/app/endpoints/test_streaming_query.py (2 hunks)

🧰 Additional context used

📓 Path-based instructions (7)

src/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Use absolute imports for internal modules (e.g., from auth import get_auth_dependency)

Files:

• src/app/endpoints/streaming_query.py

src/app/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Use standard FastAPI imports (from fastapi import APIRouter, HTTPException, Request, status, Depends) in FastAPI app code

Files:

• src/app/endpoints/streaming_query.py

**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.py: All modules start with descriptive module-level docstrings explaining purpose
Use logger = logging.getLogger(name) for module logging after import logging
Define type aliases at module level for clarity
All functions require docstrings with brief descriptions
Provide complete type annotations for all function parameters and return types
Use typing_extensions.Self in model validators where appropriate
Use modern union syntax (str | int) and Optional[T] or T | None consistently
Function names use snake_case with descriptive, action-oriented prefixes (get_, validate_, check_)
Avoid in-place parameter modification; return new data structures instead of mutating arguments
Use appropriate logging levels: debug, info, warning, error with clear messages
All classes require descriptive docstrings explaining purpose
Class names use PascalCase with conventional suffixes (Configuration, Error/Exception, Resolver, Interface)
Abstract base classes should use abc.ABC and @AbstractMethod for interfaces
Provide complete type annotations for all class attributes
Follow Google Python docstring style for modules, classes, and functions, including Args, Returns, Raises, Attributes sections as needed

Files:

• src/app/endpoints/streaming_query.py
• tests/unit/app/endpoints/test_streaming_query.py

src/{app/**/*.py,client.py}

📄 CodeRabbit inference engine (CLAUDE.md)

Use async def for I/O-bound operations and external API calls

Files:

• src/app/endpoints/streaming_query.py

src/app/endpoints/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

In API endpoints, raise FastAPI HTTPException with appropriate status codes for error handling

Files:

• src/app/endpoints/streaming_query.py

tests/{unit,integration}/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

tests/{unit,integration}/**/*.py: Use pytest for all unit and integration tests
Do not use unittest in tests; pytest is the standard

Files:

• tests/unit/app/endpoints/test_streaming_query.py

tests/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

tests/**/*.py: Use pytest-mock to create AsyncMock objects for async interactions in tests
Use the shared auth mock constant: MOCK_AUTH = ("mock_user_id", "mock_username", False, "mock_token") in tests

Files:

• tests/unit/app/endpoints/test_streaming_query.py

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)

• GitHub Check: build-pr
• GitHub Check: e2e_tests (azure)
• GitHub Check: e2e_tests (ci)

🔇 Additional comments (4)

src/app/endpoints/streaming_query.py (3)

122-122: LGTM: Validation event constant added.

The new constant follows the existing naming pattern and provides semantic clarity for distinguishing validation events from token events.

---

476-497: LGTM: Shield events now emit validation type.

The change from LLM_TOKEN_EVENT to LLM_VALIDATION_EVENT on lines 481 and 495 semantically distinguishes shield validation results from regular LLM token responses. The metrics increment on line 486 provides good observability for tracking validation violations.

---

476-497: Metric is properly defined and correctly used.

The verification confirms that llm_calls_validation_errors_total is defined in src/metrics/__init__.py (line 39) as a Prometheus Counter, and is correctly invoked with .inc() on line 486 of the streaming_query.py file. The metric is also used consistently in other endpoints and covered by existing tests. No issues found.

tests/unit/app/endpoints/test_streaming_query.py (1)

899-932: LGTM: Test assertions correctly updated for validation event type.

Both test cases (test_stream_build_event_shield_call_step_complete_no_violation and test_stream_build_event_shield_call_step_complete_with_violation) have been updated to expect "validation" event type instead of "token". The tests comprehensively verify the event structure, token content, and metrics behavior.

Also applies to: 934-975

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Jdubrick]

ptal @tisnik

[karthikjeeyar]

Verified this changes with RHDH lightspeed UI.

fix_streaming.mov

[Jdubrick]

Hello @tisnik can you take a look at this PR please? If there is anything holding up the review please let me know so I can address it!

[tisnik]

LGTM