Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 12 additions & 7 deletions docs/e2e_scenarios.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@
* Check if the authorized endpoint works fine when user_id and auth header are not provided
* Check if the authorized endpoint works when auth token is not provided
* Check if the authorized endpoint works when user_id is not provided
* Check if the authorized endpoint works when providing empty user_id
* Check if the authorized endpoint rejects empty user_id
* Check if the authorized endpoint works when providing proper user_id

## [`authorized_noop_token.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/authorized_noop_token.feature)

* Check if the authorized endpoint fails when user_id and auth header are not provided
* Check if the authorized endpoint works when user_id is not provided
* Check if the authorized endpoint works when providing empty user_id
* Check if the authorized endpoint rejects empty user_id
* Check if the authorized endpoint works when providing proper user_id
* Check if the authorized endpoint works with proper user_id but bearer token is not present
* Check if the authorized endpoint works when auth token is malformed
Expand Down Expand Up @@ -103,11 +103,11 @@
* Check if info endpoint is working
* Check if info endpoint reports error when llama-stack connection is not working
* Check if models endpoint is working
* Check if models endpoint reports error when llama-stack in unreachable
* Check if models endpoint reports error when llama-stack is unreachable
* Check if shields endpoint is working
* Check if shields endpoint reports error when llama-stack in unreachable
* Check if shields endpoint reports error when llama-stack is unreachable
* Check if tools endpoint is working
* Check if tools endpoint reports error when llama-stack in unreachable
* Check if tools endpoint reports error when llama-stack is unreachable
* Check if metrics endpoint is working
* Check if MCP client auth options endpoint is working

Expand All @@ -118,13 +118,16 @@
* Check if LLM ignores new system prompt in same conversation
* Check if LLM responds to sent question with error when not authenticated
* Check if LLM responds to sent question with error when bearer token is missing
* Check if LLM responds to sent question with error when model does not exist
* Check if LLM responds to sent question with error when attempting to access conversation
* Check if LLM responds to sent question with error when attempting to access conversation with incorrect conversation ID format
* Check if LLM responds for query request with error for missing query
* Check if LLM responds for query request with error for missing query
Comment on lines +122 to 124
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Duplicate bullet entry for "missing query" scenario.

Lines 123 and 124 contain identical text. Line 123 appears to have been inserted as a new entry that already existed at line 124, leaving a duplicate.

🐛 Proposed fix
-* Check if LLM responds for query request with error for missing query
 * Check if LLM responds for query request with error for missing query
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
* Check if LLM responds to sent question with error when attempting to access conversation with incorrect conversation ID format
* Check if LLM responds for query request with error for missing query
* Check if LLM responds for query request with error for missing query
* Check if LLM responds to sent question with error when attempting to access conversation with incorrect conversation ID format
* Check if LLM responds for query request with error for missing query
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@docs/e2e_scenarios.md` around lines 122 - 124, There is a duplicate bullet
"Check if LLM responds for query request with error for missing query" in the
e2e scenario list; remove the redundant entry or replace it with the intended
alternate scenario, updating the bullet list in docs/e2e_scenarios.md so each
test case is unique (look for the exact duplicate string "Check if LLM responds
for query request with error for missing query" and delete or modify one
occurrence to the correct scenario description).

* Check if LLM responds for query request for missing model and provider
* Check if LLM responds for query request with error for missing model
* Check if LLM responds for query request with error for missing provider
* Check if LLM responds for query request with error for missing provider
* Check if LLM responds for query request with error for unknown model
* Check if LLM responds for query request with error for unknown provider
* Check if LLM responds for query request with error for inability to connect to llama-stack
* Check if LLM responds properly when XML and JSON attachments are sent

## [`rbac.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/rbac.feature)
Expand Down Expand Up @@ -164,6 +167,8 @@
* Check if LLM responds for streaming_query request for missing model and provider
* Check if LLM responds for streaming_query request with error for missing model
* Check if LLM responds for streaming_query request with error for missing provider
* Check if LLM responds for streaming_query request with error for unknown model
* Check if LLM responds for streaming_query request with error for unknown provider
* Check if LLM responds properly when XML and JSON attachments are sent
* Check if LLM responds to sent question with error when not authenticated

6 changes: 3 additions & 3 deletions tests/e2e/features/info.feature
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Feature: Info tests


@skip-in-library-mode
Scenario: Check if models endpoint reports error when llama-stack in unreachable
Scenario: Check if models endpoint reports error when llama-stack is unreachable
Given The system is in default state
And The llama-stack connection is disrupted
When I access REST API endpoint "models" using HTTP GET method
Expand All @@ -55,7 +55,7 @@ Feature: Info tests


@skip-in-library-mode
Scenario: Check if shields endpoint reports error when llama-stack in unreachable
Scenario: Check if shields endpoint reports error when llama-stack is unreachable
Given The system is in default state
And The llama-stack connection is disrupted
When I access REST API endpoint "shields" using HTTP GET method
Expand Down Expand Up @@ -114,7 +114,7 @@ Feature: Info tests


@skip-in-library-mode
Scenario: Check if tools endpoint reports error when llama-stack in unreachable
Scenario: Check if tools endpoint reports error when llama-stack is unreachable
Given The system is in default state
And The llama-stack connection is disrupted
When I access REST API endpoint "tools" using HTTP GET method
Expand Down
Loading