Enable `filteredOperations` on the cli level

[derberg]

Current problem and reproduce steps

So I have this websocket AsyncAPI document:

Click to expand

asyncapi: 3.0.0
info:
  title: Hoppscotch Echo WebSocket
  version: 1.0.0
  description: >
    Undestand how to use Hoppscotch Echo WebSocket as a client.
    Hoppscotch Echo WebSocket server echoes back any messages sent to it. You can use this to test WebSocket connections and message flows.

servers:
  echoServer:
    host: echo-websocket.hoppscotch.io
    protocol: wss

channels:
  echo:
    description: >
      The single channel where messages are sent to and echoed back. Echo server also regularly drops a timestampl to that channel.
    address: /
    messages:
      echo:
        $ref: '#/components/messages/echo'
      timestamp:
        $ref: '#/components/messages/timestamp'
    bindings:
      ws:
        method: POST

operations:
  handleTimeStampMessage:
    action: receive
    channel: 
      $ref: '#/channels/echo'
    summary: Receive the timestamp message sent from server every second.
    messages:
      - $ref: '#/channels/echo/messages/timestamp'
  sendEchoMessage:
    action: send
    channel: 
      $ref: '#/channels/echo'
    summary: Send a message to the echo server.
    messages:
      - $ref: '#/channels/echo/messages/echo'

components:
  messages:
    timestamp:
      summary: A message sent from server with current timestamp.
      payload: 
        $ref: '#/components/schemas/currentTime'
      examples:
        - name: someRandomDate
          payload: 11:13:24 GMT+0000 (Coordinated Universal Time)
    echo:
      summary: A message exchanged with the echo server.
      payload: {}
      examples:
        - name: string
          payload: test
        - name: object
          payload: 
            test: test text
  
  schemas:
    currentTime:
      type: string
      description: A timestamp with GMT timezone.
      example: 11:13:24 GMT+0000 (Coordinated Universal Time)

now, with Microcks CLI I try to import it and run test.

I import like this, all works well:

microcks-cli import asyncapi-hoppscotch-echo.yml \
  --microcksURL=http://localhost:8080/api/ \
  --insecure \
  --keycloakClientId=microcks-serviceaccount \
  --keycloakClientSecret="ab54d329-e435-41ae-a900-ec6b3fe15c54"

then using cli I want to run test:

microcks-cli test 'Hoppscotch Echo WebSocket:1.0.0' ws://localhost:8081/api/ws/Hoppscotch+Echo+WebSocket/1.0.0/sendEchoMessage ASYNC_API_SCHEMA \
    --microcksURL=http://localhost:8080/api/ \
    --insecure \
    --waitFor=6sec \
    --keycloakClientId=microcks-serviceaccount \
    --keycloakClientSecret="ab54d329-e435-41ae-a900-ec6b3fe15c54"

the problem is that this invokes test on both operations that I have there, sendEchoMessage and handleTimeStampMessage but I do not want that, especially that in Microcks you get 2 different endpoints for 2 different operations, so when I use ws://localhost:8081/api/ws/Hoppscotch+Echo+WebSocket/1.0.0/sendEchoMessage, then handleTimeStampMessage is tested with examples from sendEchoMessage

Workaround

I found your OpenAPI file and figured I can run test with API, and that there is filteredOperations property

so below works for me like a charm:

curl -X POST http://localhost:8080/api/tests \
  -H "Content-Type: application/json" \
  -d '{
    "serviceId": "Hoppscotch Echo WebSocket:1.0.0",
    "testEndpoint": "ws://localhost:8081/api/ws/Hoppscotch+Echo+WebSocket/1.0.0/sendEchoMessage",
    "runnerType": "ASYNC_API_SCHEMA",
    "timeout": 10000,
    "filteredOperations": ["SEND sendEchoMessage"]
  }'

now only sendEchoMessage operation is tested 💪🏼

Possible solution

Workaround is there so not a big rush, but I think the easiest would be to enable a flag for microcks-cli test that would ensure only selected operation is tested

If below works, would be great

microcks-cli test 'Hoppscotch Echo WebSocket:1.0.0' ws://localhost:8081/api/ws/Hoppscotch+Echo+WebSocket/1.0.0/sendEchoMessage ASYNC_API_SCHEMA \
    --microcksURL=http://localhost:8080/api/ \
    --insecure \
    --waitFor=6sec \
    --keycloakClientId=microcks-serviceaccount \
    --keycloakClientSecret="ab54d329-e435-41ae-a900-ec6b3fe15c54" \
    --filteredOperations="SEND sendEchoMessage"

anyway, up to you, I created the issue to share a workaround mainly to save time for others. Up to you if you want to extend the CLI.

[lbroudoux]

Hey! Thanks for writing this out. I'm going to transfer this issue to the microcks-cli repo.

Looks legit and we could also certainly add a warning saying that filteredOperations must be used when launching an ASYNC_API_SCHEMA test (this control is actually done in the UI).

We have @Harsh4902 working on big refactoring at the moment and this can be included!

[github-actions]

👋 @derberg

Welcome to the Microcks community! 💖

Thanks and congrats 🎉 for opening your first issue here! Be sure to follow the issue template or please update it accordingly.

📢 If you're using Microcks in your organization, please add your company name to this list. 🙏 It really helps the project to gain momentum and credibility. It's a small contribution back to the project with a big impact.

If you need to know why and how to add yourself to the list, please read the blog post "Join the Microcks Adopters list and Empower the vibrant open source Community 🙌"

Hope you have a great time there!

🌟 ~~~~~~~~~ 🌟

📢 If you like Microcks, please ⭐ star ⭐ our repo to support it!

🙏 It really helps the project to gain momentum and credibility. It's a small contribution back to the project with a big impact.

[Harsh4902]

Thanks @derberg for suggesting this. Will definitely look at this and come up with a feasible solution😄

[Harsh4902]

@derberg We already have --filteredOperations flag in our CLI which expects a JSON string. You can use ["SEND sendEchoMessage"] as a flag argument, and it will give you an expected outcome. I hope this will help you 😃.

[lbroudoux]

I think that @derberg suggestion is much more about the formatting: not having to use JSON array for a single value and just specify the operation name as a single string.

[derberg]

I just made a test with below file

asyncapi: 3.0.0
defaultContentType: text/plain
info:
  title: Hoppscotch WebSocket Server
  version: 1.0.0
  description: >
    This document is purely for mocking the official Hoppscotch Websocket Echo server.

servers:
  echoServer:
    host: echo-websocket.hoppscotch.io
    protocol: wss

channels:
  echo:
    description: >
      The single channel where messages are sent to and echoed back. Echo server also regularly drops a timestampl to that channel.
    address: /
    messages:
      echo:
        $ref: '#/components/messages/echo'
      timestamp:
        $ref: '#/components/messages/timestamp'
    bindings:
      ws:
        method: POST

operations:
  sendTimeStampMessage:
    action: send
    channel: 
      $ref: '#/channels/echo'
    summary: Receive the timestamp message sent from server every second.
    messages:
      - $ref: '#/channels/echo/messages/timestamp'
  handleEchoMessage:
    action: receive
    channel: 
      $ref: '#/channels/echo'
    summary: Send a message to the echo server.
    messages:
      - $ref: '#/channels/echo/messages/echo'

components:
  messages:
    timestamp:
      summary: A message sent from server with current timestamp.
      payload: 
        $ref: '#/components/schemas/currentTime'
      examples:
        - name: someRandomDate
          payload: 11:13:24 GMT+0000 (Coordinated Universal Time)
    echo:
      summary: A message exchanged with the echo server.
      payload: {}
      examples:
        - name: string
          payload: "test"
        - name: boolean
          payload: true
        - name: number
          payload: 123
        - name: object
          payload: 
            test: test text
  
  schemas:
    currentTime:
      type: string
      description: A timestamp with GMT timezone.
      example: 11:13:24 GMT+0000 (Coordinated Universal Time)

Case no array

microcks-cli test 'Hoppscotch WebSocket Server:1.0.0' ws://localhost:8081/api/ws/Hoppscotch+WebSocket+Server/1.0.0/sendTimeStampMessage ASYNC_API_SCHEMA \
    --microcksURL=http://localhost:8080/api/ \
    --insecure \
    --waitFor=6sec \
    --keycloakClientId=microcks-serviceaccount \
    --keycloakClientSecret="ab54d329-e435-41ae-a900-ec6b3fe15c54"
    --filteredOperations="SEND sendTimeStampMessage"

gives error:

command not found: --filteredOperations=SEND sendTimeStampMessage

and all operation tests run

Case with array

microcks-cli test 'Hoppscotch WebSocket Server:1.0.0' ws://localhost:8081/api/ws/Hoppscotch+WebSocket+Server/1.0.0/sendTimeStampMessage ASYNC_API_SCHEMA \
    --microcksURL=http://localhost:8080/api/ \
    --insecure \
    --waitFor=6sec \
    --keycloakClientId=microcks-serviceaccount \
    --keycloakClientSecret="ab54d329-e435-41ae-a900-ec6b3fe15c54"
    --filteredOperations=["SEND sendTimeStampMessage"]

gives error

no matches found: --filteredOperations=[SEND sendTimeStampMessage]

and all operation tests run

Some more

when I do microcks-cli test --help I only get overall help and no specific overview of args

test command require <apiName:apiVersion> <testEndpoint> <runner> args`

when I do microcks-cli help there is no mention of filteredOperations

and when I go to https://microcks.io/documentation/ the search shows nothing about filteredOperations.

Conclusion

To be honest, I don't mind if I have to do ["SEND sendTimeStampMessage"] or "SEND sendTimeStampMessage".

The problem is I do not know that filteredOperations arg exists and how to use it.

Does it make sense?

[lbroudoux]

Thanks for the clarification! I will check if I can easily fix things (and help command and documentation BTW) on current CLI version without having to wait for the refactoring @Harsh4902 is working on.

[Harsh4902]

I just made a test with below file

asyncapi: 3.0.0
defaultContentType: text/plain
info:
title: Hoppscotch WebSocket Server
version: 1.0.0
description: >
This document is purely for mocking the official Hoppscotch Websocket Echo server.

servers:
echoServer:
host: echo-websocket.hoppscotch.io
protocol: wss

channels:
echo:
description: >
The single channel where messages are sent to and echoed back. Echo server also regularly drops a timestampl to that channel.
address: /
messages:
echo:
$ref: '#/components/messages/echo'
timestamp:
$ref: '#/components/messages/timestamp'
bindings:
ws:
method: POST

operations:
sendTimeStampMessage:
action: send
channel:
$ref: '#/channels/echo'
summary: Receive the timestamp message sent from server every second.
messages:
- $ref: '#/channels/echo/messages/timestamp'
handleEchoMessage:
action: receive
channel:
$ref: '#/channels/echo'
summary: Send a message to the echo server.
messages:
- $ref: '#/channels/echo/messages/echo'

components:
messages:
timestamp:
summary: A message sent from server with current timestamp.
payload:
$ref: '#/components/schemas/currentTime'
examples:
- name: someRandomDate
payload: 11:13:24 GMT+0000 (Coordinated Universal Time)
echo:
summary: A message exchanged with the echo server.
payload: {}
examples:
- name: string
payload: "test"
- name: boolean
payload: true
- name: number
payload: 123
- name: object
payload:
test: test text

schemas:
currentTime:
type: string
description: A timestamp with GMT timezone.
example: 11:13:24 GMT+0000 (Coordinated Universal Time)

Case no array

microcks-cli test 'Hoppscotch WebSocket Server:1.0.0' ws://localhost:8081/api/ws/Hoppscotch+WebSocket+Server/1.0.0/sendTimeStampMessage ASYNC_API_SCHEMA
--microcksURL=http://localhost:8080/api/
--insecure
--waitFor=6sec
--keycloakClientId=microcks-serviceaccount
--keycloakClientSecret="ab54d329-e435-41ae-a900-ec6b3fe15c54"
--filteredOperations="SEND sendTimeStampMessage"
gives error:

command not found: --filteredOperations=SEND sendTimeStampMessage

and all operation tests run

Case with array

microcks-cli test 'Hoppscotch WebSocket Server:1.0.0' ws://localhost:8081/api/ws/Hoppscotch+WebSocket+Server/1.0.0/sendTimeStampMessage ASYNC_API_SCHEMA
--microcksURL=http://localhost:8080/api/
--insecure
--waitFor=6sec
--keycloakClientId=microcks-serviceaccount
--keycloakClientSecret="ab54d329-e435-41ae-a900-ec6b3fe15c54"
--filteredOperations=["SEND sendTimeStampMessage"]
gives error

no matches found: --filteredOperations=[SEND sendTimeStampMessage]

and all operation tests run

Some more

when I do microcks-cli test --help I only get overall help and no specific overview of args

test command require <apiName:apiVersion> <testEndpoint> <runner> args`

when I do microcks-cli help there is no mention of filteredOperations

and when I go to https://microcks.io/documentation/ the search shows nothing about filteredOperations.

Conclusion

To be honest, I don't mind if I have to do ["SEND sendTimeStampMessage"] or "SEND sendTimeStampMessage".

The problem is I do not know that filteredOperations arg exists and how to use it.

Does it make sense?

@derberg Actually, in both cases, you missed the \ before the --filteredOperations flag, so the value for this flag is not set, and by default, it runs all tests. Because of this, your shell considers it a separate command and throws a command not found error in the first case.

Other than that, the argument for --filteredOperations is a pure string that will be appended to the JSON body as "filteredOperations": (value of filteredOperations flag). So, to achieve this, we have to use an escape character inside [], and your final input will be "[\"SEND sendTimeStampMessage\"]".

So Your final command will be:

microcks-cli test 'Hoppscotch WebSocket Server:1.0.0' ws://localhost:8081/api/ws/Hoppscotch+WebSocket+Server/1.0.0/sendTimeStampMessage ASYNC_API_SCHEMA \
    --microcksURL=http://localhost:8080/api/ \
    --insecure \
    --waitFor=6sec \
    --keycloakClientId=microcks-serviceaccount \
    --keycloakClientSecret="ab54d329-e435-41ae-a900-ec6b3fe15c54" \
    --filteredOperations="[\"SEND sendTimeStampMessage\"]"

We'll definitely make changes to our documentation and write it in a very detailed manner so that it becomes clearer to the users 😃.

[derberg]

nice! below worked like a charm ❤ thanks a 💯

microcks-cli test 'Hoppscotch WebSocket Server:1.0.0' ws://localhost:8081/api/ws/Hoppscotch+WebSocket+Server/1.0.0/sendTimeStampMessage ASYNC_API_SCHEMA \
    --microcksURL=http://localhost:8080/api/ \
    --insecure \
    --waitFor=6sec \
    --keycloakClientId=microcks-serviceaccount \
    --keycloakClientSecret="ab54d329-e435-41ae-a900-ec6b3fe15c54" \
    --filteredOperations="[\"SEND sendTimeStampMessage\"]"

[lbroudoux]

You can also use simple quotes to avoid escaping the double ones:

microcks-cli test 'Hoppscotch WebSocket Server:1.0.0' ws://localhost:8081/api/ws/Hoppscotch+WebSocket+Server/1.0.0/sendTimeStampMessage ASYNC_API_SCHEMA \
    --microcksURL=http://localhost:8080/api/ \
    --insecure \
    --waitFor=6sec \
    --keycloakClientId=microcks-serviceaccount \
    --keycloakClientSecret=ab54d329-e435-41ae-a900-ec6b3fe15c54 \
    --filteredOperations='["SEND sendTimeStampMessage"]'

I re-read the README and saw that there is actually an example on how to use this.

$ ./microcks-cli test 'Beer Catalog API:0.9' http://localhost:9090/api/ OPEN_API_SCHEMA \                           
        --microcksURL=http://localhost:8080/api/ \
        --keycloakClientId=microcks-serviceaccount \
        --keycloakClientSecret=7deb71e8-8c80-4376-95ad-00a399ee3ca1 \
        --insecure --verbose  --waitFor=3sec \
        --filteredOperations='["GET /beer", "GET /beer/{name}"]' \
        --operationsHeaders='{"globals": [{"name": "x-api-key", "values": "my-values"}], "GET /beer": [{"name": "x-trace-id", "values": "xcvbnsdfghjklm"}]}' \
        --oAuth2Context='{"clientId": "microcks-test", "clientSecret": "ab54d329-e435-41ae-a900-ec6b3fe15c54", "tokenUri": "https://idp.acme.org/realms/my-app/protocol/openid-connect/token", "grantType": "CLIENT_CREDENTIALS"}'

MicrocksClient got status for test "64c25f7ddec62569f9a0ed95" - success: true, inProgress: false 
Full TestResult details are available here: http://localhost:8080/#/tests/64c25f7ddec62569f9a0ed95 

I'll update the microcks-cli help command anyway to display the advanced options and flags.