Uts experiments: first pass at Claude-generated test specs for REST API #421
Conversation
| Integration tests run against the Ably sandbox environment: | ||
| - `POST https://sandbox.realtime.ably-nonprod.net/apps` to provision app | ||
| - Use `endpoint: "sandbox"` in ClientOptions | ||
| - Test real server behavior and validation |
There was a problem hiding this comment.
Add:
Test apps created using this endpoint should be created once in the setup for a test run, and explicitly cleared when complete. Multiple tests can run against a single app so long as there is no conflict between the state created between those tests. Therefore, any channels created by tests within sandbox apps should be unique for each test. The preferred approach to ensuring uniqueness is to construct channel names as a combination of a descriptive part that refers to the test (eg including the name of the test, or the ID of the spec item) plus a random part that's sufficiently large to ensure the risk of collision is negligible (eg a base64-encoded 48 bit number).
| api_key = app_config.keys[0].key_str | ||
| AFTER ALL TESTS: | ||
| # Sandbox apps auto-delete after 60 minutes |
There was a problem hiding this comment.
The sandbox app should be explicitly deleted
|
|
||
| ### Test Steps | ||
| ```pseudo | ||
| result = AWAIT client.time() |
There was a problem hiding this comment.
Do not use time() because this does not require authentication. Use the channel status endpoint instead.
|
|
||
| ### Assertions | ||
| ```pseudo | ||
| ASSERT result IS valid timestamp |
There was a problem hiding this comment.
This needs to be updated since the test will no longer use time(). However, since this test aims to check that authentication is working, it is not necessary to check that the response body satisfies any particular condition; just check that the request succeeded and wasn't rejected with an authentication or authorization failure
| @@ -0,0 +1,324 @@ | |||
| # Auth Integration Tests | |||
|
|
|||
| Spec points: `RSA4`, `RSA8`, `RSA9`, `RSA10`, `RSA14`, `RSA15` | |||
There was a problem hiding this comment.
All relevant functionality should be integration-tested with JWTs as well as with Ably native tokens (obtained using requestToken()). JWT should be the primary token format used. Native tokens, and the correct handling of token requests etc, should be tested in a way that's as independent as possible from testing the mechanisms relating to handling tokens in requests and the token renewal process via authCallback and authURL
There are clearly lots of issues with this at this stage but it's is raised as a PR to track the feedback given to Claude