feat: add smoke tests and post-deploy workflow#213
Conversation
| # ARGOCD_AUTH_TOKEN: | ||
| # required: true | ||
| # ARGOCD_SERVER: | ||
| # required: true |
There was a problem hiding this comment.
I can add a TODO: here, or we can create another ticket for this. The idea is to hook the test job up to argo and have it check the sync status, and then fire off the remote smoke tests
| # --grpc-web \ | ||
| # --sync \ | ||
| # --health \ | ||
| # --timeout 900 |
There was a problem hiding this comment.
See my comment above, this is some idea of how that integration might look
| SMOKE_PLAY_INTEGRITY_TOKEN: ${{ secrets.SMOKE_PLAY_INTEGRITY_TOKEN }} | ||
| SMOKE_PLAY_INTEGRITY_USER_ID: ${{ secrets.SMOKE_PLAY_INTEGRITY_USER_ID }} | ||
| CI_WAF_TOKEN: ${{ secrets.CI_WAF_TOKEN }} | ||
| run: uv run pytest src/tests/smoke -v |
There was a problem hiding this comment.
The smoke tests are ran in two forms essentially. It tests against the mocked_client_integration if SMOKE_BASE_URL is not set. If it is set, then it runs the tests against that deployment. App Attest, FxA, and Play Integrity auth are required. I've hooked up a test path for all except play integrity so far.
| temperature: Optional[float] = env.TEMPERATURE | ||
| max_completion_tokens: Optional[int] = env.MAX_COMPLETION_TOKENS | ||
| top_p: Optional[float] = env.TOP_P | ||
| mock_response: Optional[str] = None |
There was a problem hiding this comment.
In my testing, I couldn't get this parameter to work anymore, maybe it was an uptick in LiteLLM version that made it deprecated, but I only found success through setting a dedicated mock model through the litellm config
model_list:
- model_name: mock
litellm_params:
model: openai/gpt-4o
api_key: "mock-token"
mock_response: "this is a mocked response"
| mock_get_or_create_user, | ||
| mock_verify_assert, | ||
| mock_verify_attest, | ||
| from tests.fixtures import ( # noqa: F401 |
There was a problem hiding this comment.
These imports are required for the test fixtures, and the comment tells the linter to not complain about the unused imports
| ) | ||
|
|
||
| FXA_PASSWORD = "123dev123dev123dev" | ||
| FXA_CLIENT_ID = "5882386c6d801776" |
There was a problem hiding this comment.
This is our public stage FxA client ID which we use in a few places throughout MLPA
|
|
||
| def _chat_payload() -> dict: | ||
| return { | ||
| "model": MOCK_MODEL_NAME, |
There was a problem hiding this comment.
MOCK_MODEL_NAME resolves to mock, which should be a configured mock model on the remote we're testing against (litellm configuration mentioned in comment above)
| fi | ||
| mkdir -p "$RUNNER_TEMP/app-attest-qa" | ||
| printf '%s' "$SMOKE_APP_ATTEST_QA_CERT_B64" | base64 -d | tar -xz -C "$RUNNER_TEMP/app-attest-qa" | ||
| echo "APP_ATTEST_QA_CERT_DIR=$RUNNER_TEMP/app-attest-qa" >> "$GITHUB_ENV" |
There was a problem hiding this comment.
This decodes the base64 encoded tar.gz cert bundle from SMOKE_APP_ATTEST_QA_CERT_B64,extracts the qa certs, and exports that dir in APP_ATTEST_QA_CERT_DIR for the smoke tests
What's new
linters_and_tests.ymlto separate out the docs verification into separate steps and for running smoke tests.v0.9.2of PyFxA-Oxide withCI_WAF_TOKENchanges, required for FxA testing.mock_responsefromclasses, since LiteLLM doesn't seem to support that body parameter anymore.mock_responseas a separate modelmockfor testing and adjusted test assertions for consistency.https://mozilla-hub.atlassian.net/browse/AIPLAT-908