feat: local Keycloak mock + backend auth env wiring by vredchenko · Pull Request #198 · DiamondLightSource/smartem-devtools

vredchenko · 2026-05-12T18:05:31Z

Motivation

Two things had to land together:

The frontend Keycloak integration in Add Keycloak authentication to smartem app smartem-frontend#74 had no local-dev story — http://localhost:5173 isn't in the SmartEM client's redirect URIs on identity-test.diamond.ac.uk, and getting it added is an admin round-trip per port and per developer.
Once DiamondLightSource/smartem-decisions starts validating Bearer tokens, the backend ConfigMap and dev-k8s.sh need to carry the same KEYCLOAK_* settings so the pod can fetch JWKS in-cluster.

This PR covers both: a self-contained Keycloak mock for the frontend, and the env-wiring that lets the backend point at it (or at the real DLS realm in staging) without YAML surgery.

What's included

Self-contained Keycloak mock (`keycloak-mock/`)

Two equivalent deployment forms, both reading from one realm export:

dls-realm.json — single source of truth. Realm dls, public client SmartEM with PKCE S256, localhost:5173 and localhost:5174 in redirect URIs and Web Origins, custom fedId claim mapper to mirror DLS realm claims, two seeded users (devuser/devpass, valuser/valpass).
docker-compose.yml — Compose form, fastest standalone cycle (~20 s).
keycloak.yaml + kustomization.yaml — Kubernetes Deployment + Services + kustomize configMapGenerator for the realm JSON.

The Kustomize form is wired into k8s/environments/development/kustomization.yaml, so ./scripts/k8s/dev-k8s.sh up brings up Keycloak alongside Postgres, RabbitMQ, etc. The Compose form is independent and useful when only the frontend is needed.

Backend Keycloak env wiring

smartem-config ConfigMap (both development and staging overlays) gains:

KEYCLOAK_AUTH_REQUIRED — master switch consumed by the backend (defaults: false in dev, true in staging)
KEYCLOAK_URL — dev points at http://keycloak-service:8080 (in-cluster DNS); staging points at https://identity-test.diamond.ac.uk
KEYCLOAK_REALM, KEYCLOAK_CLIENT_ID, KEYCLOAK_VERIFY_ISS

Dev has KEYCLOAK_VERIFY_ISS=false because tokens are minted with the browser-facing URL (localhost:30090) while the pod reaches Keycloak via in-cluster DNS — signing key is identical but iss strings differ. Staging has iss verification on.

dev-k8s.sh ensure_app_configmap() reads the five vars from the environment and bakes them into the dynamically-generated ConfigMap. Operators only need to edit .env.k8s.<env> — corresponding examples in env-examples/ are updated.

Port-conflict fix (`fix(keycloak-mock): move NodePort to 30090`)

The development environment's smartem-http-api-service already claimed NodePort 30080. Applying the dev kustomization with the keycloak mock therefore failed with a port conflict. Moved keycloak-external to 30090 and updated the three doc references.

Docs

docs/development/local-keycloak.md — full how-to (when to use which form, how to point the frontend at it, how to edit the realm, limits and non-goals).
docs/development/index.md — added to the TOC.
docs/architecture/keycloak-spa-authentication.md — "Local development" pointer + corrected the smartem-frontend → SmartEM client-name discrepancy.

Quick start

# Compose form (frontend only)
cd keycloak-mock && docker compose up -d

# Or k3s form (along with the rest of the dev stack)
./scripts/k8s/dev-k8s.sh up

Then in smartem-frontend/apps/smartem/.env.local:

VITE_KEYCLOAK_URL=http://localhost:30090
VITE_KEYCLOAK_REALM=dls
VITE_KEYCLOAK_CLIENT_ID=SmartEM
VITE_AUTH_ENABLED=true

Log in as devuser / devpass.

Verified end-to-end

The full chain — SPA -> Keycloak -> SPA with token -> backend /acquisitions -> backend validates JWT against keycloak-service:8080 JWKS -> 200 + payload — runs cleanly on a local single-node k3s with all three branches applied (#74, this PR, smartem-decisions feat/keycloak-jwt-validation).

Test plan

kubectl kustomize keycloak-mock builds cleanly
kubectl kustomize k8s/environments/development builds cleanly with Keycloak and KEYCLOAK_* envs included
docker compose up -d in keycloak-mock/ imports the dls realm and exposes the admin console at http://localhost:8080
./scripts/k8s/dev-k8s.sh up brings Keycloak up alongside the rest of the stack, reachable at http://<node-ip>:30090
Frontend (Add Keycloak authentication to smartem app smartem-frontend#74 with its silent-SSO and gate fixes) logs in via the mock and the account menu shows the user identity
Backend pod with KEYCLOAK_AUTH_REQUIRED=true fetches JWKS from http://keycloak-service:8080 and accepts a real devuser token

Matches the realm-side rename in smartem-devtools (feat/keycloak-mock, DiamondLightSource/smartem-devtools#198) where the existing browser client is renamed for symmetry with the new SmartEM_Agent service client. See DiamondLightSource/smartem-devtools/decisions/0018 for context.

…list, add clock leeway Authentication is now always on - no env-driven opt-out. With the local Keycloak mock in DiamondLightSource/smartem-devtools#198 available for tests and local dev, the flag added more failure modes (silent unauthenticated traffic on misconfig) than it averted. Adds KEYCLOAK_ALLOWED_AZP enforcement so the backend can reject tokens issued to clients outside the expected set (SmartEM_User for the browser, SmartEM_Agent for the agent). Empty value preserves the prior "any valid token from the realm" behaviour. Also adds a 60-second leeway on time-based JWT claims to absorb modest workstation clock skew. Tests that previously relied on auth being off by default now use app.dependency_overrides[verify_token] to inject a stub claims dict. Same treatment in the three test files that define their own client fixture rather than using conftest. A separate real_auth_client fixture exercises the actual verify_token path in test_auth.py. uv.lock updated to pick up pyjwt[crypto]'s transitive cryptography dependency that lands with the backend extra. Towards resolving #284.

Matches the realm-side rename in smartem-devtools (feat/keycloak-mock, DiamondLightSource/smartem-devtools#198) where the existing browser client is renamed for symmetry with the new SmartEM_Agent service client. See DiamondLightSource/smartem-devtools/decisions/0018 for context.

…list, add clock leeway Authentication is now always on - no env-driven opt-out. With the local Keycloak mock in DiamondLightSource/smartem-devtools#198 available for tests and local dev, the flag added more failure modes (silent unauthenticated traffic on misconfig) than it averted. Adds KEYCLOAK_ALLOWED_AZP enforcement so the backend can reject tokens issued to clients outside the expected set (SmartEM_User for the browser, SmartEM_Agent for the agent). Empty value preserves the prior "any valid token from the realm" behaviour. Also adds a 60-second leeway on time-based JWT claims to absorb modest workstation clock skew. Tests that previously relied on auth being off by default now use app.dependency_overrides[verify_token] to inject a stub claims dict. Same treatment in the three test files that define their own client fixture rather than using conftest. A separate real_auth_client fixture exercises the actual verify_token path in test_auth.py. uv.lock updated to pick up pyjwt[crypto]'s transitive cryptography dependency that lands with the backend extra. Towards resolving #284.

The smartem-frontend Keycloak integration currently has no local development story — every dev needs `http://localhost:5173` added to the SmartEM client's Valid Redirect URIs and Web Origins on `identity-test.diamond.ac.uk`, which is an admin round-trip that has to be repeated for every new port and every new developer. A self-contained mock removes that dependency: - `dls-realm.json` is the single source of truth — realm `dls`, public client `SmartEM` with PKCE, localhost redirect URIs and Web Origins, custom `fedId` claim mapper, two seeded users. - Compose form for the fastest standalone cycle. - Kustomize form integrated into the existing development overlay, so `dev-k8s.sh up` now brings up Keycloak alongside the rest of the stack. Both forms read the same realm JSON, so editing it once propagates to whichever form a developer prefers. The architecture doc gains a "Local development" pointer and the client-name discrepancy (`smartem-frontend` vs `SmartEM`) is corrected to match the actual implementation.

…p-api The development environment's `smartem-http-api-service` already claims NodePort 30080. Applying the keycloak-mock kustomization to the development overlay therefore fails with a port conflict. Move the keycloak-external Service to 30090 and update the three doc references that mention 30080. The browser-facing URL becomes `http://<node-ip>:30090`; the ClusterIP path (`http://keycloak-service:8080`) is unchanged.

The backend (smartem-decisions) now consumes KEYCLOAK_AUTH_REQUIRED / KEYCLOAK_URL / KEYCLOAK_REALM / KEYCLOAK_CLIENT_ID / KEYCLOAK_VERIFY_ISS to gate Bearer-token validation against a Keycloak realm's JWKS. - Staging defaults to the real DLS test realm (https://identity-test.diamond.ac.uk, realm `dls`, client `SmartEM`) with auth required and iss verification on. - Development defaults to the in-cluster keycloak-mock service (http://keycloak-service:8080) with iss verification OFF. Tokens are minted with the browser-facing URL (localhost:30090) while the pod reaches Keycloak via in-cluster DNS, so issuer strings don't match even though the signing key does. Auth itself defaults to disabled in dev so existing workflows that don't pass a token still work; flip it on per developer via `.env.k8s.development`. `dev-k8s.sh ensure_app_configmap()` now also reads the five Keycloak variables from the environment and bakes them into the dynamically- generated ConfigMap, so an operator who edits `.env.k8s.staging` and runs `DEPLOY_ENV=staging ./scripts/k8s/dev-k8s.sh up` gets the right values without touching YAML. Examples in `env-examples/.env.example.k8s.{staging,development}` are updated to document the new variables.

…ent agent auth The SmartEM Agent needs a service-to-service authentication path against the JWT-protected backend. Adds a confidential SmartEM_Agent client to the mock realm (OAuth 2.0 client_credentials grant) and renames the existing browser client SmartEM to SmartEM_User for symmetry in the realm. The choice is recorded in ADR 0018 with the operational details in docs/agent/authentication.md and short cross-links from cli-reference, deployment, troubleshooting, environment-variables and http-api-client. A future upgrade to private_key_jwt client authentication is planned and the agent code is structured so that change requires no behavioural rework. Resolves DiamondLightSource/smartem-decisions#284 (decision).

…list, add clock leeway Authentication is now always on - no env-driven opt-out. With the local Keycloak mock in DiamondLightSource/smartem-devtools#198 available for tests and local dev, the flag added more failure modes (silent unauthenticated traffic on misconfig) than it averted. Adds KEYCLOAK_ALLOWED_AZP enforcement so the backend can reject tokens issued to clients outside the expected set (SmartEM_User for the browser, SmartEM_Agent for the agent). Empty value preserves the prior "any valid token from the realm" behaviour. Also adds a 60-second leeway on time-based JWT claims to absorb modest workstation clock skew. Tests that previously relied on auth being off by default now use app.dependency_overrides[verify_token] to inject a stub claims dict. Same treatment in the three test files that define their own client fixture rather than using conftest. A separate real_auth_client fixture exercises the actual verify_token path in test_auth.py. uv.lock updated to pick up pyjwt[crypto]'s transitive cryptography dependency that lands with the backend extra. Towards resolving #284.

Matches the realm-side rename in smartem-devtools (feat/keycloak-mock, DiamondLightSource/smartem-devtools#198) where the existing browser client is renamed for symmetry with the new SmartEM_Agent service client. See DiamondLightSource/smartem-devtools/decisions/0018 for context.

Adds Kubernetes manifests for the smartem-frontend image produced by smartem-frontend#94 (v0.2.0, now on GHCR). The image is environment-agnostic: it ships a placeholder config.json with dev defaults and proxies /api/ to the backend service via its own nginx, deferring DNS to request time. What lands per environment (k8s/environments/<env>/ smartem-frontend.yaml): - ConfigMap smartem-frontend-config carries the runtime config.json (Keycloak URL/realm/clientId + authEnabled). The Deployment subPath-mounts this onto /usr/share/nginx/html/config.json, overriding the placeholder shipped in the image. - Deployment smartem-frontend pulls ghcr.io/diamondlightsource/smartem-frontend:latest, sets BACKEND_HOST=smartem-http-api-service so the SPA pod's nginx proxies /api/ to the backend service. - Service smartem-frontend-service: NodePort 30100 for development (next free in the 30000s range; matches the existing smartem-http-api / Keycloak / RabbitMQ / Postgres / Adminer pattern), ClusterIP for staging and production. Per-environment config.json values: - development: keycloak.url http://localhost:30090 (the Keycloak mock NodePort - browser-reachable), authEnabled false to match KEYCLOAK_AUTH_REQUIRED=false on the dev backend. Flip to true to exercise the full login chain. - staging: identity-test.diamond.ac.uk, authEnabled true. - production: identity.diamond.ac.uk, authEnabled true. Ingress (staging and production only - dev keeps the NodePort pattern): - k8s/environments/{staging,production}/ingress.yaml route a single host to smartem-frontend-service. The SPA pod's nginx handles /api/ proxying to the backend internally, so one route covers everything. Hostnames are placeholders (smartem-staging.example.com / smartem.example.com) and flagged TODO until real values are decided. scripts/k8s/dev-k8s.sh: print the new http://localhost:30100 (frontend) and http://localhost:30090 (Keycloak, missed when #198 landed) in the access-URLs section. Verified locally: kubectl kustomize build is clean for all three environments. End-to-end browser flow (SPA login, authenticated /api call) will be exercised on the user's local k3s after merge.

github-actions Bot added documentation Improvements or additions to project documentation devops CI/CD, deployment, infrastructure, or tooling work labels May 12, 2026

vredchenko mentioned this pull request May 14, 2026

Add Keycloak authentication to smartem app DiamondLightSource/smartem-frontend#74

Merged

7 tasks

vredchenko changed the title ~~feat: local Keycloak mock for SmartEM frontend dev~~ feat: local Keycloak mock + backend auth env wiring May 14, 2026

This was referenced May 14, 2026

feat: validate Keycloak Bearer tokens via FastAPI dependency DiamondLightSource/smartem-decisions#285

Merged

DLS Keycloak: add SmartEM_Agent client, rename SmartEM to SmartEM_User #200

Open

vredchenko and others added 4 commits May 21, 2026 11:08

vredchenko force-pushed the feat/keycloak-mock branch from 8803cc1 to 9a1ecf2 Compare May 21, 2026 10:08

vredchenko merged commit a3a2de6 into main May 21, 2026
7 checks passed

vredchenko deleted the feat/keycloak-mock branch May 21, 2026 10:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: local Keycloak mock + backend auth env wiring#198

feat: local Keycloak mock + backend auth env wiring#198
vredchenko merged 4 commits into
mainfrom
feat/keycloak-mock

vredchenko commented May 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

vredchenko commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

What's included

Self-contained Keycloak mock (keycloak-mock/)

Backend Keycloak env wiring

Port-conflict fix (fix(keycloak-mock): move NodePort to 30090)

Docs

Quick start

Verified end-to-end

Test plan

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vredchenko commented May 12, 2026 •

edited

Loading

Self-contained Keycloak mock (`keycloak-mock/`)

Port-conflict fix (`fix(keycloak-mock): move NodePort to 30090`)