feat: agentic-server stateful conversation/responses endpoints.

[maralbahari]

Summary

To be reviewed after #46

Wires agentic-core's executor into agentic-server, giving the gateway
stateful conversation and response management on top of the existing vLLM
proxy. Implements the Layer 2 (HTTP API) component of
ADR-03.

New HTTP endpoints:

Endpoint	Method	Routes to
/v1/conversations	POST	create_conversation(): creates a DB-backed conversation record; response body includes id (the conversation_id) along with created_at, object, and metadata
/v1/responses	POST	executor or proxy, depending on store field; executor path returns a ResponsePayload JSON object (id, object, created_at, model, status, output, usage, previous_response_id, conversation_id, …) for non-streaming requests, or an SSE stream of the same shape for streaming; proxy path forwards the vLLM response verbatim

Routing logic (store field):

• store=true (default per spec) → executor path: execute() handles rehydration, LLM inference, and persistence
• store=false → proxy path: request forwarded directly to vLLM, no DB involvement

Both paths share a single /v1/responses handler that reads the body once, peeks at store, and dispatches without re-buffering.

Rehydration scope (conversation_id field):

Within the executor path, /v1/responses selects its rehydration strategy based on the presence of conversation_id in the request body:

• conversation_id present → rehydrate_from_conversation(): loads the full conversation turn history from the DB and reconstructs context for the next turn
• conversation_id absent, previous_response_id present → rehydrate_from_response(): loads only the chained response history, with no conversation record involved

State design:

AppState holds both states simultaneously, with no feature flags or optional fields:

AppState {
    proxy_state: ProxyState          // always present, handles store=false
    exec_ctx: Arc<ExecutionContext>  // always present, handles store=true
    llm_api_base: String
}

Config gains an optional db_url field. The server defaults to sqlite://./agentic_api.db when unset.

Per-request auth override:

If the incoming Authorization: Bearer <token> differs from the configured key,
a lightweight ExecutionContext clone is created for that request so auth is
not shared across concurrent requests.

Test Plan

Unit / integration tests (13 new, all passing):

tests/responses_test.rs (4 tests):

• store=false proxies JSON response from mock vLLM
• store=false proxies SSE stream from mock vLLM with correct content-type
• store=true reaches executor path, not the proxy's 200
• Oversized body returns 413

tests/conversations_test.rs (3 tests):

• store=false in body returns 400
• Empty body defaults store=true, reaches executor path
• No body defaults store=true, reaches executor path

tests/health_test.rs and tests/cors_test.rs refactored to share helpers via tests/common/mod.rs.

All 135 tests pass across the workspace.

cargo test

Running Tests

cargo test -p agentic-server

Running Benchmarks

# All benchmarks (proxy + conversation rehydration)
cargo bench --bench benches

# Conversation rehydration only
BENCH_TURNS=3 cargo bench --bench benches -- conversation_rehydration

# Against a real vLLM (model auto-detected from /v1/models)
BENCH_TURNS=3 LLM_BASE_URL=http://localhost:9090 \
    cargo bench --bench benches -- conversation_rehydration

# Explicit model name
BENCH_TURNS=3 LLM_BASE_URL=http://localhost:9090 \
    BENCH_MODEL=Qwen/Qwen3-30B-A3B-FP8 \
    cargo bench --bench benches -- conversation_rehydration

Benchmark Results

Model: Qwen/Qwen3-30B-A3B-FP8 · GPU: A100 · vLLM http://localhost:5050 · sample-size=10 · turns 1–10.

BENCH_TURNS=10 LLM_BASE_URL=http://localhost:9090 \
    BENCH_MODEL=Qwen/Qwen3-30B-A3B-FP8 \
    cargo bench -p agentic-server --bench benches -- \
        conversation_rehydration response_rehydration --sample-size 10

---

conversation_rehydration

Turn	Non-streaming (median)	CI (low–high)	Streaming (median)	CI (low–high)
1	2.06 s	1.61–2.50 s	1.66 s	1.37–1.97 s
2	2.24 s	1.91–2.57 s	2.08 s	1.78–2.33 s
3	2.59 s	2.23–2.98 s	2.19 s	1.80–2.63 s
4	2.46 s	2.08–2.85 s	2.83 s	2.33–3.31 s
5	2.17 s	1.68–2.62 s	2.60 s	2.14–3.15 s
6	2.50 s	2.03–2.98 s	1.99 s	1.72–2.31 s
7	2.34 s	1.93–2.81 s	2.72 s	2.24–3.25 s
8	2.44 s	2.03–2.85 s	2.54 s	2.16–2.94 s
9	2.45 s	1.88–3.09 s	2.71 s	2.30–3.16 s
10	2.84 s	2.52–3.21 s	2.80 s	2.43–3.17 s

---

response_rehydration

Turn	Non-streaming (median)	CI (low–high)	Streaming (median)	CI (low–high)
1	2.85 s	2.52–3.13 s	2.98 s	2.53–3.47 s
2	2.32 s	2.06–2.59 s	2.67 s	2.37–2.96 s
3	2.77 s	2.56–3.04 s	2.33 s	1.84–2.87 s
4	2.79 s	2.46–3.10 s	2.43 s	2.16–2.70 s
5	2.23 s	1.78–2.76 s	2.40 s	2.12–2.67 s
6	2.13 s	1.78–2.48 s	2.94 s	2.34–3.58 s
7	2.03 s	1.75–2.34 s	2.90 s	2.45–3.54 s
8	2.66 s	2.26–3.10 s	1.81 s	1.38–2.32 s
9	2.69 s	2.43–2.96 s	1.75 s	1.44–2.03 s
10	2.42 s	2.12–2.73 s	2.44 s	2.18–2.74 s

---

Analysis

Both paths stay well under 3 s across all turn counts. Gateway overhead (DB reads, rehydration, prompt reconstruction) is not the bottleneck; LLM inference time dominates, which is why confidence intervals are wide (often spanning ~0.5–1 s) even with 10 samples.

conversation_rehydration shows a mild upward trend with turn count. Non-streaming median grows from ~2.1 s at turn 1 (no prior context) to ~2.8 s at turn 10 (9 prior turns rehydrated), an incremental cost of roughly 0.08 s per prior turn. This is expected: each additional turn adds to the reconstructed prompt sent to the LLM. The cost is low enough that it does not accumulate dangerously over long conversations.

response_rehydration shows no clear trend with chain depth. Medians are flat and noisy across turns (2.03–2.85 s non-streaming), which suggests the DB fetch overhead per chained response is negligible relative to inference variance.

Streaming and non-streaming medians are comparable per turn. Because benchmarks seed prior turns before Criterion starts timing, each measurement isolates a single turn's round-trip cost. The similarity confirms that SSE framing adds no meaningful overhead on the gateway side.

conversation_rehydration is modestly faster than response_rehydration at low turn counts (2.06 s vs 2.85 s at turn 1). At higher turn counts they converge. The turn-1 gap reflects the fact that response rehydration always issues a DB lookup for the prior response even on the first measured turn, while conversation rehydration with no prior context skips that step.

[maralbahari]

@franciscojavierarceo @ashwing @leseb Ready for review

[ashwing]

Nice work wiring the executor into the server — clean routing design. Left some inline comments, mostly architectural questions for the future (tool dispatch, concurrency). Nothing blocking.

[ashwing]

The body is parsed twice: read_body deserializes to serde_json::Value to peek at store, then execute_responses re-parses into RequestPayload. For the proxy path this is fine (raw bytes forwarded), but for the executor path the double parse adds CPU cost that scales with input size.

Not blocking — conversations are small today. But when history gets long, consider parsing once into Value and using serde_json::from_value() for the executor path.

[maralbahari]

@ashwing I refactored further to avoid the double parsing.

[ashwing]

When the client sends a different Bearer token, the entire ExecutionContext is cloned (both handlers, client Arc, config) to set one field. Under concurrency this creates many short-lived Arc<ExecutionContext> allocations.

Could client_auth move to being a parameter of execute() or a field on RequestContext (which is already per-turn)? That way ExecutionContext stays truly shared — one Arc for the server lifetime.

[maralbahari]

@ashwing yeh I agree I do not like the current way of handling client_auth but the change requires a bit of a refactoring in agentic-core. so we can use builder pattern where we use RequectContext().auth().run(). to not expand this PR larger with such refactoring I'll open another PR to address this.

[ashwing]

Clean design — when tool dispatch lands, the change is just:

// Current:
match execute(payload, resolve_exec_ctx(state, &parts)).await {
// Future:
match execute_loop(payload, resolve_exec_ctx(state, &parts), tool_ctx).await {

Should ToolContext live in AppState (configured at startup)? Or constructed per-request from the request's tools array? If per-request, the handler builds it here.

[maralbahari]

@ashwing I think it should be handled per-request the app state is only one upon application run while the ToolContext would change and vary per-request.

[ashwing]

If a client sends {"store": false, "previous_response_id": "resp_abc"}, this routes to proxy_responses which forwards bytes to vLLM. But vLLM doesn't know about previous_response_id — it ignores it.

The client thinks they're continuing a conversation, but they're getting a fresh response with no history. Should store=false + previous_response_id either (a) validate the ID exists then proxy, or (b) reject with 400?

[maralbahari]

@ashwing thanks for catching. I added a check in response function before diverging between proxy and gateway response rehydration path to return with InvalidRequest error.

[ashwing]

Architectural note for future: two simultaneous requests with the same conversation_id both rehydrate the same item_ids snapshot, both infer, both persist. The second persist overwrites — first request's items get lost from the ordered list.

Not a bug today (single-user, low concurrency). Worth a TODO for when multi-user or high-concurrency scenarios arise. Options: optimistic locking (version column), or append-only with conflict detection.

[maralbahari]

@ashwing I think this should be handled in agentic-core/storage module not in server level cause it would be messy to handle in server level. while in storage module we can get help of DB transaction for concurrent request for such cases. will add this into Issue.

[ashwing]

If the body is valid JSON but missing model (empty string after deser), the executor forwards it to vLLM which returns its own 400/422. This works (error surfaces via ExecutorError::LLMRequest) but the error message comes from vLLM, not us.

Similarly POST /v1/conversations ignores all fields except store. Is that intentional per spec, or should we validate required fields at the handler layer for clearer error messages?

[maralbahari]

@ashwing for v1/converstion it is intentional cause it is only creating a conversation session. for the model request validity actually I tested by not passing correct name and our gateway catches the error from proxy as we should to tell the user their requested model is wrong. we could actually use v1/models from vLLM server to actually get the model name if user request payload didn't send the model name or even verify the request payload matches the what is actually hosted through v1/models. also we should expose v1/models to our agentic-api gateway to proxy to upstream.

I am thinking once we have most of the components in we can have a request payload validation once per request. right now the validation of store, previouse_response_id, conversation_id and etc are all scattered and making the bookkeeping difficult. we could think of a design the request payload is instantiated we do the validations per case documented in its struct . like request_payload.validation_for_proxy , request_payload.validate_for_conv_rehydration` etc.

[leseb]

a few things from my agent:

• [P1] Kill vLLM when DB initialization fails
  server.rs:77
  In serve mode, if DATABASE_URL is invalid/unwritable or schema
  setup fails after the child LLM becomes ready, this ? returns
  before the cleanup block at the end of the function, leaving the
  spawned vLLM process running. Please either initialize state before
  spawning or kill/wait the child on this error path.

• [P2] Preserve no-store continuations through the executor
  handler.rs:180-183
  For store:false requests that include a stored previous_response_id
  or conversation_id, the core executor already supports rehydrating
  prior history while skipping persistence for the new response. This
  preemptive 400 breaks that stateless-continuation use case and forces
  clients either to persist the new turn or lose context; these requests
  should route through the executor instead of being rejected.

• [P2] Return 4xx for malformed response requests
  handler.rs:63-64
  When a client sends malformed JSON or a schema-invalid body to
  /v1/responses, this wraps the serde_json::Error as JsonError;
  ExecutorError::http_status() falls through to 500 for that variant,
  so bad client input is reported as an internal server error. Map
  request-body parse failures to an invalid-request/parse error here
  so callers get a 4xx.
  ════════════════════════════════════════════════════════════

[maralbahari]

a few things from my agent:

• [P1] Kill vLLM when DB initialization fails
  server.rs:77
  In serve mode, if DATABASE_URL is invalid/unwritable or schema
  setup fails after the child LLM becomes ready, this ? returns
  before the cleanup block at the end of the function, leaving the
  spawned vLLM process running. Please either initialize state before
  spawning or kill/wait the child on this error path.
• [P2] Preserve no-store continuations through the executor
  handler.rs:180-183
  For store:false requests that include a stored previous_response_id
  or conversation_id, the core executor already supports rehydrating
  prior history while skipping persistence for the new response. This
  preemptive 400 breaks that stateless-continuation use case and forces
  clients either to persist the new turn or lose context; these requests
  should route through the executor instead of being rejected.
• [P2] Return 4xx for malformed response requests
  handler.rs:63-64
  When a client sends malformed JSON or a schema-invalid body to
  /v1/responses, this wraps the serde_json::Error as JsonError;
  ExecutorError::http_status() falls through to 500 for that variant,
  so bad client input is reported as an internal server error. Map
  request-body parse failures to an invalid-request/parse error here
  so callers get a 4xx.
  ════════════════════════════════════════════════════════════

@leseb
fixed the 1st and the 3rd item.
for the second item in this PR I enabled the path to run the rehydration execution path when store is set to false but context id is given,
and I created a new PR fix #56. cuz in core the contract is set to only check store value to persist or not. I checked the case with OpenAI behavior as well. added a new cassettes when the first turn we set store as True but the consecutive turn store is set to false with given context id . so our gateway api contract is updated based on OpenAI in the PR #56 fix.