-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Add agent routing and model routing pages for RoutedAgent/RoutedLlm in ADK JS #1709
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
b1c6881
28e1265
89f6ee9
b41275f
4c24d79
70f0ca4
53b918e
3b0d056
f3a4a3e
8169ae8
45afd58
5a742e8
6c943ec
fb95a81
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # Route between models | ||
|
|
||
| <div class="language-support-tag"> | ||
| <span class="lst-supported">Supported in ADK</span><span class="lst-typescript">TypeScript v1.0.0</span><span class="lst-preview">Experimental</span> | ||
|
koverholt marked this conversation as resolved.
|
||
| </div> | ||
|
|
||
| !!! example "Experimental" | ||
|
|
||
| Model routing is experimental and may change in future releases. We welcome | ||
| your | ||
| [feedback](https://github.com/google/adk-js/issues/new?template=feature_request.md)! | ||
|
|
||
| An `LlmAgent` uses a single model by default. When you need to dynamically | ||
| select between different models for each request, you can define a routing | ||
| function that chooses which model to use. `RoutedLlm` provides this capability, | ||
| enabling model fallback on error, A/B testing between models, and auto-routing | ||
| by input complexity. If the selected model fails before producing any output, | ||
| the routing function is called again with error context so it can select a | ||
| different model. | ||
|
|
||
| Pass a `RoutedLlm` as an `LlmAgent`'s `model` parameter. Use `RoutedLlm` when | ||
| only the model varies between routes. If you also need to switch instructions, | ||
| tools, or sub-agents, use [`RoutedAgent`](../routing.md) instead. | ||
|
|
||
| ## How routing works | ||
|
|
||
| The `LlmRouter` function receives the map of available models and the current | ||
| `LlmRequest`, and returns the key of the model to use: | ||
|
|
||
| === "TypeScript" | ||
|
|
||
| ```typescript | ||
| type LlmRouter = ( | ||
| models: Readonly<Record<string, BaseLlm>>, | ||
| request: LlmRequest, | ||
| errorContext?: { failedKeys: ReadonlySet<string>; lastError: unknown }, | ||
| ) => Promise<string | undefined> | string | undefined; | ||
| ``` | ||
|
|
||
| The `models` parameter accepts either a `Record<string, BaseLlm>` with explicit | ||
| keys, or an array of `BaseLlm` instances. If an array is provided, each model's | ||
| name is used as its key. | ||
|
|
||
| Failover follows the same rules as | ||
| [`RoutedAgent`](../routing.md#how-routing-works): the router is re-called with | ||
| `errorContext` only if the selected model fails before yielding any response. | ||
| After yielding, errors propagate without retry. The router can return | ||
| `undefined` to stop retrying and propagate the last error. | ||
|
|
||
| **Live connections:** `RoutedLlm.connect()` selects the model at connection | ||
| time. Once a live connection is established, the model cannot be switched | ||
| mid-stream. | ||
|
|
||
| ## Basic usage | ||
|
|
||
| The following example creates a `RoutedLlm` that tries a primary model first and | ||
| falls back to a secondary model if the primary fails. The router checks | ||
| `errorContext.failedKeys` to avoid re-selecting the failed model: | ||
|
|
||
| === "TypeScript" | ||
|
|
||
| ```typescript | ||
| --8<-- "examples/typescript/snippets/agents/models/routing/basic-usage.ts:full" | ||
| ``` | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Similarly, I'd prefer that we name this file /agents/routing.md rather than /agents/agent-routing.md to keep the file names from getting too long.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. furthermore, I'm not sure I agree this is an "Agents" feature, but an "Agent Runtime" feature, because the RoutedAgent is selecting agents, and not actually running an agent process itself.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Renamed in 4c24d79.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I went back and forth on this one. However, |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
| # Route between agents | ||
|
|
||
| <div class="language-support-tag"> | ||
| <span class="lst-supported">Supported in ADK</span><span class="lst-typescript">TypeScript v1.0.0</span><span class="lst-preview">Experimental</span> | ||
|
koverholt marked this conversation as resolved.
|
||
| </div> | ||
|
|
||
| !!! example "Experimental" | ||
|
|
||
| Agent routing is experimental and may change in future releases. We welcome | ||
| your | ||
| [feedback](https://github.com/google/adk-js/issues/new?template=feature_request.md)! | ||
|
|
||
| When building agents for different tasks, you can define a routing function that | ||
| selects which one handles each invocation at runtime. `RoutedAgent` provides | ||
| this capability, enabling agent fallback on error, A/B testing, planning modes, | ||
| and auto-routing by input complexity. If the selected agent fails before | ||
| producing any output, the routing function is called again with error context so | ||
| it can select a fallback. | ||
|
|
||
| `RoutedAgent` is different from [workflow agents](workflow-agents/index.md) like | ||
| `SequentialAgent` or `ParallelAgent`, which orchestrate multiple agents in a | ||
| fixed pattern, and from [LLM-driven | ||
| delegation](multi-agents.md#b-llm-driven-delegation-agent-transfer), where the | ||
| LLM decides which agent to hand off to. With `RoutedAgent`, you write an | ||
| explicit routing function that selects **one** agent per invocation. For | ||
| model-level routing, see [Model routing](models/routing.md). | ||
|
|
||
| ## How routing works | ||
|
|
||
| Both `RoutedAgent` and [`RoutedLlm`](models/routing.md) are powered by a shared | ||
| routing utility that handles selection and failover. | ||
|
|
||
| The router function receives the map of available agents and the current | ||
| context, and returns the key of the agent to run. It can be synchronous or | ||
| async: | ||
|
|
||
| === "TypeScript" | ||
|
|
||
| ```typescript | ||
| type AgentRouter = ( | ||
| agents: Readonly<Record<string, BaseAgent>>, | ||
| context: InvocationContext, | ||
| errorContext?: { failedKeys: ReadonlySet<string>; lastError: unknown }, | ||
| ) => Promise<string | undefined> | string | undefined; | ||
| ``` | ||
|
|
||
| **The `agents` parameter** accepts either a `Record<string, BaseAgent>` with | ||
| explicit keys, or an array of agents. If an array is provided, each agent's | ||
| `name` property is used as its key. | ||
|
|
||
| **Failover behavior:** | ||
|
|
||
| - The router is first called without `errorContext` to make the initial | ||
| selection. | ||
| - If the selected agent throws an error **before yielding any events**, the | ||
| router is called again with `errorContext` containing `failedKeys` and | ||
| `lastError`. | ||
| - If the selected agent throws an error **after yielding events**, the error | ||
| propagates directly without retry, because partial results have already been | ||
| emitted. | ||
| - A key that has already been tried cannot be re-selected. If the router returns | ||
| a previously failed key, the error propagates. | ||
| - If the router returns `undefined`, routing stops and the last error is thrown. | ||
|
|
||
| ## Basic usage | ||
|
|
||
| Create multiple agents, define a router function that returns a key, and wrap | ||
| them in a `RoutedAgent`. The following example routes between two agents based | ||
| on an external configuration value that can change between invocations: | ||
|
|
||
| === "TypeScript" | ||
|
|
||
| ```typescript | ||
| --8<-- "examples/typescript/snippets/agents/routing/basic-usage.ts:full" | ||
| ``` | ||
|
|
||
| Change `config.selectedAgent` to `'agent_b'` before the next invocation to | ||
| route to a different agent. | ||
|
|
||
| ## Fallback on error | ||
|
|
||
| When an agent fails, the router is called again with `errorContext` so it can | ||
| select a fallback. Failover only applies if the agent fails before yielding any | ||
| events (see [How routing works](#how-routing-works)). The following example | ||
| checks `errorContext.failedKeys` to avoid re-selecting the failed agent: | ||
|
|
||
| === "TypeScript" | ||
|
|
||
| ```typescript | ||
| --8<-- "examples/typescript/snippets/agents/routing/fallback.ts:config" | ||
| ``` | ||
|
|
||
| ## Planning mode | ||
|
|
||
| A router can read any external state to select between agents with different | ||
| instructions, models, and tools. This lets you implement a planning mode where | ||
| the agent switches behavior dynamically. For example, a basic agent might have | ||
| read and write tools, while a planning agent is restricted to read-only access | ||
| and uses a more powerful model for analysis. | ||
|
|
||
| The following example shows a different `RoutedAgent` configuration. See [basic | ||
| usage](#basic-usage) for the full runner setup. | ||
|
|
||
| === "TypeScript" | ||
|
|
||
| ```typescript | ||
| --8<-- "examples/typescript/snippets/agents/routing/planning-mode.ts:config" | ||
| ``` | ||
|
|
||
| Set `planningMode = true` before an invocation to route to the planning agent | ||
| with its restricted tool set and different instructions. | ||
|
|
||
| ## Auto-routing by complexity | ||
|
|
||
| The router function can call a lightweight classifier model to categorize input | ||
| and route to different agents accordingly. Because the router can be async, you | ||
| can make LLM calls inside it before selecting an agent. | ||
|
|
||
| The following example shows a different `RoutedAgent` configuration. See [basic | ||
| usage](#basic-usage) for the full runner setup. | ||
|
|
||
| === "TypeScript" | ||
|
|
||
| ```typescript | ||
| --8<-- "examples/typescript/snippets/agents/routing/auto-routing.ts:config" | ||
| ``` | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| // Copyright 2026 Google LLC | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| // --8<-- [start:full] | ||
| import { | ||
| BaseLlm, | ||
| Gemini, | ||
| LlmRequest, | ||
| LlmAgent, | ||
| RoutedLlm, | ||
| InMemoryRunner, | ||
| } from '@google/adk'; | ||
|
|
||
| const primaryModel = new Gemini({ model: 'gemini-flash-latest' }); | ||
| const fallbackModel = new Gemini({ model: 'gemini-pro-latest' }); | ||
|
|
||
| const router = ( | ||
| models: Readonly<Record<string, BaseLlm>>, | ||
| request: LlmRequest, | ||
| // errorContext is provided when a previously selected model fails | ||
| errorContext?: { failedKeys: ReadonlySet<string>; lastError: unknown }, | ||
| ) => { | ||
| if (!errorContext) { | ||
| return 'primary'; // Try primary first | ||
| } | ||
| if (errorContext.failedKeys.has('primary')) { | ||
| return 'fallback'; // Fall back if primary failed | ||
| } | ||
| return undefined; // No more options, propagate the error | ||
| }; | ||
|
|
||
| const routedLlm = new RoutedLlm({ | ||
| models: { primary: primaryModel, fallback: fallbackModel }, | ||
| router, | ||
| }); | ||
|
|
||
| // Use RoutedLlm as the model for an LlmAgent | ||
| const agent = new LlmAgent({ | ||
| name: 'my_agent', | ||
| model: routedLlm, | ||
| instruction: 'You are a helpful assistant.', | ||
| }); | ||
|
|
||
| const runner = new InMemoryRunner({ agent, appName: 'my_app' }); | ||
|
|
||
| const session = await runner.sessionService.createSession({ | ||
| appName: 'my_app', | ||
| userId: 'user_1', | ||
| }); | ||
|
|
||
| const run = runner.runAsync({ | ||
| userId: 'user_1', | ||
| sessionId: session.id, | ||
| newMessage: { role: 'user', parts: [{ text: 'Hello!' }] }, | ||
| }); | ||
|
|
||
| for await (const event of run) { | ||
| if (event.content?.parts?.[0]?.text) { | ||
| console.log(event.content.parts[0].text); | ||
| } | ||
| } | ||
| // --8<-- [end:full] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| { | ||
| "name": "adk-docs-examples", | ||
| "version": "1.0.0", | ||
| "description": "TS code examples for the ADK Documentation", | ||
| "main": "index.js", | ||
| "scripts": { | ||
| "test": "echo \"Error: no test specified\" && exit 1", | ||
| "build": "tsc", | ||
| "clean": "rm -rf dist" | ||
| }, | ||
| "keywords": ["adk", "google", "agent", "typescript", "gemini"], | ||
| "author": "", | ||
| "license": "Apache-2.0", | ||
| "type": "commonjs", | ||
| "devDependencies": { | ||
| "@types/node": "^20.14.2", | ||
| "typescript": "^5.9.2" | ||
| }, | ||
| "dependencies": { | ||
| "@google/adk": "^1.0.0" | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| { | ||
| "compilerOptions": { | ||
| /* Build Options */ | ||
| "target": "es2020", | ||
| "module": "nodenext", | ||
| "moduleResolution": "nodenext", | ||
| "outDir": "./dist", | ||
| "declaration": true, | ||
| "declarationMap": true, | ||
| "sourceMap": true, | ||
|
|
||
| /* Strictness */ | ||
| "strict": true, | ||
| "noUnusedLocals": true, | ||
| "noUnusedParameters": true, | ||
|
|
||
| /* Module Interop */ | ||
| "esModuleInterop": true, | ||
| "skipLibCheck": true, | ||
| "forceConsistentCasingInFileNames": true | ||
| }, | ||
| "exclude": ["node_modules", "dist"] | ||
| } |
Uh oh!
There was an error while loading. Please reload this page.