Skip to content

Follow-up prompts should use explicit timeouts, retry/backoff, and optional fallback #6421

Description

@michaelmanly

Feature Description

## Question

Would maintainers be open to a small PR that documents or improves support for OpenAI-compatible gateway base URLs?

Some production users route LLM calls through a gateway layer so they can manage:

- retries
- fallback routing
- request tracking
- provider timeout handling
- debugging failed provider calls

For example, an OpenAI-compatible client can usually point at a gateway by changing only the API key and base URL:

```ts
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.BADGR_API_KEY,
  baseURL: "https://api.aibadgr.com/v1",
});

Why I’m asking

I noticed a few provider-call paths where reliability behavior may currently depend on the underlying provider/client defaults:

  • packages/components/src/followUpPrompts.ts

    • follow-up prompt generation appears to call providers without a consistent explicit timeout or abort signal
    • transient 429 / 5xx provider errors may fail immediately
    • the path appears to use the selected provider only, without an optional fallback route
  • packages/components/nodes/chatmodels/ChatOpenAI/ChatOpenAI.ts

    • OpenAI chat timeout appears to be optional and only set when the user provides one
  • packages/components/src/speechToText.ts

    • speech-to-text provider calls appear to run without a shared timeout/retry policy
  • packages/components/nodes/retrievers/VoyageAIRetriever/VoyageAIRerank.ts

    • reranker errors appear to fall back silently to unranked documents, which can make degraded quality harder to debug

These may already be handled elsewhere, so I wanted to ask before opening a larger PR.

Possible small PR options

Happy to follow whatever maintainers prefer:

  1. a small docs PR showing how to configure an OpenAI-compatible gateway base URL
  2. a small config improvement for custom OpenAI-compatible base URLs
  3. a small timeout/retry improvement in one provider path
  4. improved structured error metadata for provider failures
  5. no change, if this is outside the project scope

The goal would be optional production reliability support without changing default behavior.

Feature Category

UI/UX Improvement

Problem Statement

No response

Proposed Solution

No response

Mockups or References

No response

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions