.NET: DevUI limitation should be documented and addressed if possible

[Vijay-Nirmal]

Limitation Details

• Only workflows and agents that are dependency injected can be visualized in the DevUI.
• Workflows must be registered using the agents using AddAsAIAgent to be visualized and funcational.
• Non agentic workflows must take ChatMessage as input in order to run the workflow in the DevUI.
• Conversations API failes if agents in not dependency injected but the parent workflow is.
• Non agentic workflow should be named using the WithName (provide name param for agentic workflow), which must match the workflow name in the dependency injection registration.
• Chat History/Agent Thread is not maintaince
• By default, only AgentsAsExecutor can be executed in the DevUI; otherwise, every executor should be inherited from ChatProtocolExecutor or needs to support getting both List<ChatMessage> and TurnToken as input. See the sample below:

// Disclaimer: This might not be the intended way to implement an executor, but is a workaround to make it work in DevUI.
internal sealed class UppercaseExecutor() : Executor("UppercaseExecutor")
{
    private List<ChatMessage> _messages = [];

    protected override Microsoft.Agents.AI.Workflows.RouteBuilder ConfigureRoutes(Microsoft.Agents.AI.Workflows.RouteBuilder routeBuilder)
    {
        return routeBuilder
            .AddHandler<List<ChatMessage>>(this.RouteMessages)
            .AddHandler<TurnToken, string>(this.RouteTurnTokenAsync);   // here `string` is the input type of the next executor (output of this)
           // Add your orginal input type as a new handler here
    }

    private ValueTask RouteMessages(List<ChatMessage> messages, IWorkflowContext context, CancellationToken cancellationToken)
    {
        this._messages = messages;
        return ValueTask.CompletedTask;
    }

    private ValueTask<string> RouteTurnTokenAsync(TurnToken token, IWorkflowContext context, CancellationToken cancellationToken)
    {
        // Not sure why DevUI wraps the actual input this way...
        return ValueTask.FromResult(JsonDocument.Parse(this._messages.Last().Text).RootElement.GetProperty("input").GetString()!.ToUpperInvariant());
    }
}

• Details about inner agents (Agents inside Executor) itself like Tools calls will not be shown

Expectation

• For errors/exceptions, DevUI should show a toast message or some indication of what went wrong instead of just failing silently.
• DevUI should support non agentic workflows and executors more seamlessly without the need for workarounds.
• Simplify the process of registering and visualizing workflows in DevUI, such as not requiring the use of WithName to match the DI registration name or calling AddAsAIAgent.
• Provide alternative ways to register workflows and agents that can be visualized in DevUI without strict dependency injection requirements.
• Improved documentation on how to set up workflows and agents with all the requirements and limitations for DevUI visualization.

[alliscode]

Thanks for filling this issue @Vijay-Nirmal.

@victordibia, @ReubenBond it would be good to document the current state and also create tracking issues for some of these things.

[jozkee]

I couldn't figure out how to use the executor from

agent-framework/dotnet/samples/GettingStarted/Workflows/ConditionalEdges/01_EdgeCondition/Program.cs

Line 149 in 863a632

internal sealed class SpamDetectionExecutor : Executor<ChatMessage, DetectionResult>

With AddAsAIAgent().

I tried the workaround described by @Vijay-Nirmal but that didn't fix my issue. I can't switch it to SpamDetectionExecutor : ChatProtocolExecutor because I don't see a HandleAsync method that let me send the output to the next edges.

[Vijay-Nirmal]

@jozkee Try something like below using Executor<T1, T2> directly (Since Executor<T1, T2> in hearited from Executor). Note: Details about SpamDetectionAgent itself like Tools calls will not be shown

var workflow = new WorkflowBuilder(spamDetectionExecutor)
    .WithName("email-assistant")
    .AddEdge(spamDetectionExecutor, emailAssistantExecutor, condition: GetCondition(expectedResult: false))
    .AddEdge(emailAssistantExecutor, sendEmailExecutor)
    .AddEdge(spamDetectionExecutor, handleSpamExecutor, condition: GetCondition(expectedResult: true))
    .WithOutputFrom(handleSpamExecutor, sendEmailExecutor)
    .Build();
builder.AddWorkflow("email-assistant", (sp, key) => workflow).AddAsAIAgent();


internal sealed class SpamDetectionExecutor : Executor<ChatMessage, DetectionResult>
{
    private readonly AIAgent _spamDetectionAgent;
    private List<ChatMessage> _messages = [];

    public SpamDetectionExecutor(AIAgent spamDetectionAgent) : base("SpamDetectionExecutor")
    {
        this._spamDetectionAgent = spamDetectionAgent;
    }

    protected override Microsoft.Agents.AI.Workflows.RouteBuilder ConfigureRoutes(Microsoft.Agents.AI.Workflows.RouteBuilder routeBuilder)
    {
        base.ConfigureRoutes(routeBuilder);
        return routeBuilder
            .AddHandler<List<ChatMessage>>(this.RouteMessages) // For DevUI
            .AddHandler<TurnToken, DetectionResult>(this.RouteTurnTokenAsync); // For DevUI
    }

    private ValueTask RouteMessages(List<ChatMessage> messages, IWorkflowContext context, CancellationToken cancellationToken)
    {
        this._messages = messages;
        return ValueTask.CompletedTask;
    }

    private ValueTask<DetectionResult> RouteTurnTokenAsync(TurnToken token, IWorkflowContext context, CancellationToken cancellationToken)
    {
        // Not sure why DevUI wraps the actual input this way...
        return this.HandleAsync(new ChatMessage(ChatRole.User, JsonDocument.Parse(this._messages.Last().Text).RootElement.GetProperty("input").GetString()!.ToUpperInvariant()), context, cancellationToken);
    }

    public override async ValueTask<DetectionResult> HandleAsync(ChatMessage message, IWorkflowContext context, CancellationToken cancellationToken = default)
    {
        var newEmail = new Email
        {
            EmailId = Guid.NewGuid().ToString("N"),
            EmailContent = message.Text
        };
        await context.QueueStateUpdateAsync(newEmail.EmailId, newEmail, scopeName: EmailStateConstants.EmailStateScope, cancellationToken);

        var response = await this._spamDetectionAgent.RunAsync(message, cancellationToken: cancellationToken);
        var detectionResult = JsonSerializer.Deserialize<DetectionResult>(response.Text);

        detectionResult!.EmailId = newEmail.EmailId;

        return detectionResult;
    }
}

[jozkee]

@Vijay-Nirmal thanks that fixes it for the workflow view. For the agent view I still see an error, I would expect the workflow as agent to respond with the output specified in WithOutputFrom.

Also, if I remove .AddAsAIAgent(), the workflow is still visible in DevUI but is non-functional.

[jozkee]

Additionally, in the sample I mentioned, it shows you how to wrap an agent with an Executor<TInput, TOutput>, it would be nice that you could simply use myAgent.BindAsExecutor<TOutput>() rather than having to implement the wrapper type. BindAsExecutor() right now returns an ExecutorBinding wrapping a ChatProtocolExecutor which doesn't output anything.