Tracing configuration similar to MetricsBuilder

[lmolkova]

When configuring distributed tracing, it's useful to be able to enable and disable sources based on name(space) and other properties.

For example, Azure SDKs create multiple sources per library:

• Azure.Messaging.ServiceBus.SenderClient,
• Azure.Messaging.ServiceBus.ReceiverClient
• .. but also also to trace individual message via Azure.Messaging.ServiceBus.Message.

Some customer want per-message tracing, some don't, and we need to be able to tell them to enable Azure.* (or Azure.Messaging.ServiceBus.*) but disable the Azure.Messaging.ServiceBus.Message.

Another example is more common: client libraries want to report logical operations (that trace public API surface calls), but want to make protocol level opt-in. E.g. Azure.CosmosDb.Operation is a source for logical operations that we want users and OTel distros to enable by default and Azure.CosmosDb.Request would contain more verbose network traces that users should be able to opt into.
With MetricsBuilder-like API this would look like

builder.Services.AddDistributedTracing(b => b.EnableTracing("Azure.")
                                  .DisableTracing("Azure.CosmosDb.Request"));

It's not super-convenient for end-users but should be reasonable for OTel Distros. OTel might provide better ways to do it (see below).

So, it would be great to have parity with metrics and provide builder-like API allowing to enable and disable sources.

At the same time, name is not enough/future-proof in both (tracing and metrics) cases.

• When ActivitySource supports scope attributes [Feature request]: Support instrumentation scope attributes on ActivitySource #93795, they could be useful in EnableSource to enable/disable based on. Same is already the case for meter - scope attributes and version could be useful in IMetricsBuilder.EnableMetrics
• OTel at some point might introduce some concepts like verbosity and mechanism to enable/disable tracers/sources based on it. See Turn off spans from a scope while retaining downstream spans and without breaking traces open-telemetry/opentelemetry-specification#3867 for the context.

It would be great to support some form of EnableSource(Predicate<ActivitySource>) or a list of EnableSource(string name, string? version, IEnumerable<>? scopeAttributes, ....) overloads.

Separate from this issue, but it would be great to have consistent IMetricsBuilder.Enable|Disable(Func<Meter, Instrument, bool>) API or something else that allows to control metrics based on all meter properties.

See open-telemetry/opentelemetry-dotnet#5353 for more details.

API Proposal

edited by @rosebyte

namespace Microsoft.Extensions.DependencyInjection
{
    public static class TracingServiceExtensions
    {
        // Adds tracing services to the specified IServiceCollection.
        public static ITracingBuilder AddTracing(this IServiceCollection services);

        // Adds tracing services and invokes tracing configuration.
        public static ITracingBuilder AddTracing(this IServiceCollection services, Action<ITracingBuilder> configure);
    }
}

namespace System.Diagnostics
{
    public abstract class ActivitySourceFactory : IDisposable
    {
        public ActivitySource Create(ActivitySourceOptions options);
        public ActivitySource Create(string name, string? version = null, IEnumerable<KeyValuePair<string, object?>>? tags = null);
        public void Dispose();

        protected abstract ActivitySource CreateCore(ActivitySourceOptions options);
        protected virtual void Dispose(bool disposing);
    }

    // Changed from sealed to non-sealed.
    public class ActivitySource : IDisposable
    {
        // Returns the scope object (typically, the IActivitySourceFactory it was created by).
        public object? Scope { get; }

        // ActivitySource instances are cached in the factory, so the protected dispose pattern
        // (together with unsealing ActivitySource) prevents consumers from disposing them prematurely.
        protected virtual void Dispose(bool disposing);
    }

    public class ActivitySourceOptions
    {
        // Optional opaque object to attach to the ActivitySource (typically, the IActivitySourceFactory it was created by).
        public object? Scope { get; set; }
    }

    public partial class ActivityListener
    {
-       public string? Name { get; }
-       public ActivityListener(string? name);

        public void RefreshSources();
    }
}

namespace Microsoft.Extensions.Diagnostics.Tracing
{
    // A builder to improve discoverability and to prevent polluting IServiceCollection with another batch of extension methods.
    public interface ITracingBuilder
    {
        IServiceCollection Services { get; }
    }

+    public sealed class ActivityListenerBuilder
+    {
+         public string Name { get; }
+         public SampleActivity<ActivityContext>? Sample { get; set; }
+         public SampleActivity<string>? SampleUsingParentId { get; set; }
+         public Action<Activity>? ActivityStarted { get; set; }
+         public Action<Activity>? ActivityStopped { get; set; }
+         public ExceptionRecorder? ExceptionRecorder { get; set; }
+     }
}

    // Represents scopes used by TracingRule to distinguish global vs local activity sources.
    [Flags]
    public enum ActivitySourceScopes
    {
        // No scope is specified.
        None   = 0,

        // ActivitySource instances created via constructors.
        Global = 1,

        // ActivitySource instances created via IActivitySourceFactory.
        Local  = 2,
    }

    // A set of parameters used to determine which activities are enabled for which listeners.
    public class TracingRule
    {
        public TracingRule(string? sourceName, string? operationName, string? listenerName, ActivitySourceScopes scopes, bool enable);

        // The ActivitySource.Name to match, either exact or single-wildcard pattern (e.g. "Azure.*").
        public string? SourceName { get; }

        // The Activity.OperationName to match, exact match only (no wildcards).
        // Allows enabling/disabling specific activities within a source, e.g. disabling
        // "Azure.Messaging.ServiceBus.Message" while the rest of "Azure.*" stays enabled.
        public string? OperationName { get; }

        // The IActivityListener.Name to match, exact match.
        public string? ListenerName { get; }

        // Configured scopes.
        public ActivitySourceScopes Scopes { get; }

        // Whether matched activities are enabled.
        public bool Enable { get; }
    }

    // Options for configuring the tracing system.
    public class TracingOptions
    {
        // The list of rules that identifies which activity sources, activities, and listeners are enabled.
        public IList<TracingRule> Rules { get; }
    }

    // Extension methods for configuring tracing listeners and rules.
    public static class TracingBuilderExtensions
    {
-       // Registers a new listener type.
-       public static ITracingBuilder AddListener<[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] T>(this ITracingBuilder builder)
-           where T : class, IActivityListener;

-       // Registers a new listener type with a factory.
-       public static ITracingBuilder AddListener<T>(this ITracingBuilder builder, Func<IServiceProvider, T> factory)
-           where T : class, IActivityListener;

-       // Registers a listener instance.
-       public static ITracingBuilder AddListener(this ITracingBuilder builder, IActivityListener listener);

+       public static ITracingBuilder AddListener(this ITracingBuilder builder, string name, Action<ActivityListenerBuilder> configure);
 
+       public static ITracingBuilder AddListener(this ITracingBuilder builder, string name, Action<IServiceProvider, ActivityListenerBuilder> configure);

        // Removes all registered listeners.
        public static ITracingBuilder ClearListeners(this ITracingBuilder builder);

        // Enables activities for a source/activity/listener/scopes combination.
        public static ITracingBuilder EnableTracing(this ITracingBuilder builder, string? sourceName = null, string? operationName = null, string? listenerName = null, ActivitySourceScopes scopes = ActivitySourceScopes.Global | ActivitySourceScopes.Local);

        // Enables activities in options for a source/activity/listener/scopes combination.
        public static TracingOptions EnableTracing(this TracingOptions options, string? sourceName = null, string? operationName = null, string? listenerName = null, ActivitySourceScopes scopes = ActivitySourceScopes.Global | ActivitySourceScopes.Local);

        // Disables activities for a source/activity/listener/scopes combination.
        public static ITracingBuilder DisableTracing(this ITracingBuilder builder, string? sourceName = null, string? operationName = null, string? listenerName = null, ActivitySourceScopes scopes = ActivitySourceScopes.Global | ActivitySourceScopes.Local);

        // Disables activities in options for a source/activity/listener/scopes combination.
        public static TracingOptions DisableTracing(this TracingOptions options, string? sourceName = null, string? operationName = null, string? listenerName = null, ActivitySourceScopes scopes = ActivitySourceScopes.Global | ActivitySourceScopes.Local);
    }
}

namespace Microsoft.Extensions.Diagnostics.Tracing.Configuration
{
    // AddConfiguration is in its own static class (mirrors MetricsBuilderConfigurationExtensions),
    // so it can ship in Microsoft.Extensions.Diagnostics without leaking IConfiguration into the
    // Microsoft.Extensions.Diagnostics.Abstractions surface.
    public static class TracingBuilderConfigurationExtensions
    {
        // Registers configuration for tracing.
        public static ITracingBuilder AddConfiguration(this ITracingBuilder builder, IConfiguration configuration);
    }   

+   public abstract class ActivityListenerConfigurationFactory
+   {
+       public abstract IConfiguration GetConfiguration(string listenerName);
+   }
}

[lmolkova]

based on the discussion with otel-dotnet people, they are not open to take contributions and are not ready to take "maintenance burden".

Given that it's problematic for any big set of libraries and turning of sources is one of a common asks we have on Azure SDKs specifically in .NET, I'd like to check if we can provide ability to turn off sources on the BCL level.

[lmolkova]

@jeffschwMSFT @mkArtakMSFT I think it belongs in dotnet/runtime repo (Microsoft.Extensions.Diagnostics component) and the area should be diagnostics. Could you please check?
Thanks!