Extend expression trees to cover more/all language constructs

[gafter]

Expression trees today support a limited subset of expression lambdas. This issue is to track the request to extend them to support more or all of the language constructs that can be expressed in a lambda.

A prototype for work in this direction can be found on https://github.com/bartdesmet/ExpressionFutures/tree/master/CSharpExpressions and https://github.com/bartdesmet/roslyn/tree/ExpressionTrees (based on a fork of Roslyn a while ago).

/cc @bartdesmet

[bbarry]

previous discussion: https://github.com/dotnet/roslyn/issues/2060

notable things missing:

• null-coalescing operators such as a?.b, a?[b]
• support for dynamic
• support for await
• default parameters
• named parameters
• base access
• assignment
• multi-dim array initialization
• dictionary initialization
• increment/decrement
• statements
• unsafe code
• out and ref parameters
• method group references
• ref returns
• tuples
• throw expression
• patterns

[pei-boomsoftware]

extension members

[NetMage]

Adding comment from issue #503:

Expression Trees should be extended to support the C# 6.0 null propagating operator so that it can be used with Linq, with a translation to allow providers such as Linq to SQL to add support.

The discussion on CodePlex doesn't seemed to have moved over to here, and has been sitting around for 3 years - I think it is worth revisiting, and ideally implementing. An ideal use case is an Outer Join in Linq to Objects.

https://roslyn.codeplex.com/discussions/571077

[stevozilik]

+1
Not having Null-propagating operator ?. in Expression Trees is a major blocker for unit-testing left outer joins

[stakx]

I'd like to add my vote for this proposal for the sake of the Moq 4.x mocking library, whose API relies heavily on expression tree lambdas. In particular, support for the following two language features in expression tree lambda would be hugely beneficial to Moq. We could then fix the majority of currently reported bugs, and add some long requested features:

1. assignments
2. ref returns — @bbarry, perhaps you could add this language feature to your list above?

I'm happy to further explain my reasons for prioritising these two, but don't want to go off on a tangent unbidden.

[olmobrutall]

With the (awesome) addition of nullable reference types in C# 8.0 having null-propagation operation in expression trees goes from "very important" to "you almost can not use Entity Framework without this feature".

Looking forward for knowing the design details of how this could be implemented in order to adapt my LINQ provider.

[alrz]

notable things missing:

I wonder if local functions could be translated to SQL CTEs.

[stakx]

Bump.

@gafter, I was wondering whether there's any news from the C# language team about this. I understand that it's probably more interesting to design and plan brand-new language features like Span<T> etc. than to maintain an established one, but at the same time we have this amazing feature -- code as data, in the form of expression trees -- that gets totally left behind. C# is now at version 7.1, while expression trees are still at approx. C# 4 or 5. The gap keeps widening, and the relative limitations of expression trees therefore become more obvious and more frustrating.

Is there any chance this can be given a higher priority? Is there anything the community can do to help advance expression trees?

[scalablecory]

Async/await support is probably the largest undertaking on that list, but I'd love it. It would significantly help projects performing complex code-gen.

[stakx]

@scalablecory - I totally agree this would be nice. That being said, I question the likeihood of getting async / await support in expression trees when even fundamental and much more basic language features such as = (assignment) aren't supported. 😢

I think it would be awesome if we could just get the ball rolling, i. e. we could get agreement from the C# language team and/or the Roslyn team to look at the most basic missing features for a start. (I'd be happy to help in any way I reasonably can.) If we manage to get there, perhaps the more complex features will eventually follow, too.

[gafter]

It was discussed briefly on Monday... noting for example that we currently do not have support for ?. and friends, and that will become more annoying as we add support for non-nullable reference types.

/cc @dotnet/ldm It would help if we had a member of the LDM who championed this (even though it isn't technically a language feature). Any takers?

[bbarry]

I think there would need to be a language change to support statements in expression trees because expression tree types explicitly states:

• Not all lambda expressions can be converted to expression trees. For instance, lambda expressions with statement bodies, and lambda expressions containing assignment expressions cannot be represented. In these cases, a conversion still exists, but will fail at compile-time. These exceptions are detailed in Anonymous function conversions.

emphasis added.

For everything else:

the precise rules for constructing an expression tree when a lambda expressionis converted to an expression tree type, are both outside the scope of this specification.

Should it be (libraries that walk these often assume specific patterns; the fact that according to the language spec they appear to be able to change between compiler versions is surprising isn't it)?

(OT, it looks like there are numerous spaces missing from this document after the word "expression"; are PRs being accepted for these sorts of things yet?)

[gafter]

@bbarry PRs to fix typos in the spec are most welcome.

[olmobrutall]

@gafer Is there any issue where the changes on the expression API are being discussed?

[gafter]

Not that I am aware

[olmobrutall]

I don’t see ?. in expression trees anywhere in the selected features for c# 8.

Any chance it will be included? Is considered a part of nullable reference types?

I had three bugs in production because of SQL implicit nulls in left joins...

What can we do to help this move forward?

[zspitz]

@orthoxerox Couldn't Roslyn itself be used for runtime code generation? Except that AFAICT Roslyn cannot be run in a single-threaded environment.

[olmobrutall]

https://github.com/dotnet/csharplang/blob/main/meetings/2018/LDM-2018-06-06.md

For now, let's mull it over for a couple of months, pursue information, crisp up the plan. Only after a while will we be able to free up compiler resources.

That was 50 months ago :)

[CyrusNajmabadi]

Code generators only came into existence in the last two years, and are held to same general roslyn standard that this API develops at the very least at the pace of the language.

[bachratyg]

@olmobrutall

In order to make the transition process as easy as possible, something like following helper classes

I believe this functionality is already available via Expression.Reduce

[bachratyg]

Code generators only came into existence in the last two years, and are held to same general roslyn standard that this API develops at the very least at the pace of the language.

So if a code generator encounters an unknown construct in the code that it's not prepared to handle (e.g. required) it would generate bad code or simply crash?

[CyrusNajmabadi]

So if a code generator encounters an unknown construct in the code that it's not prepared to handle (e.g. required) it would generate bad code or simply crash?

IMO, it should emit a diagnostic stating that it is not compatible with this language construct.

[olmobrutall]

I believe this functionality is already available via Expression.Reduce

AFAIK, Expression.Reduce is for lowering custom expression nodes to standard ones, for example before calling .Compile().

I don’t think Expression.Coalesce reduces to a Expression.Condition, so I don’t think Expression.NullPropagation should either.

Instead Expression.NullPropagation should compile to IL just fine out of the box

[stakx]

Given the discussion above it doesn't seem very probable that the present System.Linq.Expressions trees will simply get extended with additional node types. I'm wondering what some alternative options might be.

Creating a totally new expression tree abstraction whose design somehow avoids the extensibility issues of what we're dealing here? (What would be the point of abandoning SLE only to end up in the exact same situation in a few years' time, only the compiler team would then have two legacy expression tree models instead of one?) What would such a radical re-design have to look like?

Or – I'm sure it's been discussed already, but I can't remember if & where I saw that idea – wouldn't it be possible to introduce an Roslyn extensibility point akin to source generators, which would allow user-code libraries to provide compile-time "code quotation factories" that turn a Roslyn AST of type D (where D is any delegate type) into an AST of type E<D> (where E would be any custom type linked e.g. via custom attributes to the appropriate quotation factory)? Roslyn would recognize E<D> as a code quotation type, fire up the provided factory & invoke it, and replace the original delegate expression with whatever that factory produces; basically a mechanism for user-defined implicit coercion, restricted to delegate types.

The point of this idea being to make code quotations fully customizable and thereby shifting the maintenance burden away from the compiler into libraries which can be cleanly versioned.

Those "quotation factories" could even signal (by emitting Roslyn diagnostics) what kind of expressions are supported / not supported by whatever target library the factories are associated with.

(Whether or not System.Linq.Expressions would be extracted from the compiler into an external code quotation factory lib (but still in the FCL) would remain to be seen.)

When I last thought about this idea a few years ago, I thought it would stand no chance, since it might tie the C# spec too tightly to one implementation (namely Roslyn) because of the AST model required during transformation (you wouldn't want to invent yet another code tree model, right?)... but since then source generators have come into being; would this idea be so much different from them?

[WhitWaldo]

A lot of the concern on this thread speaks to the use of expression trees in EF, but I also wanted to highlight their extensive use in the somewhat recently open-sourced Reaqtive project. This project provides a platform for hosted reactive operations allowing for stateful stateful checkpointing and recovery of standing and active subscriptions.

It provides libraries for serialization of expression trees (allowing for queries and their components to be saved in stateful storage while at rest), which is huge in itself. However, the implementation of and extendibility of the observers and observables could be made dramatically simpler if .NET had support for async expression trees, so I'd ask that these be considered in the next overhaul of expression trees, whatever shape that takes.

I understand the argument in here for breaking old dependencies on existing expression trees, but I don't necessarily see why this isn't something that is run in the .NET Core style of move-faster-likely-breaking-things as things transitioned from .NET 4 to .NET 5.

Given that the implementation of each of the proposed changes is thought to be pretty straightforward and because Bart has already done some exploration in what might be needed to support different branches of support with the ExpressionTreeLike types and even has a sample implementation of this, why can this not simply be something that is opt-in, similar to that of the nullability flag in csproj wherein developers can choose to use the legacy expression tree (selected by default) or can opt into a more modern expression tree that will continually evolve as C# does.

I don't know to what extent .NET is able to do feature flags, as it were, in the code, but what if modern expression support were an opt-in on a feature-by-feature basis? Today, my .csproj contains:

<PropertyGroup>
  <Nullable>enable</Nullable>
</PropertyGroup>

Why can't modern expression trees be similar? This would enable all available modern expression tree functionality (similar to setting language support to vNext in that it's likely something will break with new changes).

<PropertyGroup>
  <ModernExpressions>true</ModernExpressions> 
</PropertyGroup>

And especially since there's such a laundry list of individual features, allow them to be opt-in by name:

<PropertyGroup>
  <ModernExpressions>
    <NullCoaselscing>true</NullCoalescing>
    <Async>true</Async>
    <Dynamic>true</Dynamic>
  </ModernExpressions>
</PropertyGroup>

Those features that are inextricably linked to all of them are included just by enabling ModernExpressions (doesn't happen often, but does happen per the opt-in contract). But this means you don't have to enable all the new features all at once in a single release and trivially opens the door to revisiting to add new expressions features in a way that people can use if they want or simply opt-out of.

At launch, there's zero net impact until developers decide to proactively opt into the new functionality - a great many will, but if their application breaks, I suspect they'll be more reasonable about the migration. After launch, there's some impact as developers upgrade packages indiscriminately, but the analyzer saves them from themselves here. And in the longer-term, new applications get to enjoy rich expression tree support and older applications keep on using the legacy approach.

[aradalvand]

It's strange that something as important as this is not even being discussed. What's blocking this?!

[CyrusNajmabadi]

A proof of concept through at least C# 9 or so has already been created demonstrating a viable path forward ... (nullability, for example)

Nullability has absolutely no runtime impact. That's precisely why it was ok to go that route, and it's a strong reason why it could be added without being impactful to the ecosystem. That's very different from any approach that could actually end up impacting anything at runtime. :(

[WhitWaldo]

That's a good point, but at the same time, it begs the question of whether the future of C# development is just syntax updates devoid of any runtime changes for the same underlying concern of breaking some ancient backwards compatibility somewhere.

I was reading through the proof of concept last night to get a better understanding how it it's implemented and the changes to Roslyn itself don't appear to be that substantial which led me to the following. Your concern is that by changing the runtime for expressions, it could have disastrous and unknown impacts to developers consuming that functionality. The approach used in the proof of concept appears to be one that allows for over-time extensibility so as to avoid this point-in-time feature lock, but implementing that approach requires breaking the existing language runtime contract. The best path forward here then is to just leave existing expression trees exactly as they are indefinitely moving forward and chalk it up to a poor design decision in early C# days.

But then why can't you then implement the newer, extensible expression trees with the bare minimum runtime support necessary under a different namespace removed from the domain of the language team? I anticipate you saying that this just moves the needle from C# 3.0 feature lock to C# 11.0 - fair, so why not shift the actual implementation of these modern expression trees out to a separate package. The .NET library ecosystem is broad and rich and has a readily understood contract in place regarding versioning, lifecycle and modernity expectations.

The language ensures backwards compatibility to a back-breaking level - I get that, but separate implementation packages don't generally have nearly the same concern for backwards compatibility. But further, if I need what's in v1.0 and v2.0 breaks my implementation, I need only install v1.0 and compatibility is on me as the developer to ensure for a safe upgrade path.

Could this be a more acceptable path forward?

• Using a different namespace means that no changes are made to the existing implementation ensuring strict backwards compatibility
• Tweaks to the runtime are simply made in a greenfield namespace and care is taken to design this as extensibly as possible going forward to avoid the same pitfall
• Shifting the implementation to a separate package means the approach can be iterated on and shifts responsibility from the language team to the developers to ensure a package upgrade doesn't break their application in a way, while still allowing them to access and install these older versions for self-imposed feature lock

Thanks for the ongoing consideration!

[CyrusNajmabadi]

devoid of any runtime changes

Yes. We work steadfastly to not make runtime changes, except in very narrow circumstances, and when teh net benefit will likely be enormous, and highly unlikely to break anyone.

For example, we are considering changing how exceptions are thrown in async/await. This should not be noticeable at runtime, unless doing very specific groveling of the low level async/await machinery. Similarly, we may allow a params Span<T> overload to be 'better' than a params T[] overload (with the presumption that types that want to use params-span will be well-behaved and will only have different performance characteristics, not runtime semantic differences.

However, the above is very different from expression trees.

[CyrusNajmabadi]

But then why can't you then implement the newer, extensible expression trees with the bare minimum runtime support necessary under a different namespace removed from the domain of the language team?

I don't see any reason why we could not do that. I may have missed it, but did someone give a reason why that would not be possible? :)

I anticipate you saying that this just moves the needle from C# 3.0 feature lock to C# 11.0 - fair, so why not shift the actual implementation of these modern expression trees out to a separate package. The .NET library ecosystem is broad and rich and has a readily understood contract in place regarding versioning, lifecycle and modernity expectations.

I would not say this would have C# 11 lockin. Indeed, if doing this again, i would model around Roslyn, with the same promises (or more specifically, the lack thereof :)) around compat. Specifically, i would say that you should expect breaks, and that as the language revs, this API will move to match it, and it's incumbent on the consumer to deal with that (the same way we deal with roslyn changes today).

Thanks for the ongoing consideration!

Definitely :)

[WhitWaldo]

Circling back here.. if simply using a different namespace for Expressions v2 is a viable path forward for modernizing it, what are the remaining blockers, if any, to moving forward? Is there anything I can personally do to help shuffle this along apart from merely advocating for it?

[Atulin]

Found this discussion while writing yet another EF query that required me to do

.Select(x => new Dto {
    Foo = x.Foo == null ? x.Date : x.Foo
})

instead of the sane thing,

.Select(x => new Dto {
    Foo = x.Foo ?? x.Date
})

and I'm guessing I'll have to use it a while longer still

[mattwar]

That's just the coalesce operator, which has always been an expression tree node type. I'm guessing that's just a failure on the EF provider to not understand how to translate the coalesce operator into the SQL COALESCE function.

[Atulin]

This comment — dotnet/efcore#33336 (comment) — mentions that it is a limitation of expression trees.

That said... I was looking for instances of the issue in my codebase to show real-life examples, and I'm having trouble finding any. So either I hallucinated it, or it's not actually the issue I had in mind. I'll pop back into this discussion should I actually find something

[MaceWindu]

?? was never introduced to expression tree, which is probably main PITA with expression trees

[aradalvand]

Having just re-read this entire thread, all the while carefully considering the concerns and explanations provided by the likes of @CyrusNajmabadi, @333fred, et al. I must say they are making about as much sense as how many "new" C# syntactic features have had support added for them in expression trees over the last decade or so, which is, of course, a glorious zero.

Let's consider this comment by @333fred — as I think it succinctly and clearly presents the supposed justification:

The significant cost of implementing new nodes does not come from the compiler. It's fairly easy to do in the compiler. Its significant cost comes from every downstream tool now being broken because there is a new node in the tree they were not expecting, and us needing to work with EF and other consumers of expressions to create a way to signal language version support.

(emphasis mine)

Sounds reasonable, right? Except that it should take no more than 5 seconds for anyone remotely familiar with C#/.NET's and the surrounding ecosystem's progression over the last couple of years, to realize two things, which immediately obliterate this reasoning:

Swap the terms "new node" with "new built-in anything" and you'll get an equally sound argument for never adding any feature at any level in any area to the framework/standard library ever; because "what about existing tools that won't immediately support it?", I guess. It is unbelievably unconvincing.

There is literally a fundamental analogy here between any hypothetical new expression nodes, and any new standard constructs in general — with the most notable recent example being what happened with TimeOnly and DateOnly: They got added to the standard library, tools like EF, various serializers, etc. didn't immediately have support for them, so any usage in such unsupported contexts resulted in either a runtime exception or unexpected behavior at the time, until support gradually got added. How would new expression nodes be fundamentally ANY different than that?

Notice that according to the same logic as the one that we're being provided here as to why expressions are stuck at the C# of 10 years ago, apparently the addition of TimeOnly and DateOnly would've been a heinous, unthinkably nightmarish idea, because:

The significant cost of implementing [TimeOnly and DateOnly] does not come from the [standard library]. It's fairly easy to do in the [standard library]. Its significant cost comes from every downstream tool now being broken because there is a new [built-in type] in the [framework's standard library] they were not expecting, and us needing to work with EF and other consumers of [user code] to create a way to signal [framework] version support.

It gets even worse, this argument is, in fact, even LESS applicable in the case of expression trees, because all consumers of expression trees (most prominently EF), ALREADY ONLY support a subset of "all possible expression trees". An expression tree consumer not supporting a certain thing at a given time is already the status quo, and as such, fully expected.

Funny. Some people in this thread (e.g. @olmobrutall here) tried to point out the same thing, heck, even some comments on related Stack Overflow questions made this point, and made more basic logical sense than anyone from the other side here. I'm just trying to help reveal the folly in this reasoning more clearly and bring more awareness to the issue.

[CyrusNajmabadi]

Could the entirety of expression trees be replaced with a mechanism that would produce Roslyn SyntaxNodes?

No. SyntaxNodes are just raw syntax. They do not contain important, necessary, semantics (like "what on earth does this foo syntax mean here").

[AartBluestoke]

Why doesn't the Roslyn compiler itself suffer from the same difficulty? Every time a new SyntaxNode type is added, isn't there some potential to break the shape of the parsed tree, and thus break any parsing code? How are language features ever added to C#?

But i think it does - new code structures can't be compiled using old roslyn's, they would return a syntax error.
The new bytecode output is fully backwards compatible with the runtime - and that is where massive effort in Roslyn is, and outcomes of executing that compiled code don't change (up to performance, and a few documented "undefined" outcome cases like what value does an uninitialized value hold)
I think the problem here is that new AST's have to be run through old AST interpreters.

The compiler is also not constrained to emit the same byte code from version to version, only that executing the bytecode comes with the same outcome (eg, i think how errors are handled in async can produce different bytecode in different roslyn version, but no effective difference in outcomes other that performance considerations)

An old AST might have a switch (node N){
typeof(this): x=this.thing
typeof(that): x=that.foo
... covering examples over all known examples
}
and no null check on x here. a new node type causes acrash.
even any existing code being interpreted as the new node type would cause a crash (eg, upstream library introduces an optimised 'null_coalesce ' node, which then gets passed to the runtime.

In this scenario, the source code hasn't changed, and the destination framework hasn't changed, just a library somewhere in the middle of a stack getting a version bump which "only" contains optimisations could be enough to induce runtime AST crashes.

There is also the development cost - issues with Roslyn interpretation are caught with red squigllies as you type. AST's are pure interpretation, so you only find out you've done a 'silly' at runtime, potentially quite a bit later in the dev cycle.

[ranma42]

Analyzers and/or compile-time translation of the expressions might help detecting what unsupported expressions.
These two are new-ish and still very much in progress in several contexts (see, for example, the ongoing work to translate queries AoT in EFCore), but maybe they could help in avoiding the issue of runtime failures.

[olmobrutall]

The code won’t fail if you don’t use the feature.

How do you communicate "the feature" to customers? Basically how can an API, documentation, etc ... properly communicate what type of C# syntax is / isn't legal. (…) Basically it's not about APIs supporting specific expression tree types, it's about supporting the types in the many different contexts in which they can occur.

What expression trees is a LINQ provider going to accept is a grey area already. Some mini-LINQ providers maybe only accept MemberAccessExpression because they are only meant to refer to properties (like strongly-typed reflection) while others could support almost everything. I can see that is tempting for you to be able to restrict this at compile time with some versioning or a massive attribute enumerating the supported ExpressionNodeTypes but it is very complicated, think of this query:

Query<Person>().SelectMany(p=>p.Age>18?p.Cars:p.Parents.Cars)

The ?: maybe is accepted in many context but not here because you cant use a SQL CASE for the source of a query.

Even without considering new nodes, the developer already has to know which methods and properties are supported by the LINQ provider and what not, this is a much bigger gray area anyway.

So… just not solve this communication issue. LINQ to expressions base some pitfalls
but is still very useful.

This is further complicated because there is no specification for how expression tree nodes are generated. The original native compiler implementation was effectively a 1:1 mapping of internal data structures to nodes. Consumers are incredibly fragile to the shape of the generated trees (basically fragile to internal data structures of the native compiler).

Roslyn nodes are full fidelity, so I can see there is tension a versioning issues.

Expression trees on the other side are more abstract and easily consumable:

• there is no trivia
• no difference between whether you call an extension method as static or as instance method
• no difference between query syntax or method chasing

Also, the API validates at run-time the types of the arguments etc quite strictly, so there is not a lot of freedom to make another valid expression tree really. I can only think maybe inserting/removing some unnecessary castings or a different method overloads resolution (already happening?)

Could you make an example of c# code that could produce two different valid expression trees and would benefit from a change?

The best way we've found to model this is treating expression trees like language versions. Essentially you can pick the language version that expression trees are generated with and consumers can declare what is the max version, or set of versions, they support. That version would end up getting passed along with the tree so that consumers could better handle the nodes. That gives us better tools to push consumers and produces to a place of goodness, not guessing.

The original spirit of the expression tree api was to implement like 40% of the language (only expressions) to provide 90% of the value when translating to SQL, and I think this was very wise.

Maybe there are some use cases for a new SuperExpressionTrees that supports statements and all the new features (shaders?, auto-derivation for tensors computations?) but the change will be massive and not backwards compatible, so if it ever gets done should be an different API and not an expansion of the current one.

But if LINQ was designed with the same spirit but today I think they would have increased the amount of language supported just a little bit more:

• Support ?. navigating any reference in SQL using . produces a LEFT OUTER JOIN. If the references is nullable this is semantically a ?.

Query<Person>().Select(a=>a.Car?.RegistrationDate)

but this is not supported.. so you have to write !. instead… semantically crazy but works most of the time… except in the select statement

Query<Person>().Select(a=>a.Car!.RegistrationDate) // works… except of the Car is null, then you get a cast exception from null to DateTime

some other additions that don’t even require new exprsssion nodes:

• Support short syntax for tuples.
• Support optional parameters.

This is why it is not just like adding DateOnly or TimeOnly. Expression trees aren't measured in that simple way. The nodes exist in a fragile ecosystem with heavily intertwined decisions.

I need an example to see why is this so complicated, the current compiler is able to translate Roslyn nodes into old expression trees. I don’t think minimally increasing the target nodes would be so hard but an example would help.

If you use the feature, and is implemented in the LINQ provider is all 🦄 and 🌈

Sure but how do you discover what the provider supports? Type some code, execute, get an exception, change the style a bit and repeat until success? That doesn't sound like a good developer experience.

As I mention, knowing what is supported is very complicated and the list of supported members is more important than the list of nodes. A Roslyn analyzer is probably a much better solution than versioning at the compiler level… but is very challenging because the Queryable API is shared.

If you all can accept the idea of implementing a small increase of the current expression api, instead of a big change for a more ambitious but risky change I’m willing to prepare a PR. I made it already in C# 5 or 6 back in codeplex times

[WhitWaldo]

It's been a minute since I looked at it, but there's already a fork of Roslyn that's been outfitted to support all the language changes up through C# 11 from Bart de Smet here.

I'm going to have to read through it again and compare to the various points of feedback above as it doesn't use a version number and otherwise works around the System.Linq limitations.

[colejohnson66]

Breaking out a tangent, but: Is there a reason a "new" expression tree system couldn't be "just" a 1:1 mapping of the AST itself? Why do they need to be representative of the "lowered" code? I hate to bring Rust into this, but the input to their "proc macros" are nothing more than a (brace-balanced) token stream. It doesn't even need to be legal Rust syntax.

I know expression trees aren't designed for this, but they're almost perfect for creating DSLs — you can write an interpreter around the nodes to parse out logic. That's basically what EF consumers are doing — turning a C#-like DSL into SQL. And, as C# continues to evolve, this C#-like DSL continues to stray further and further away. If expression trees were merely the AST without any "lowering", you could write much more powerful DSLs, and you wouldn't need to be worried about how compiler optimizations affect the output tree — because the AST comes before optimization.

Basically, rip off the band-aid and create something new. Such a new system would obviously need to be documented as non-exhaustive, and developers should be encouraged to document what their inputs must be. But by being the AST, and not the lowered/optimized code, new compiler optimizations won't affect inputs.

In the future — perhaps — macros could be explored as, essentially, Roslyn preprocessors, using this AST.

[erik-kallen]

This could be done with source generators and interceptors, I assume.

[colejohnson66]

@CyrusNajmabadi How would it expose your internal model? I’m suggesting exposing the raw tokens, not the semantic model. There’s no such thing as “member access”, just a period token. Perhaps, initially, the compiler could ensure it’s valid syntax. As @erik-kallen said, EF consumers could either consume them at runtime or use source generators to generate code at compile time. That would easily lend itself to macros using source generators (where the compiler wouldn’t enforce syntax validity).

[CyrusNajmabadi]

If it's the raw tokens then it's the concrete syntax model, not the abstract one.

[colejohnson66]

And why shouldn't that be exposed? Other languages manage just fine with exposing token streams.

[CyrusNajmabadi]

And why shouldn't that be exposed?

1. Because no one is asking for that. Expression tree consumers, for example, make use of hte semantics associated with the nodes.
2. Because it would be exposing a C# token model as 'the model' everyone would code to. Expression trees work regardless of language. They are lang agnostic.

You are fundamentally asking for somethign else now. And we don't expose something as major as that just off of one person asking for it, with exactly zero consumers saying it's what they actually need over expr trees.

[EmperorArthur]

Hello, just wanting to make sure I'm on the same page, since there are seemingly two different discussions going on. Forgive me if this is re-hashing the old points again, but I feel this might help more than me.

There are fundamentally two different parts to this discussion.

Things Currently Supported By Raw Expression Nodes, But Not Supported In Lambda Expressions

This includes things like throw, which map 1:1 with existing nodes. It also includes things which might be more efficient with new nodes, but could be converted to existing nodes. Like the null-coalescing operator (??) and null-conditional operators (?., ?[]).

CSharpExpressions was already mentioned as an example of solving some of those by implementing new nodes, but allowing most nodes to be reduced to a more complex tree using existing expression nodes. However, another example which does get used in production is LINQKit. Because it's the only way I can easily call a typed lambda expression from another lambda expression.

Things Not Currently Supported By Raw Expression Nodes

These are things which would explicitly require new node types to work. An example noted by CSharpExpressions is below.

Also note that some language constructs such as indexer initializers introduced in C# 6.0 can't be provided as reducible extensions to the LINQ APIs

My Issue / Question

Someone please correct me if I am misunderstanding, but I don't understand the controversy of fixing the first issue. The only possible downside is it would allow someone to more easily write expressions which contain some node types not supported by all query transformers. However, it is possible today to write those same not supported expressions today. They just have to be done by hand.

Meanwhile, the convenience of the null-coalescing operator (??) and null-conditional operators (?., ?[]) can't be understated. When given the choice between materializing a SQL query so all language features can be used, or writing 15 lines of ternary expressions most programmers will choose the former. Yet this can easily lead to horrendous access patterns.

Plus, without LINQKit everything either has to be in one large lambda, violating almost every code readability standard out there, or everything which calls something else must be a hand constructed expression. Which is something most C# programmers, including the rest of my team, aren't familiar with and can't work on.

Would someone please explain to me why fixing this first issue is so controversial.

[marchy]

You should just rip off the bandaid and start pushing updates to the API.

If you want to play it safe - do it on a small surface area of new language features to give the community / providers time to update for compatibility.

Then on the next C# release push out the next wave of updates once library authors get the gist of what’s to come.

Stop overprotecting the community though. We will pick up the pieces. PR’s will be raised, compatibility issues will be fixed. In MUCH less time than it takes you guys to debate all the downsides of how to minimized breakage.

Let’s get on with it already!

[Atulin]

Last I heard, backwards compatibility will remain a priority because big players are paying big money for support, and breaking their .NET Framework 3.5 projects would turn big money into slightly-less-big money as they move somewhere else.

Us, little ones, we just enjoy the crumbs off the table.

[colejohnson66]

Anyone on .NET Framework is not affected by changes in .NET Core 1-3 or .NET 5+

[EmperorArthur]

Anyone on .NET Framework is not affected by changes in .NET Core 1-3 or .NET 5+

Yes and no. As someone who deals with both, you'd be amazed at how much stuff still works with Framework. In this case, the only fallout will be yet another thing that can't be copied and pasted into a decade old code base.

[AlexeyTliss]

I honestly believe this feature is much more needed than Span or Memory since most of big players already manage memory properly where its needed, but runtime code generation is still missing, which is a blocker for many

[colejohnson66]

Highly disagree. I, like most expression tree users, would love to see expression tree evolution, but spans are a god-send when it comes to performance. Managing memory prior to spans was a massive PIA. Want to do anything with a substring or subarray? That’s an allocation you (almost never) can avoid. Sure, ArraySegment<T> existed, but it was ugly to use. Spans unified everything!

[AlexeyTliss]

We write an LLM train api, needless to say we process hundredrs of gigabytes of text, and we find StringBuilder and raw string access to be way more effective - and most impotant - way more simple to write and read - than using spans.

[CyrusNajmabadi]

@AlexeyTliss Your use case is one of millions of customers we have to consider. It's fine that you think this is more important for your own needs. But that doesn't mean the same holds true for the ecosystem as a whole.

[voroninp]

@CyrusNajmabadi , but let's be honest, expressions are lagging behind C# features like hell. =(
Even block statements are not supported. ='(

[fabianoliver]

I'd love to voice support for an evolution of expression trees as well.

Since it was mentioned the depriorization of work on this is due to not meeting the bar for investment/not having enough customer relevancy compared to other features, I thought I'd just throw in my anecdotal experience.

In 4 quite different types of jobs that I've worked in over the past 15 years or so, I always ended up working on something where using expression trees was extremely beneficial, if not borderline inevitable. And I should say, none of these use cases were at all related to EF Core (just mentioning as my impression is much of the discussion - understandably - seems to gravitate a bit around Entity Framework. But there are so, so many other great use cases too!).

My typical use case is almost always performance related. As random examples, my last use cases, both from within the past few weeks alone:
A) Generating efficient deserializers for avro schemas that are pulled from schema registry at runtime (i.e. JSON source gen-like compile time generation is not possible)
B) Translating user GraphQL queries to efficient graph resolvers (which again needs to happen at runtime as those are highly dynamic)

So oftentimes stuff that is typically highly reflection based, but needs to be made significantly more efficient for some type of repeated invocations.

The big issue for me is that there isn't much of an alternative really. Sometimes you can work your way around compiling a new assembly on the fly via Roslyn and loading that up. I've done that for another recent project. But in my view, it's not a great replacement in many cases - personally,

1. I found it MUCH harder to use. For a start, I found no resources detailing how do compile and load a dynamic assembly using exactly the same references as the hosting assembly - ultimately had to dig around to Roslyn repository for a fair while to find ScriptMetadataResoler & co. I could well be wrong, but the seeming lack of online resources in that area would indicate to me that's just not a very widely adopted approach by the wider community
2. You'll immediately break on any platform that doesn't support dynamic code emission
3. Best I know, there's no way to unload these assemblies again; unlike compiled expression trees, which are GC'able (imagine my use case B above, compiling user-provided queries for an app that might be running weeks or months - not ideal)
4. The most minor of all reasons, but: It might just not be nice thing to do to pull in all Roslyn dependencies. If you ship a nuget library, there's definitely a bit of a qualitative difference between "here's my dependency-free package" vs "here's my package which will need to pull in a dozen huge Roslyn assemblies"

As for the elephant in the room, i.e. compatibility concerns: I definitely understand those in the context of query providers (at least somewhat; albeit last time I used EF, many many years ago, only some expressions were supported anyway, so you never quite knew what worked or not until runtime - did that change?).
However, relatively wide-ranging use cases like the ones outlined here don't even have that concern. Those only rely on being able to compile the expression tree; as long as that works, anything is fair game.

Long story short: Support for modern features in expression trees is sorely missed! For my personal work (!), this would be a much more beneficial than most other ongoing language features at the moment, so I'd definitely be really excited if those were revisited!

[erik-kallen]

only some expressions were supported anyway, so you never quite knew what worked or not until runtime

It is absolutely like that! And for the features that do "work", you also need to check what SQL they will generate.

[ilmax]

Linq recently got some new features (LeftJoin and RightJoin) hence the argument made in this comment may not be a good reason anymore to avoid any changes in the library.
It would be awesome to add support for the elvis operator (?.) and some other low hanging fruits to make working with expression trees a better experience overall.

[CyrusNajmabadi]

Anyway named arguments are not my main issue.

Everyone has their own views on what the main issue should be :)

[ranma42]

Could null-coalescence and other null helpers be implemented, then? Things like foo ?? bar could easily be lowered to foo == null ? bar : foo,

Side note: ?? is already handled by the existing Expressions.

foo?.bar could be lowered to foo == null ? null : foo.bar, and so on.

That alone would make expressions much more ergonomic, especially in the context of EF, where nullable nav properties are a common occurence.

A very interesting library that actually does this (in a somewhat specific scenario) is https://github.com/koenbeuk/EntityFrameworkCore.Projectables/

It achieves this by implementing:

• a code generator that converts the (Roslyn/code generator) syntax tree of annotated methods into properties that build an "appropriate" Expression
• an interceptor that integrates with EFCore to replace the invocations to annotated methods with the corresponding expressions.

For example, it supports ?. as follows:

using System;
using System.Linq.Expressions;
using EntityFrameworkCore.Projectables;
using EntityFrameworkCore.Projectables.Extensions;

Expression<Func<int?, int>> expr1 = (x) => x ?? 0;
Expression<Func<Example, int?>> expr2 = (x) => Example.OptionallyGet(x);

Console.WriteLine(expr1);
// x => (x ?? 0)

Console.WriteLine(expr2);
// x => OptionallyGet(x)

Console.WriteLine(expr2.ExpandProjectables());
// x => IIF((x != null), Convert(x.Field, Nullable`1), Convert(null, Nullable`1))

public class Example {
	public int Field;
	
	[Projectable(NullConditionalRewriteSupport = NullConditionalRewriteSupport.Rewrite)]
	public static int? OptionallyGet(Example x) => x?.Field;
}

I think this is a very interesting approach and investigating it further could be very effective, as it allows each "consumer" of the expressions to implement a slightly different translation from C# to their expressions. For example in the case of pure expressions, repeating a subexpression (as done by EntityFrameworkCore.Projectables) is just fine, while on side-effectful expressions it would certainly be a bad idea.

This approach can also provide compile-time diagnostics about issues in the expression translation; for example

	[Projectable]
	public static int? OptionallyGet(Example x) => x?.Field;

fails with the following error marked on x?.Field

'x?.Field' has a null-conditional expression exposed but is not configured to rewrite this (Consider configuring a strategy using the NullConditionalRewriteSupport property on the Projectable attribute)(EFP0002)

Another reason why I find this interesting is that this also opens the door to converting C# ASTs to any other syntax, regardless of what can be represented in System.Linq.Expressions; for example, one could emit "extension nodes" or even unrelated trees, such as F# code quotations.

The major disadvantages I can see regarding such an approach are:

• risk of fragmentation of what is supported; as long as the translator supports the same subset of ASTs as the consumer of the expressions, this might not be a real problem (and if it provides compile-time feedback, it might even be an improvement over the status quo)
• duplication of efforts - among different translators/consumers; compiler-provided Expressions vs custom-generated ones; in the implementations required to support C#, VisualBasic F# (and any other context that emits Expression?)

Even with these significant issues, I think this is a direction that might make sense to investigate and does not necessarily require effort within the compiler, but can instead be implemented with the existing tooling.

[olmobrutall]

Anyway named arguments are not my main issue.

Everyone has their own views on what the main issue should be :)

This is an error that got this morning:

I think I have received around 200+ similar errors in the last years. Almost always related to using !. instead of ?. in an expression that returns a value type. The LEFT OUTER JOIN / OUTER APPLY produces a null but this can not be converted to a value type (in this case bool).

Just to mention that "waiting forever" also produces exceptions in production that depend on data (the query returns null).

While adding support ConditionalAccessExpression will trade a compile-time error by a runtime exception:

• Only for new code (using ?.)
• As long as the LINQ provider doesn't implement it (it's relatively simple).
• The exception will also happen in every execution of the query, not dependently of data, so there is less chances that it goes to production.

Can I get green light to attempt a PR? Pleeeeease

[bachratyg]

Let's just flesh this one out and get everyone on board with the general direction first.

• Would the BCL guys accept a new expression class w/ ExpressionType.Extension at all? (hope so)
• All other nodes can be told apart by NodeType, this one only via reflection. Is this an issue? (EF already does this, it should be good enough)
• What would be the interpreter/codegen story? Override Reduce and fall back to statement expressions akin to e.g. MemberInitExpression so that the lambda compiler does not need to be modified? (as far as I can see it already supports reduce)
• Is it okay to special case BoundKind.ConditionalAccess in ExpressionLambdaRewriter?

[bachratyg]

Continuing from this, it's possible to do it with csharplang/roslyn involvement only.

Consider this helper method:

public static B ConditionalAccess<A, B>(this A? receiver, Func<A, B> accessor) => receiver == null ? null : accessor(receiver);

Whenever a compiler sees a ?. it emits an Expression.Call if a regular/extension method with this name/shape is present (a lot of stuff e.g. foreach, await, deconstruction works this way so this is not a new concept). This method could be provided by the end-user, some library/nuget or even generated on-demand by roslyn. The important thing is that it has a well-known name/namespace so that providers can detect and special-case it.

The main drawback here is that this pattern only works for additions that can be expressed as a method call so no out var/await/etc.

[erik-kallen]

Perhaps a library-only solution using interceptors could work? Something like

MyExpression<int, int, int> = MyExpression.Make((a, b) => a + b);

and a source generator could intercept the call and generate appropriate factory method calls?

[ranma42]

@bachratyg that does not compile because it is still directly relying on the quoting performed by Roslyn.

Instead, this compiles:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;

var src = new string?[] { "asd" }.AsQueryable();

var q = MyExpression.Make(() =>
    from a in src.AsEnumerable()
    where a?.ToString() == ""
    select a
);

var q2 = src.Make(s =>
    from a in s
    where a?.ToString() == ""
    select a
);
    

static class MyExpression
{
    public static Expression<T> Make<T>(T wrapped)
        => throw new NotImplementedException();

    public static Expression<U> Make<T, U>(this IQueryable<T> source, Func<IEnumerable<T>, U> wrapped)
        => throw new NotImplementedException();
}

[bachratyg]

You should probably not use AsQueryable() then

That's just to have some variable to build on. In prod it's likely a DbSet or some IQueryable method argument.

[erik-kallen]

The problem, as pointed out by both me and @ranma42, is that the IQueryable methods take an Expression<Func> parameter. The sourcegen will have to work on IEnumerable instead, whose extnsion methods instead take a Func

[bachratyg]

@erik-kallen there's no question about why this needs to be done. I have concerns about how to do this so the result is good enough.

Boilerplate-wise this:

var q = MyExpression.Make(() =>
    from a in src.AsEnumerable()
    where a?.ToString() == ""
    select a
);

is no improvement over this:

var q =
    from a in src
    where a.Elvis(s => s.ToString()) == ""
    select a

The latter does not even need sourcegen and all the trouble it brings.
Can we do better?

[erik-kallen]

No, there is no way to get a conditional operator into a quoted expression.

Whether it is an improvement is in the eye of the beholder. And the .Elvis() can't be translated to SQL anyway.

Also, a sourcegen solution could theoretically support

var q = from a in src where a.Name?.ToString() == "" select a.Data switch { 1 => "a", 2 => "b", _ => "other }

[agocke]

Maybe I can add some context here.

We did make a big change to expression trees in recent memory: the move to Roslyn. The compiler is responsible for encoding the tree represented in expression trees, and when we rewrote the compiler to C# in Roslyn we rewrote the implementation for the tree. This was tricky because the native compiler actually just dumped the in-memory tree into the expression tree. Roslyn doesn't use the same tree, so we had to map from the new Roslyn trees to the old native trees.

We did a pretty good, but not great, job. There were a number of bugs around different nodes being generated for the same source. This is how we discovered that providers are very sensitive to the exact shape of the tree. There were many bugs filed around tree differences.

At first we tried to fix those. However, it didn't take long for us to get a new set of bugs about how "fixing" the tree broke their provider since they had adapted to the new tree.

This was made worse by the fact that dumping implementation details (the native compiler's tree) into public surface area was not good. Some of the trees were almost impossible to generate well from Roslyn. And some of the trees were just incorrect, because the native compiler had a kind of buggy or incorrect internal representation of a particular syntax. Roslyn has this new thing that is a public, supported, designed surface area for semantic nodes (IOperation), but it's completely disjoint from the expression tree system.

Roslyn has also added a bunch of source generation features that overlap with expression tree scenarios. They also address some big problems with expression trees. For instance, expression trees allow you to express things that would be illegal in C#, like violating accessibility. While this is useful, it's also problematic. By generating source that split is avoided. It also enables Native AOT support since certain expression trees require dynamic code compilation to execute, and all expression trees have to be interpreted otherwise. Source generation usually creates compatibility around build time, not run time. There are limitations to source generation but also benefits in that you usually figure out the problems when you recompile, not when you update the .NET runtime in production.

Lastly, there's the matter of testing. While the test suite isn't bad, it's not to the level of completeness that would catch all possible breaks, as can be seen with the previous fixes. That makes changes riskier and more expensive.

All together, the state of expression trees is a challenging environment to make changes in. While improvements could provide benefit, any change would have to carefully weigh cost and risk to be accepted.

[FredyFerrari]

Maybe I can add some context here.

Thank your for this clarifying and very interesting context!

[agocke]

Tests are OSS and in the dotnet/runtime repo and dotnet/Roslyn repo. I don’t know of any more comprehensive private suite.

[Atulin]

Source generators are amazing, easily one of my most favourite features added in the recent years.

Unfortunately, unless EF and its ilk somehow start using sgen for query generation, the deficiencies of expressions will remain a pain point.

[CyrusNajmabadi]

EF doesn't need to do anything here. this can be supplied with another SG that intercepts the calls instead.

[zspitz]

Tests are OSS and in the dotnet/runtime repo and dotnet/Roslyn repo. I don’t know of any more comprehensive private suite.

I've written a comprehensive library of expression trees that I used in my ExpressionTreeToString library. For testing, I wanted to create compiler generated expressions in C#, VB.NET, and via the factory methods.

The unit tests then checks that each expression tree produces the expected string for various formatters; it also tests against the native .NET ToString and DebugView representations.

Some overloads of the factory methods don't have corresponding test objects, specifically those overloads with an additional MethodInfo parameter. I was thinking it might be possible to find all references to these additional overloads within Roslyn to see where they might be used, but I didn't get around to it.