fix(prqlc, append): apply column order on CTEs too

[Fanaen]

This is my third attempt to solve the issue of append positional mapping.

It enables the workaround for #2680

Problem

• SQL's UNION need the same number of column and in both queries
• PRQL currently changes main query's columns without changing sub query as well
• PRQL outputs a SQL query that leads to a type mismatch or a column count mismatch error. More elaborate cases are a mix of those two. You may try a simplified version of either with the Query Results tab in the playground:

# Column swap leading to type mismatch
from invoices | select { invoice_id, invoice_date }
append (
  from invoices | select { invoice_id, invoice_date }
)
select { invoice_date, invoice_id }
# -> E: Unimplemented type for cast (BIGINT -> TIMESTAMP)

from invoices | select { invoice_id, invoice_date }
append (
  from invoices | select { invoice_id, invoice_date }
)
select { invoice_date }
# -> E: Set operations can only apply to expressions with the same number of result columns

Solution

This fix operate in anchoring stage only.

There's two parts:

• when a pipeline is compiled, it checks the columns before and after and remember the difference for each affected relation.
• when a CTE relation is compiled, it looks if there a mapping for it and reorder or even drops required columns.

[max-sixty]

thanks @Fanaen — I haven't gone through the whole thing to compare, but it looks like a reasonable approach

do you know why the tests are failing here? same issue as the prior PR? we can skip sqlite tests if the tradeoff is better support for the main dialects

[Fanaen]

Thanks @max-sixty! Yeah, no need to compare. This attempt is very different, and I think a lot better. All the extra baggage of the previous attempts would have been detrimental to the readability of this one, hence the new PR. I'll scrap the other one.

I'll keep that in mind! Yep, same tests, same issues: only this time, the SQL results are good!

This fix should be fully dialect independent. I wanted integration tests with different table sources, and I did not see that the table columns were not matching in some dialects.

This should be easy to fix. I'll look into that tomorrow.

[Fanaen]

We're good to go!

There is one issue I encountered while working on this. When there is a aggregate whose result is SELECT ... COUNT(*) AS col FROM tbl GROUP BY, this leads to a column connected to nothing, and it's removed. This case wasn't working properly before, so I believe it could be a future fix.

[max-sixty]

nice, thanks a lot @Fanaen !