fix: add Dremio-specific edr_dateadd and improve edr_cast_as_timestamp by devin-ai-integration[bot] · Pull Request #957 · elementary-data/dbt-data-reliability

devin-ai-integration · 2026-03-03T13:46:17Z

Summary

Moves Dremio-specific macro overrides from the elementary CLI project into dbt-data-reliability where they belong.

edr_dateadd: Refactored to use adapter.dispatch (matching the pattern used by other edr_* macros like edr_cast_as_timestamp). The existing logic moves to default__edr_dateadd. A new dremio__edr_dateadd is added to work around two bugs in dbt-dremio's native dateadd:

Calls .replace() on the interval parameter which fails when interval is an integer
Wraps the result in select TIMESTAMPADD(...) creating $SCALAR_QUERY errors when embedded in larger expressions

dremio__edr_cast_as_timestamp: Enhanced with two additional REGEXP_REPLACE calls to normalize ISO 8601 timestamps that Dremio's Gandiva engine cannot parse — specifically the T date-time separator and Z UTC suffix.

Review & Testing Checklist for Human

Verify adapter.dispatch refactor doesn't regress non-Dremio adapters. The structural change from direct dbt.dateadd call to adapter.dispatch('edr_dateadd', 'elementary') affects all adapters. Confirm that default__edr_dateadd is correctly resolved for Postgres, Snowflake, BigQuery, etc.
Check that the hardcoded datepart list in dremio__edr_dateadd covers all dateparts used in the codebase. Currently supports: year, quarter, month, week, day, hour, minute, second. Any other datepart (e.g. millisecond) will raise a compiler error.
Validate the triple-nested REGEXP_REPLACE in dremio__edr_cast_as_timestamp — the regex chain '(\d)T(\d)' → '$1 $2', '(\.\d{3})\d+' → '$1', 'Z$' → '' should correctly normalize ISO 8601 to YYYY-MM-DD HH:MM:SS.sss.
Recommended test plan: Run the Dremio e2e tests from elementary PR #2135 after this is merged, with the dateadd/cast_column overrides removed from the elementary CLI project — this validates the macros work end-to-end from dbt-data-reliability.

Notes

This PR is a prerequisite for cleaning up elementary PR #2135, which currently carries these overrides in the CLI's monitor dbt project.
The interval.replace('order by 1', '') in dremio__edr_dateadd is a workaround for dbt-dremio appending order by 1 to its scalar subquery output — this is fragile but necessary until dbt-dremio fixes its dateadd macro upstream.
Link to Devin session
Requested by: @haritamar

Summary by CodeRabbit

New Features
- Added cross-database date-add support with explicit Dremio-compatible expressions for reliable date arithmetic and clear handling of supported date parts.
Improvements
- Improved timestamp parsing to better handle ISO 8601 variants: normalizes separators, trims excess sub-millisecond digits, and normalizes timezone markers for more robust casting.

- Convert edr_dateadd to use adapter.dispatch for adapter-specific overrides - Add dremio__edr_dateadd that fixes dbt-dremio's dateadd bugs: 1. Casts interval to string before calling .replace() (fixes integer interval) 2. Outputs TIMESTAMPADD expression directly (no scalar subquery wrapper) - Update dremio__edr_cast_as_timestamp to normalize ISO 8601 timestamps: 1. Remove 'T' date-time separator (Gandiva can't parse it) 2. Remove 'Z' UTC suffix (Gandiva rejects unknown zone) 3. Truncate sub-millisecond precision Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

devin-ai-integration · 2026-03-03T13:46:24Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

github-actions · 2026-03-03T13:46:28Z

👋 @devin-ai-integration[bot]
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in the elementary repository.

coderabbitai · 2026-03-03T13:46:33Z

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

Linear integration is disabled

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between b666ec7 and c12a6fd.

📒 Files selected for processing (2)

macros/utils/cross_db_utils/dateadd.sql
macros/utils/data_types/cast_column.sql

📝 Walkthrough

Walkthrough

Adds an adapter-dispatched edr_dateadd macro with a Dremio-specific implementation and introduces a Dremio timestamp-cast macro that normalizes ISO8601 strings (replace T→space, truncate fractional seconds to 3 digits, remove trailing Z) before casting.

Changes

Cohort / File(s)	Summary
Date Addition Macro Framework `macros/utils/cross_db_utils/dateadd.sql`	Adds `edr_dateadd(datepart, interval, from_date_or_timestamp)` dispatcher, `default__edr_dateadd(...)` fallback (resolves to `dbt.dateadd` or `dbt_utils.dateadd` or errors), and `dremio__edr_dateadd(...)` emitting `TIMESTAMPADD` expressions with normalized datepart handling and explicit unsupported-part error.
Timestamp Normalization / Casts `macros/utils/data_types/cast_column.sql`	Adds `dremio__edr_cast_as_timestamp(timestamp_field)` that applies nested `REGEXP_REPLACE` transformations to convert `T`→space, truncate fractional seconds to three digits, and remove trailing `Z`, then casts to `TIMESTAMP` for Dremio/Gandiva.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 I hopped through timestamps, nudged a stubborn "T",

I trimmed the ticks to three so Gandiva agrees,
I wove dateparts into Dremio's tune,
Now queries bound bright beneath the moon,
🥕✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: adding Dremio-specific edr_dateadd macro and improving edr_cast_as_timestamp. Both are primary changes clearly reflected in the file modifications and PR objectives.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch devin/1772545473-dremio-dateadd-cast-timestamp

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

…, handle uppercase ORDER BY 1 Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

haritamar · 2026-03-03T14:03:01Z

+{% macro dremio__edr_dateadd(datepart, interval, from_date_or_timestamp) %}
+    {% set datepart = datepart | lower %}
+    {% set interval = interval | string %}
+    {% set interval = interval.replace('order by 1', '').replace('ORDER BY 1', '') %}


Not sure I understand this. Why would the interval contain order by 1?
This feels like a hack

You're right, this is a hack from dbt-dremio's original dremio__dateadd macro (line 32 of dbt/include/dremio/macros/utils/date_spine.sql). Since our edr_dateadd uses adapter.dispatch and bypasses dbt-dremio's macro entirely, the interval will never contain order by 1 here — it was only relevant in dbt-dremio's own recursive calls. Removing it.

haritamar · 2026-03-03T14:03:38Z

 {%- macro dremio__edr_cast_as_timestamp(timestamp_field) -%}
-    cast(REGEXP_REPLACE({{ timestamp_field }}, '(\.\d{3})\d+', '$1') as {{ elementary.edr_type_timestamp() }})
+    cast(
+        REGEXP_REPLACE(


Do we really need three regexp_replace?
Can't we replace with one?

Good point — consolidating into a single REGEXP_REPLACE with alternation pattern.

I tried consolidating into one REGEXP_REPLACE with alternation ((\d)T(\d)|(\.\d{3})\d+|Z$), but it doesn't work because each branch needs a different replacement string (space for T, truncation for sub-ms, empty for Z). Dremio's REGEXP_REPLACE only takes a single replacement string, and unmatched group backreferences produce empty strings — so '$1$3 $2' adds spurious spaces when only the sub-ms or Z branch matches.

Updated the comment to explain this and added a concrete example: '2024-01-15T12:30:00.123456Z' → '2024-01-15 12:30:00.123'.

…mprove comments - Remove unnecessary 'order by 1' stripping from dremio__edr_dateadd (not needed since adapter.dispatch bypasses dbt-dremio's macro entirely) - Keep three REGEXP_REPLACE calls since each has a different replacement pattern (space, truncation, removal) that can't be combined into a single call - Add concrete example in comment showing the transformation Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

…tamp Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

devin-ai-integration

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 4 additional findings.

devin-ai-integration bot assigned haritamar Mar 3, 2026

This comment was marked as resolved.

Sign in to view

fix: address CodeRabbit review - cast to string before REGEXP_REPLACE…

35f05d3

…, handle uppercase ORDER BY 1 Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

haritamar reviewed Mar 3, 2026

View reviewed changes

devin-ai-integration bot and others added 2 commits March 3, 2026 14:05

Merge branch 'master' into devin/1772545473-dremio-dateadd-cast-times…

4899a7a

…tamp Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

haritamar approved these changes Mar 3, 2026

View reviewed changes

style: apply sqlfmt formatting to dateadd.sql

c12a6fd

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

haritamar merged commit dd42ece into master Mar 3, 2026
8 of 24 checks passed

haritamar deleted the devin/1772545473-dremio-dateadd-cast-timestamp branch March 3, 2026 17:21

devin-ai-integration bot commented Mar 3, 2026

View reviewed changes

devin-ai-integration bot mentioned this pull request Mar 3, 2026

feat: add DuckDB, Trino, Dremio & Spark support to CI and CLI elementary-data/elementary#2135

Merged

5 tasks

Conversation

devin-ai-integration bot commented Mar 3, 2026 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review & Testing Checklist for Human

Notes

Summary by CodeRabbit

Uh oh!

devin-ai-integration bot commented Mar 3, 2026

🤖 Devin AI Engineer

Uh oh!

github-actions bot commented Mar 3, 2026

Uh oh!

coderabbitai bot commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

This comment was marked as resolved.

Uh oh!

haritamar Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

haritamar Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration bot commented Mar 3, 2026 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Mar 3, 2026 •

edited

Loading