Skip to content

[Feature][Metric] Custom SQL support export error data#586

Merged
zixi0825 merged 3 commits into
datavane:devfrom
xxzuo:feature-560
Jul 11, 2026
Merged

[Feature][Metric] Custom SQL support export error data#586
zixi0825 merged 3 commits into
datavane:devfrom
xxzuo:feature-560

Conversation

@xxzuo

@xxzuo xxzuo commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Purpose of this pull request

Summary

This PR adds a new custom_count_sql metric that allows users to define custom SQL for invalid/error data extraction and automatically calculates the invalid row count from that SQL.

It also refactors custom SQL table discovery so metric plugins can define which SQL field should be used for metadata/table extraction, instead of hard-coding actual_aggregate_sql.

Changes

  • Add the custom_count_sql metric plugin.
  • Register the new metric plugin through META-INF/services.
  • Add invalidate_items_sql as a metric configuration key.
  • Add UI support for SQL-style metric config fields using TextArea.
  • Add localized placeholders for invalidate_items_sql.
  • Add SqlMetric#getTableDiscoverySql and SqlMetric#setTableDiscoverySql.
  • Update Local, Spark, and Flink engine config builders to use the new table discovery API.
  • Update JobServiceImpl to resolve metadata using the metric-provided table discovery SQL.
  • Make CustomCountSql:
    • discover tables from invalidate_items_sql
    • generate invalid item output from the custom SQL
    • count invalid rows from ${invalidate_items_table} for normal engine flow
    • count directly from the custom SQL for Local direct execution

close #560

Check list

  • Code changed are covered with tests, or it does not need tests for reason:
  • Change does not need document change

xxzuo added 3 commits January 31, 2026 09:52
- Replace direct ACTUAL_AGGREGATE_SQL access with SqlMetric.getTableDiscoverySql in custom SQL table extraction across engines
- Update rewritten SQL through SqlMetric.setTableDiscoverySql
- Add getTableDiscoverySql and setTableDiscoverySql implementations for CustomCountSql
- Use SqlMetric table discovery SQL in JobServiceImpl when resolving metadata
- Centralize table discovery SQL handling to improve extensibility and maintainability
@zixi0825
zixi0825 merged commit c96e47e into datavane:dev Jul 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][Metric] Custom SQL support export error data

2 participants